Thread: asparagus - natural language testing based on dejagnu/TCL

Originally Posted by graprof

you are right, it's formalised NL. valid expressions are expected to start with a `given' step, followed by a bunch of `when's `and's and `then's. The syntax of these steps is up to the developers though, as long as they implement the used steps in TCL.

Nice. So (apart from the NL) what sets this apart from a system or guideline where programmers must assert pre-/post-conditions, invariants, etc, etc? I guess there is the advantage that non-programmers can write the "test" with very little training but I'm still worried about non-programmers writing the tests. Certainly INPUT from non-programmers is important, heck we can't all think of everything, but isn't this akin to a non-mathematician speculating about all sorts of things that cannot possibly be real or achievable in reality? As I said, I do like the concept but... I can't see the advantages (yet).

Maybe I should clarify -

This is not a test harness. Instead, this *uses* the test harness provided by dejagnu, which makes it stupidly easy to integrate this kind of testing into autotools managed projects (while the standalone `runtest' variant also works).
Beyond that, you usually want to separate the test assertions from the step definitions, mainly because you want to avoid typing the same stuff all over again. Following this train of thought, it is logical to extract the most commonly used steps into a framework to enable people to share their step definitions, so that they don't have to invent their own wheels individually.

Of course you could implement your own test harness, but why would you want to do that, if the canonical solution (dejagnu) provides an excellent harness?
Of course you could implement your tests using plain TCL, but in my experience this is a real pain to do, so why not use something simpler?

What I wanted was to provide a convenient way to write tests in TCL, utilizing the dejagnu test harness for my autotools projects. What I wanted to avoid was manually writing TCL assertions.

The natural language stuff is just an addition I threw into the mix, because I think it is the coolest thing ever, but your milage may vary.

In the decade I spent with C and programming in general, and also during my CS Studies, I began to appreciate writing things that people of other professions could read, which is not easy to do if you are from a C background, but oh so important, especially when it comes to tests. If you take n programmers and put them into a room, you will have n+1 different testing approaches, which sucks if these people have to collaborate, because it eats time these people could have used more productively.

If these developers would be trained to write their expectations using a formal language that is programming language independent, their synchsonization overhead would be much smaller, and the integration of other developers into the project gets easier.

Originally Posted by phantomotap

That is a fine goal in and of itself; rereading your original post, it seems as if you only intended to express the requirements of such assertions in a different form. (I read as saying, basically, what the traits the TCL code should express and ways to validate those tests.) If the goal is truly generative as you say, that is an entirely different beast. Referencing the above--if you intend to produce the "exp" file--or equivalent--directly from the "natural language" with no TCL coding required on the other side, that seems like a fine goal.

Well, here one might surely argue that I'm biased; again, I despise TCL.

Soma

no no, you will still have to write your step definitions in TCL, but the formal language expectations in the .exp files allow massive reuse of these steps, because many tests share similar steps, and the most common steps are already provided e.g. `Then I should see "<text>'

Originally Posted by graprof

Now, if you have ever done something with ruby on rails, you may know cucumber testing, which is an awesome way of writing natural language tests that are fun to write, easy to read and generally awesome all the way!

I wanted to incorporate the fun cucumber-esque way of writing natural language tests into dejagnu, to have seamless integration into any autotools projects out there, to help developers find the fun in writing tests, and subsequently produce better software.

To give an example, a simple test could look like this:

Code:

  # test that `ls -a` lists the parent directory 
  Given an executable "ls" \
    when I run with parameters "-a" \
    then I should see ".."

which is undeniably a very intuitive way to specify assertions.

[snip]

If you would like to collaborate because you are passionate about your testing, or just have any questions on why I am doing all this, or why natural language testing is cool, or about anything just let me know

I don't agree with everything Edsger Dijkstra had to say, but I think he was spot on when he said "Projects promoting programming in natural language are intrinsically doomed to fail". But, natural language aside, my gut tells me a DSL for testing is always going to be more trouble that it's worth.

Do you have a more complete cucumber test(s) that demonstrates an advantage over standard testing?

I have recently whipped up a data generator that pulls schema definitions from files or stdin, using a bison/flex parser, and have written a couple of assertions on the behaviour of the parser, here is an excerpt:

Code:

# test for conditions triggering -Wempty-schema
#   should be enabled by default
set test_name "\[-Wempty-schema\]"
given an executable "../src/dbtesma" \
  when I run \
  and I send \x04 \
  then I should see "warning: no table definition" \
  when I run with parameters "-Wno-empty-schema" \
  and I send \x04 \
  then I should not see "warning: no table definition"


# test for conditions triggering -Wempty-table
#   should be enabled by default
set test_name "\[-Wempty-schema\]"
given an executable "../src/dbtesma" \
  when I run \
  and I send "table { name = \"test\" }\n\x04" \
  then I should see "warning: no column definitions in table `test'" \
  when I run with parameters "-Wno-empty-table" \
  and I send "table { name = \"test\" }\n\x04" \
  then I should not see "warning: no column definitions in table `test'" \


# test for conditions triggering -Wconversion
#   should be enabled by default
set test_name "\[-Wconversion\]"
given an executable "../src/dbtesma" \
  when I run \
  and I send "table { name = \"test\" rows = 999999999999999999999 }\n\x04" \
  then I should see "warning: `999999999999999999999': Numerical result out of range" \
  when I run with parameters "-Wno-conversion" \
  and I send "table { name = \"test\" rows = 999999999999999999999 }\n\x04" \
  then I should not see "warning: `999999999999999999999': Numerical result out of range"


# test for warning elevation when -Werror is specified
set test_name "\[-Werror\]"
given an executable "../src/dbtesma" \
  when I run \
  and I send "table { name = \"test\" } table { }\n\x04" \
  then I should see "warning: no column definitions in table `test'" \
  and I should see "error: `table' definition missing required `name' property" \
  when I run with parameters "-Werror" \
  and I send "table { name = \"test\" } table { }\n\x04" \
  then I should see "warning: no column definitions in table `test'" \
  and I should not see "error: `table' definition missing required `name' property" \
  

# test for warning suppression when -g is specified
set test_name "\[-q\]"
given an executable "../src/dbtesma" \
  when I run \
  and I send \x04 \
  then I should see "warning: no table definition" \
  when I run with parameters "-q" \
  and I send \x04 \
  then I should not see "warning: no table definition" \
  when I run with parameters "-q -Werror" \
  and I send \x04 \
  then I should see "warning: no table definition"

By the way, I agree with Dijkstra on natural language processing, but this does not really apply here, as the test framework does not care about the interpretation and the semantics of natural language, which is the difficult problem he was probably refering to. Instead, the meaning of the steps is made explicit by defining them in TCL.