Thread: asparagus - natural language testing based on dejagnu/TCL

  1. #16
    misoturbutc Hodor's Avatar
    Join Date
    Nov 2013
    Posts
    1,795
    Quote Originally Posted by graprof View Post
    you are right, it's formalised NL. valid expressions are expected to start with a `given' step, followed by a bunch of `when's `and's and `then's. The syntax of these steps is up to the developers though, as long as they implement the used steps in TCL.
    Nice. So (apart from the NL) what sets this apart from a system or guideline where programmers must assert pre-/post-conditions, invariants, etc, etc? I guess there is the advantage that non-programmers can write the "test" with very little training but I'm still worried about non-programmers writing the tests. Certainly INPUT from non-programmers is important, heck we can't all think of everything, but isn't this akin to a non-mathematician speculating about all sorts of things that cannot possibly be real or achievable in reality? As I said, I do like the concept but... I can't see the advantages (yet).

  2. #17
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    [Edit]
    Just to be Sally, I'm not trying to harsh your project. If you like "Cucumber", you should absolutely go for it. I'd question the utility of a "natural language" harness in many languages; like I said, I can see the "natural language" expression being a good fit in the right circumstances. Your case of "dejagnu" and TCL may be such circumstances.

    Of course, it may be that you've bound together "Cucumber" as a harness and the remote tools provided. To be sure, I use five C++ compilers every day through a secondary tool. If you intend to use "Cucumber" over TCL and "dejagnu" because of binding between environments, I completely understand that need.
    [/Edit]

    I am not even sure why that is, the only valid reason you have brought up against it is its verbosity, which I still think is negligible.
    O_o

    I didn't say anything about "verbosity" until after you had so referenced. I said the "natural expression" harness was tedious; I am expressing the exact opposite opinion you offered in saying that a "natural language" harness would "help developers find the fun in writing tests".

    I think it is sad that most people from a traditional programming background are a bit biased against natural language testing.
    I'm "biased" over expressing "acceptance testing" exactly once. Perhaps with enough advancement in parsing and "compiling" natural language, we might one day see a harness expressed with natural language which may generate relevant compliance code on behalf of the developer.

    As is, with "Cucumber", the separation of expression and implementation of "acceptance testing" is tedious. You have to write the harness, the expression of "acceptance testing", and you also have to write the implementation (`when(/**/)', then(/**/), and similar) partially or completely duplicating the expression within the target language while also manually "stepping" through "conformance points".

    Many language specific harnesses exist, but I have no use form them if I am responsible for doing so much work. Yes. I am well aware the software development isn't easy. No one has ever said to me that software development would be only a little work. Actually writing code that passes all relevant tests is difficult enough when the compiler/interpreter has enough "juice" to directly express "acceptance tests"; I'm not going to be happy using a harness that makes that job more time consuming just because someone with no software development experience wants to write their own tests.

    Soma
    Last edited by phantomotap; 02-21-2014 at 04:04 AM.
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  3. #18
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    Maybe I should clarify -

    This is not a test harness. Instead, this *uses* the test harness provided by dejagnu, which makes it stupidly easy to integrate this kind of testing into autotools managed projects (while the standalone `runtest' variant also works).
    Beyond that, you usually want to separate the test assertions from the step definitions, mainly because you want to avoid typing the same stuff all over again. Following this train of thought, it is logical to extract the most commonly used steps into a framework to enable people to share their step definitions, so that they don't have to invent their own wheels individually.

    Of course you could implement your own test harness, but why would you want to do that, if the canonical solution (dejagnu) provides an excellent harness?
    Of course you could implement your tests using plain TCL, but in my experience this is a real pain to do, so why not use something simpler?

    What I wanted was to provide a convenient way to write tests in TCL, utilizing the dejagnu test harness for my autotools projects. What I wanted to avoid was manually writing TCL assertions.

    The natural language stuff is just an addition I threw into the mix, because I think it is the coolest thing ever, but your milage may vary.

    In the decade I spent with C and programming in general, and also during my CS Studies, I began to appreciate writing things that people of other professions could read, which is not easy to do if you are from a C background, but oh so important, especially when it comes to tests. If you take n programmers and put them into a room, you will have n+1 different testing approaches, which sucks if these people have to collaborate, because it eats time these people could have used more productively.

    If these developers would be trained to write their expectations using a formal language that is programming language independent, their synchsonization overhead would be much smaller, and the integration of other developers into the project gets easier.

  4. #19
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Beyond that, you usually want to separate the test assertions from the step definitions, mainly because you want to avoid typing the same stuff all over again. Following this train of thought, it is logical to extract the most commonly used steps into a framework to enable people to share their step definitions, so that they don't have to invent their own wheels individually.
    O_o

    You know, let us just stop here for moment.

    I don't know Ruby all that well, but I've been using Javascript for... three months... so am fairly versed. Every example I've seen violates what you are here saying. Let us simply assume the examples I've seen are just a bad use of "Cucumber".

    Of course you could implement your own test harness, but why would you want to do that, if the canonical solution (dejagnu) provides an excellent harness? Of course you could implement your tests using plain TCL, but in my experience this is a real pain to do, so why not use something simpler?
    I'm not saying "create your own test harness". (I'm not talking about building your own "dejagnu" application.) I'm talking about the expression of rules as built against an existing harness. (I'm talking about the "exp" file.) Of course as above, the examples I've seen with "Cucumber" may just be awful examples of using "Cucumber".

    What I wanted to avoid was manually writing TCL assertions.
    That is a fine goal in and of itself; rereading your original post, it seems as if you only intended to express the requirements of such assertions in a different form. (I read as saying, basically, what the traits the TCL code should express and ways to validate those tests.) If the goal is truly generative as you say, that is an entirely different beast. Referencing the above--if you intend to produce the "exp" file--or equivalent--directly from the "natural language" with no TCL coding required on the other side, that seems like a fine goal.

    Well, here one might surely argue that I'm biased; again, I despise TCL.

    Soma
    Last edited by phantomotap; 02-21-2014 at 04:31 AM. Reason: grue
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  5. #20
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    Quote Originally Posted by phantomotap View Post
    That is a fine goal in and of itself; rereading your original post, it seems as if you only intended to express the requirements of such assertions in a different form. (I read as saying, basically, what the traits the TCL code should express and ways to validate those tests.) If the goal is truly generative as you say, that is an entirely different beast. Referencing the above--if you intend to produce the "exp" file--or equivalent--directly from the "natural language" with no TCL coding required on the other side, that seems like a fine goal.

    Well, here one might surely argue that I'm biased; again, I despise TCL.

    Soma
    no no, you will still have to write your step definitions in TCL, but the formal language expectations in the .exp files allow massive reuse of these steps, because many tests share similar steps, and the most common steps are already provided e.g. `Then I should see "<text>'

  6. #21
    Unregistered User Yarin's Avatar
    Join Date
    Jul 2007
    Posts
    2,158
    Quote Originally Posted by graprof View Post
    Now, if you have ever done something with ruby on rails, you may know cucumber testing, which is an awesome way of writing natural language tests that are fun to write, easy to read and generally awesome all the way!

    I wanted to incorporate the fun cucumber-esque way of writing natural language tests into dejagnu, to have seamless integration into any autotools projects out there, to help developers find the fun in writing tests, and subsequently produce better software.

    To give an example, a simple test could look like this:

    Code:
      # test that `ls -a` lists the parent directory 
      Given an executable "ls" \
        when I run with parameters "-a" \
        then I should see ".."
    which is undeniably a very intuitive way to specify assertions.

    [snip]

    If you would like to collaborate because you are passionate about your testing, or just have any questions on why I am doing all this, or why natural language testing is cool, or about anything just let me know
    I don't agree with everything Edsger Dijkstra had to say, but I think he was spot on when he said "Projects promoting programming in natural language are intrinsically doomed to fail". But, natural language aside, my gut tells me a DSL for testing is always going to be more trouble that it's worth.

    Do you have a more complete cucumber test(s) that demonstrates an advantage over standard testing?

  7. #22
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    I have recently whipped up a data generator that pulls schema definitions from files or stdin, using a bison/flex parser, and have written a couple of assertions on the behaviour of the parser, here is an excerpt:

    Code:
    # test for conditions triggering -Wempty-schema
    #   should be enabled by default
    set test_name "\[-Wempty-schema\]"
    given an executable "../src/dbtesma" \
      when I run \
      and I send \x04 \
      then I should see "warning: no table definition" \
      when I run with parameters "-Wno-empty-schema" \
      and I send \x04 \
      then I should not see "warning: no table definition"
    
    
    # test for conditions triggering -Wempty-table
    #   should be enabled by default
    set test_name "\[-Wempty-schema\]"
    given an executable "../src/dbtesma" \
      when I run \
      and I send "table { name = \"test\" }\n\x04" \
      then I should see "warning: no column definitions in table `test'" \
      when I run with parameters "-Wno-empty-table" \
      and I send "table { name = \"test\" }\n\x04" \
      then I should not see "warning: no column definitions in table `test'" \
    
    
    # test for conditions triggering -Wconversion
    #   should be enabled by default
    set test_name "\[-Wconversion\]"
    given an executable "../src/dbtesma" \
      when I run \
      and I send "table { name = \"test\" rows = 999999999999999999999 }\n\x04" \
      then I should see "warning: `999999999999999999999': Numerical result out of range" \
      when I run with parameters "-Wno-conversion" \
      and I send "table { name = \"test\" rows = 999999999999999999999 }\n\x04" \
      then I should not see "warning: `999999999999999999999': Numerical result out of range"
    
    
    # test for warning elevation when -Werror is specified
    set test_name "\[-Werror\]"
    given an executable "../src/dbtesma" \
      when I run \
      and I send "table { name = \"test\" } table { }\n\x04" \
      then I should see "warning: no column definitions in table `test'" \
      and I should see "error: `table' definition missing required `name' property" \
      when I run with parameters "-Werror" \
      and I send "table { name = \"test\" } table { }\n\x04" \
      then I should see "warning: no column definitions in table `test'" \
      and I should not see "error: `table' definition missing required `name' property" \
      
    
    # test for warning suppression when -g is specified
    set test_name "\[-q\]"
    given an executable "../src/dbtesma" \
      when I run \
      and I send \x04 \
      then I should see "warning: no table definition" \
      when I run with parameters "-q" \
      and I send \x04 \
      then I should not see "warning: no table definition" \
      when I run with parameters "-q -Werror" \
      and I send \x04 \
      then I should see "warning: no table definition"
    By the way, I agree with Dijkstra on natural language processing, but this does not really apply here, as the test framework does not care about the interpretation and the semantics of natural language, which is the difficult problem he was probably refering to. Instead, the meaning of the steps is made explicit by defining them in TCL.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 2
    Last Post: 06-27-2013, 03:22 PM
  2. sum of n natural numbers
    By Monochrome in forum C Programming
    Replies: 6
    Last Post: 11-01-2011, 09:14 AM
  3. C and Natural Language Processing
    By hilarius in forum C Programming
    Replies: 1
    Last Post: 11-25-2009, 01:20 AM
  4. Natural Mergesort
    By wuzzo87 in forum C Programming
    Replies: 31
    Last Post: 04-14-2007, 09:41 PM
  5. Natural Language Parsing
    By edk in forum C++ Programming
    Replies: 7
    Last Post: 04-12-2002, 07:06 PM

Tags for this Thread