Thread: asparagus - natural language testing based on dejagnu/TCL

  1. #1
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9

    asparagus - natural language testing based on dejagnu/TCL

    Hello There!

    Thanks for checking out this cheap plug of mine.
    My name is Andreas Grapentin, and I am most (maybe a little) known as the creator of the open source C memory and runtime profiler graprof.

    As a dedicated C developer, I like to check out new and traditional tools to do stuff that I need done, and during this process I tend to stick with the tools that are mature and work well, which is why I use autotools as build system for all my projects.
    As you may, or may not be aware, the canonical solution for writing tests with autotools is dejagnu, wich is a testing framework based on TCL. Unfortunately, the dejagnu documentation is not good enough to provide the smooth learning curve desired by todays developers, which is why it is hugely underrated nowadays.

    Now, if you have ever done something with ruby on rails, you may know cucumber testing, which is an awesome way of writing natural language tests that are fun to write, easy to read and generally awesome all the way!

    I wanted to incorporate the fun cucumber-esque way of writing natural language tests into dejagnu, to have seamless integration into any autotools projects out there, to help developers find the fun in writing tests, and subsequently produce better software.

    To give an example, a simple test could look like this:

    Code:
      # test that `ls -a` lists the parent directory 
      Given an executable "ls" \
        when I run with parameters "-a" \
        then I should see ".."
    which is undeniably a very intuitive way to specify assertions.

    To enable developers to do all that, I created asparagus on github, which basically is a list of dirty hacks in TCL that processes statements as the example above into a list of executed steps implemented in TCL.

    Currently, asparagus is in a state where it just started to work, and I am starting to integrate it into my C projects to figure out where this could possibly go, but I would like to improve asparagus to a point that it is easy and pleasurable for developers to use.
    For that I need people to actually use it, to tell me what I can do to improve it, or to just flat-out send me pull requests with improvements.

    If you would like to collaborate because you are passionate about your testing, or just have any questions on why I am doing all this, or why natural language testing is cool, or about anything just let me know

    cheers,
    Andy

  2. #2
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Code:
    # test that `ls -a` lists the parent directory
    Given an executable "ls" \
      when I run with parameters "-a" \
      then I should see ".."
    O_o

    Why in all that is Holy/Unholy would you ever want that? Fun? That looks incredibly tedious as a testsuite entry for a language as expressive as Ruby. I'd expect more `"ls".with("-a").equals("..")' or something equally packed and expressive.

    *shrug*

    In any event, I do wonder why you aren't using the existing "parser" from "Cucumber". Are you writing in TCL just for the sake of compatibility with existing "dejagnu"?

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  3. #3
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    Quote Originally Posted by phantomotap View Post

    O_o

    Why in all that is Holy/Unholy would you ever want that? Fun? That looks incredibly tedious as a testsuite entry for a language as expressive as Ruby. I'd expect more `"ls".with("-a").equals("..")' or something equally packed and expressive.

    *shrug*

    In any event, I do wonder why you aren't using the existing "parser" from "Cucumber". Are you writing in TCL just for the sake of compatibility with existing "dejagnu"?

    Soma
    The initial idea behind cucumber was to enable quality assurance engineers and product owners to specify assertions about the product in natural language, because these people usually don't speak programming languages very well, but you needed a standardized formal means of communication between these people and developers. These natural language assertions were then mapped to ruby-implemented steps via regexes, and the developers implemented these steps, until the assertion passed, in which case, if the test definition was correct and complete, the feature was done.

    However, writing tests that way for small projects also has a bunch of advantages, let's look at a direct comparison

    your proposition was:
    Code:
    "ls".with("-a").equals("..")
    and my proposition was:
    Code:
    # test that `ls -a` lists the parent directory
    Given an executable "ls" \
      when I run with parameters "-a" \
      then I should see ".."
    On first glance, your version is very compact, mine is verbose. I give you that. But what is the cost of a byte of code? Not to mention that an equivalent example in TCL would be about twice as large.

    Also, think about the symbol to character ratio - which is a good metric for maintainability. Imagine the tests getting more complex, and then image a new developer coming into the project, trying to figure out what your tests are doing.

    And on the topic of new developers, if your test coverage is good, and you have natural language tests, then new developers can skim over your tests and easily understand the key features of your project.

    Lastly, think about the verbosity of natural language test as a tool, not a burden. Natual language testing enforces you to write a sentence per feature, which requires a lot a lot a lot of clarity and understanding, in my opinion this is very powerful.

    So, all things considered, natural language is going to be the way to go if you want to do testing. It is easy to understand, easy to scale, easy to maintain, programming language independent, and so on. The verbosity that comes with it is really not that much of a deal.

    Your question made me wonder - do you have a computer science background? If not, how long have you been programming?

    Thanks
    Andy

  4. #4
    misoturbutc Hodor's Avatar
    Join Date
    Nov 2013
    Posts
    1,787
    I'm pretty sure you're missing open mailbox, take letter, read letter, go east, go north, go east, open window, enter window, move rug, open trapdoor, go west, take lantern, go east, go up, light lantern, take sack, go down, turn off lantern, enter trapdoor, take sword...

  5. #5
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    I'm pretty sure you're missing open mailbox, take letter, read letter, go east, go north, go east, open window, enter window, move rug, open trapdoor, go west, take lantern, go east, go up, light lantern, take sack, go down, turn off lantern, enter trapdoor, take sword...
    O_o

    I'm so very pleased just now your in danger of being abducted.

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  6. #6
    misoturbutc Hodor's Avatar
    Join Date
    Nov 2013
    Posts
    1,787
    Quote Originally Posted by phantomotap View Post
    O_o

    I'm so very pleased just now your in danger of being abducted.

    Soma
    mwahahahaaha! Light lantern! I am safe again. No grue for you.

  7. #7
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    But what is the cost of a byte of code?
    O_o

    Are you next going to speak of "LoC"?

    Also, think about the symbol to character ratio - which is a good metric for maintainability.
    The answer of "what is more maintainable" depends on faculty for and experience with a language, the idiomatic expressions of a given idea, and the conformance and consistency of code with respect to those standards.

    And on the topic of new developers, if your test coverage is good, and you have natural language tests, then new developers can skim over your tests and easily understand the key features of your project.
    If they don't understand an expression of "acceptance testing" within a harness, they can't hope to understand any of my other code.

    Lastly, think about the verbosity of natural language test as a tool, not a burden.
    I absolutely refuse to accept verbose code as anything other than a burden.

    I absolutely refuse to accept terse code as anything other than a burden.

    Natual language testing enforces you to write a sentence per feature, which requires a lot a lot a lot of clarity and understanding, in my opinion this is very powerful.
    Your only valid response to my query is "The tool exists to allow non-programmers to express required traits.". That is a fine goal as far as it goes, but the tool can't force you to use it everywhere you should. Dedication to quality comes from people; if the developers are devoted enough to create the kind of harness you reference, it literally doesn't matter what tools are used.

    So, all things considered, natural language is going to be the way to go if you want to do testing.
    As with everything else in the programming community, some people are going to love the idea while some loath the idea.

    The verbosity that comes with it is really not that much of a deal.
    Code need not be verbose or terse. You made a point to say that the TCL version would require more code; that is an irrelevant metric. The question is "What code is necessary to meaningful express an idea without being superfluous or monotonous?".

    Code:
    int i;
    if(/**/)
    {
        i = /**/;
    }
    else
    {
        i = /**/;
    }
    The expression of the idea looks perfectly natural in C, but maybe the idea is more complex than the elided code implies.

    Code:
    int i = GetValue(/**/);
    The expression of the idea looks perfectly natural in C, but maybe the idea is simpler than the function call implies.

    Code:
    i = if whatever?
            getSomething()
        else
            getSomethingElse()
        end
    The expression of the idea looks perfectly natural in Ruby, but maybe the ideas expressed aren't as distinct as the code implies.

    Code:
    i = some_value || some_other_value
    The expression of the idea looks perfectly natural in Ruby, but maybe the values referenced aren't as simple as the code implies.

    Code:
    MyClass s;
    if(/**/)
    {
        s = /**/;
    }
    else
    {
        s = /**/;
    }
    The expression of the idea looks perfectly natural in C++, but maybe the classes aren't cheap to mutate.

    Code:
    SmartPointer<MyClass> s;
    if(/**/)
    {
        s = new MyClass(/**/);
    }
    else
    {
        s = new MyClass(/**/);
    }
    The expression of the idea looks perfectly natural in C++, but maybe the class uses a "pimpl".

    Given the right circumstances, I can see the "natural language" expression being perfectly natural. However, I've looked at "Cucumber" over Javascript. (I am far more familiar with Javascript.) I see only combinations of Javascript with "natural language"; I see everything "said" about each component of "acceptance testing" being stated in multiple forms; I see only tedious code.

    You are correct in claiming that people who don't C++ will not understand my C++ harness, but I only have to "say" things one time, and I get to say them with C++. You say the "natural language" version "is going to be the way to go"? Let me tell you, most of my C++ harness allows one to write "firewalls" preventing the code from ever being compiled if "acceptance testing" fails.

    Your question made me wonder - do you have a computer science background? If not, how long have you been programming?
    o_O

    I've been programming for about three months...

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  8. #8
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    Quote Originally Posted by phantomotap View Post

    I've been programming for about three months...

    Soma
    O_o

    you must be joking, right?

  9. #9
    misoturbutc Hodor's Avatar
    Join Date
    Nov 2013
    Posts
    1,787
    Quote Originally Posted by phantomotap View Post
    I've been programming for about three months...
    Trickster! You told me yesterday that you'd been programming since January; that's less than 3 months! I cannot be fooled so easily because I have a cunning eye for details like this.

  10. #10
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Light lantern!
    > Your lantern has run out of fuel.
    > It is pitch black. You might get eaten by a grue.

    Soma
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  11. #11
    Master Apprentice phantomotap's Avatar
    Join Date
    Jan 2008
    Posts
    5,108
    Trickster! You told me yesterday that you'd been programming since January; that's less than 3 months!
    ^____________^

    Soma, Soma. Soma!
    “Salem Was Wrong!” -- Pedant Necromancer
    “Four isn't random!” -- Gibbering Mouther

  12. #12
    misoturbutc Hodor's Avatar
    Join Date
    Nov 2013
    Posts
    1,787
    Quote Originally Posted by phantomotap View Post
    > Your lantern has run out of fuel.
    > It is pitch black. You might get eaten by a grue.

    Soma
    crap

  13. #13
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    Quote Originally Posted by phantomotap View Post
    ^____________^

    Soma, Soma. Soma!
    I refuse to be tricked into underestimating you.

    I think it is sad that most people from a traditional programming background are a bit biased against natural language testing. I am not even sure why that is, the only valid reason you have brought up against it is its verbosity, which I still think is negligible.
    But maybe my optimization goals in code quality are just different than yours.

    Cheers,
    Andy

  14. #14
    misoturbutc Hodor's Avatar
    Join Date
    Nov 2013
    Posts
    1,787
    Quote Originally Posted by graprof View Post
    I refuse to be tricked into underestimating you.

    I think it is sad that most people from a traditional programming background are a bit biased against natural language testing. I am not even sure why that is, the only valid reason you have brought up against it is its verbosity, which I still think is negligible.
    But maybe my optimization goals in code quality are just different than yours.

    Cheers,
    Andy
    I can't speak for others, but in my opinion it's (natural language testing) kind of strange. I seriously don't want people with no programming background writing tests for any project I'm working on -- it's hard enough as it is convincing those type of people that what they're asking is not as easy as it sounds. Despite that I do like the idea, in general. It's not going to be as natural language as you like though (unless you're hiding something that the world would really like to know about), so it's going to be "formalised" natural language. If the system can parse "given n elements of s I expect that the distance between values of n for the function f(n) will return a derivation of f() as a string" then I'll be impressed.

    As part of that formalisation couldn't brevity also be a goal?

  15. #15
    Registered User
    Join Date
    Sep 2013
    Location
    Germany
    Posts
    9
    Quote Originally Posted by Hodor View Post
    I can't speak for others, but in my opinion it's (natural language testing) kind of strange. I seriously don't want people with no programming background writing tests for any project I'm working on -- it's hard enough as it is convincing those type of people that what they're asking is not as easy as it sounds. Despite that I do like the idea, in general. It's not going to be as natural language as you like though (unless you're hiding something that the world would really like to know about), so it's going to be "formalised" natural language. If the system can parse "given n elements of s I expect that the distance between values of n for the function f(n) will return a derivation of f() as a string" then I'll be impressed.

    As part of that formalisation couldn't brevity also be a goal?
    you are right, it's formalised NL. valid expressions are expected to start with a `given' step, followed by a bunch of `when's `and's and `then's. The syntax of these steps is up to the developers though, as long as they implement the used steps in TCL.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. Replies: 2
    Last Post: 06-27-2013, 03:22 PM
  2. sum of n natural numbers
    By Monochrome in forum C Programming
    Replies: 6
    Last Post: 11-01-2011, 09:14 AM
  3. C and Natural Language Processing
    By hilarius in forum C Programming
    Replies: 1
    Last Post: 11-25-2009, 01:20 AM
  4. Natural Mergesort
    By wuzzo87 in forum C Programming
    Replies: 31
    Last Post: 04-14-2007, 09:41 PM
  5. Natural Language Parsing
    By edk in forum C++ Programming
    Replies: 7
    Last Post: 04-12-2002, 07:06 PM

Tags for this Thread