Thread: Automatic Testing Strategies

  1. #1
    GA ichijoji's Avatar
    Join Date
    Nov 2002
    Posts
    179

    Automatic Testing Strategies

    Hey all,

    I'm designing an automated unit test system for my engine and it has rapidly become clear that there is a nearly unlimited number of ways to do it. I was wondering if anyone around here had implemented something like this and how your strategy worked.
    Illusion and reality become impartiality and confidence.

  2. #2
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    Which OS/Compiler ?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  3. #3
    GA ichijoji's Avatar
    Join Date
    Nov 2002
    Posts
    179
    Developing on gcc and fc5, but robust to msvc 8.0 as well.
    Illusion and reality become impartiality and confidence.

  4. #4
    and the hat of int overfl Salem's Avatar
    Join Date
    Aug 2001
    Location
    The edge of the known universe
    Posts
    39,660
    A couple of ideas - as you say, the possibilities are endless.

    - Use http://www.swig.org/ to expose key parts of your API to various scripting languages.

    - Using your gcc/fc5 platform, make lots of use of 'gcov' to keep an eye on the test coverage. Use this to guide you in creating new tests for the code other tests cannot reach.

    - Create a 'smoke test' application around your engine, designed to exercise most of the mainline code. It should be a short test (a few seconds), so it can be run often. It's used to spot early any major breakage.

    How are you going to measure "success"? Does your engine produce text files which can be compared with 'diff', or is it more complicated because you have lots of images?
    If you dance barefoot on the broken glass of undefined behaviour, you've got to expect the occasional cut.
    If at first you don't succeed, try writing your phone number on the exam paper.

  5. #5
    Crazy Fool Perspective's Avatar
    Join Date
    Jan 2003
    Location
    Canada
    Posts
    2,640
    If you want a good unit testing infrastructure, jake JUnit and copy the idea.

    You can also use something like Bicycle to automate the test running and generate pretty graphs.

  6. #6
    GA ichijoji's Avatar
    Join Date
    Nov 2002
    Posts
    179
    The scripting way seems good to me. Using my python interface this is what I have implemented since last night:

    The makefile copies the test driver file over to the active directory and runs it with the directory to search for test files in (../src) and the search pattern for those files (test_*.py), then when it's done it cleans up and comes back. From there, the test driver searches out all of the files that match that pattern in that directory and runs them. In each test file, I inherit a class from UnitTest and run the test. UnitTest.start searches for all the functions in the inherited class that start with a prefix (test_) and runs them, watching for failed assertions and exceptions, then reports the results.

    This is pretty handy because it means I can just put a test_whatever.py file next to the source file I'm testing, then put a class in there inherited from UnitTest with a bunch of test_thisThing functions and they will automatically run when I run make test.
    Last edited by ichijoji; 04-25-2007 at 10:47 AM.
    Illusion and reality become impartiality and confidence.

  7. #7
    Cat without Hat CornedBee's Avatar
    Join Date
    Apr 2003
    Posts
    8,895
    Boost.Test is a unit testing framework for C++. Whether it lends itself well to testing a game engine is a different question, but at least parts - e.g. math routines - could be tested.
    All the buzzt!
    CornedBee

    "There is not now, nor has there ever been, nor will there ever be, any programming language in which it is the least bit difficult to write bad code."
    - Flon's Law

  8. #8
    C++ Witch laserlight's Avatar
    Join Date
    Oct 2003
    Location
    Singapore
    Posts
    28,413
    My impression is that ichijoji is already using a unit test framework. If that is so, what framework are you using, ichijoji?
    Quote Originally Posted by Bjarne Stroustrup (2000-10-14)
    I get maybe two dozen requests for help with some sort of programming or design problem every day. Most have more sense than to send me hundreds of lines of code. If they do, I ask them to find the smallest example that exhibits the problem and send me that. Mostly, they then find the error themselves. "Finding the smallest program that demonstrates the error" is a powerful debugging tool.
    Look up a C++ Reference and learn How To Ask Questions The Smart Way

  9. #9
    GA ichijoji's Avatar
    Join Date
    Nov 2002
    Posts
    179
    Just the one I described before. It boils down to a simple unit testing system, but I'm not sure how I'm going to go about higher level testing.
    Illusion and reality become impartiality and confidence.

Popular pages Recent additions subscribe to a feed

Similar Threads

  1. automatic variables
    By FOOTOO in forum C Programming
    Replies: 5
    Last Post: 03-08-2005, 06:30 PM
  2. OpenGL: Depth Testing, Matrixes, ect.?
    By Zeusbwr in forum Game Programming
    Replies: 8
    Last Post: 12-08-2004, 09:37 AM
  3. C++ bit testing
    By Vicious in forum C++ Programming
    Replies: 3
    Last Post: 09-19-2004, 11:44 AM
  4. Automatic vs. Static Duration
    By JMB in forum C++ Programming
    Replies: 7
    Last Post: 10-14-2002, 07:16 PM
  5. Automatic Time in C++Builder ??
    By Gugge in forum Windows Programming
    Replies: 0
    Last Post: 06-18-2002, 12:58 PM