TextTest

Open-source framework for automating AcceptanceTests, available from SourceForge.

The approach is "text-based": i.e. based on the age-old idea of regression-testing by comparing the text produced by a program against a "gold standard". And, if you don't have any suitable text, of creating extra text output for this purpose.

Sound like a ten-line shell script will do the job? We (Carmen Systems) started with such a script in 2001. But test data requires book-keeping, and then you want a good dialogue for handling when a test fails, and then you write 100 tests and discover that you need to maintain all that data in some sensible way, and then you want to be able to batch-run the tests overnight and get the results posted somewhere in the morning. Then you want performance testing and integration with load-balancing tools like LSF and Sun Grid Engine. And then you realise you need something a bit more powerful.

So we rewrote it in Python and released it. It's been used to test a wide variety of programs, from our own resource optimizers (see AcceptanceTestOnlyDevelopment), to itself (naturally), to various database-facing tools written at Carmen, to Astra Zeneca's various molecule analysis tools. The only extensive testing on GUIs that I know of has been on its own GUI, but this went smoothely enough to recommend it for GUIs as well.

And of course, being text-based, it's independent of what language you write your program in, and is more truly acceptance testing than an approach that tries to "get under the skin" of the program and test its component classes. There is no test code, so no possibility that that can itself contain bugs or be testing the wrong thing.

Download from http://sourceforge.net/projects/texttest. There is not much more information on SourceForge, you need to actually download the tarball to find out more.

-- GeoffBache