(responding to essentially the entire thread)
I would strongly prefer a separate directory tree of files to prevent
clutter. I've seen many projects take this approach where the tests
are dislocated from the main tree and very much prefer the isolation.
This is what I've done with the current function/system tests. If the
separation ever causes the tests to get out of sync with the main
tree, then these will instantly cause a failure of the test suite and
hence synchronisation between different directory trees won't be an
issue.
I've been following the Biopython project for a few years, and in my
mind this project is a reasonable example of how unit tests in python
can be used.
The Biopython tests are organised like:
Tests/test_Fred.py
with all tests in one directory, separate from the modules that are being tested, and using the naming scheme that Gary suggests.
They also hold any data required for input/comparison for the tests in subdirectories under the Tests directory, named module-wise like:
Tests/Fred/fred_input_data_001.dat
Tests/Fred/fred_input_data_002.dat
Biopython doesn't seem to have much in the way of the higher level system/regression-style tests, due to the nature of that project being a mostly a collection of small
modules, as far as I can tell they don't require testing of higher level 'protocols' which such as
relax uses.
As far as running the test suite (ie parsing directories for test_Fred.py files and running them) .. here's a link to the Biopython code that does just that (clean-room warning: link is to BSD-ish licensed code :) ):
http://cvs.biopython.org/cgi-bin/viewcvs/viewcvs.cgi/biopython/Tests/run_tests.py?rev=1.7&cvsroot=biopython&content-type=ext/vnd.viewcvs-markup
It even uses the unittestgui with the all-important 'green line', that goes red if any test fails :)
So, if I understand correctly, the plan is to have the test_suite with something like:
test_suite/unit_tests (for the real module-wise unit tests, using Pythons unittest module)
and
test_suite/system_tests (which is closer to what is in the current test_suite, and could be at the same high level as the users scripts in sample_scripts. Eventually the existing tests can be modified to use the unittest framework for verifying 'correctness' of operation).
Cheers,
Andrew
Re: unit tests vs... the current test_suite