> I would like a clear distinction between the different classes of
> tests so that relax can generate a report in the end. I.e. all the
> unit tests are clustered into a single TestSuite (which can be
> composed of multiple TestSuite objects for grouping) and all the
> system/functional tests clustered into a single separate TestSuite
> object. Although the system/functional tests could possibly use the
> unittest framework, the unit tests and system tests don't have to be
> within the same framework.
>
No indeed and this is one of the thoughts we had. The TestSuite tree of
instances should be something like this
system_tests
|
--------------------------------------------
| | | |
model_selection jw_mapping ... unit_tests
|
-----------------------------
| | |
test_x test_y ...
(if the tree doesn't display you need to use a monospaced font)
I don't know if I would classify the unit tests as system tests though
(for example see http://en.wikipedia.org/wiki/Unit_testing and
http://en.wikipedia.org/wiki/System_testing). Instead I was thinking
of the following structure for the tests:
test suite:
unit tests:
test_relax.py
test 1
test 2
test_float.py
test 1
test 2
system / functional tests:
model-free:
reading:
test 1
test 2
optimisation:
test 1
test 2
reduced spectral density mapping:
test 1
test 2
The unit tests can run first, printing a message saying that the tests
passed. Then the system/functional tests can be run, printing a
similar message. For example if both pass the following could be
printed (with the unittest print out in between):
Unit tests ................. [ passed ]
Functional tests ........... [ passed ]
Test suite ................. [ passed ]
I.e. the first ideally tests if all parts of relax work on a function
by function level and the second ideally runs all types of analyses on
the system level. There's no need to print out a message for each of
the system tests. relax could however print out '[ OK ]' or '[ passed
]' messages for each category of unit and system tests if the
--test-suite command line flag is used together with --debug (or some
other flag).
> I envisage the system tests to work quite differently in that the
> instance 'self.relax' is passed into each test class (this is
> essential). Most of the automatic stuff in the unit tests will be
> unnecessary. For example functions such as
> 'get_module_relative_path()', etc. will be unnecessary.
Indeed this is true and my plan is to create a super class which just
did test discovery without having to find relative paths etc the root
path would just be provided
A base class placed in the base directory 'test_suite'?
(We talked about subclassing TestCase as well so it has an idea of it's
directory and the root test_suite directory to give tests places to find
data from)
That's probably the best way to do it. Maybe the modified TestCase
could be called RelaxTestCase?
> Actually on
> the subject of these functions, the ones not defined as class methods,
> they could all be shifted into the classes and accessed by typing
> 'self.get_module_relative_path()'. Apart from the print statements at
> the end of 'unit_test_runner.py', these functions are only accessed by
> the other class methods. These may as well be converted to an object
> oriented approach.
>
Now that is an interesting question: Should they go in the class? Most
probably. However, should they be virtual methods that can be overidden?
Almost definitley not in my opinion (overridden methods should always be
designed to be so) So I suggest they go in as static methods of the
class [this was my plan anyway I just have to get down to trying it]
also most of these methods should be private [they are not part of the
public interface])
That sounds like a really good setup. By static method, do you mean
using the @staticmethod function decorator? If so, is there a
benefit? And by virtual method do you mean a standard class method?
Making the methods private is ok by me. I haven't used private names,
but in the situation where someone is adding tests to the test suite -
this would be quite useful. For sanity, I would use the following
ordering of the class methods throughout all of relax. First would
come the special methods 'self.__xxx__()' ordered alphabetically with,
of course, 'self.__init__()' as the very first method. Then in
alphabetical order would be the private methods 'self.__xxx()'.
Finally, and again alphabetically ordered, would be the standard class
methods.
This leads to another question what is the minimum python version that
we are aiming for in the 1.3 branch?
(i.e. what facilities are available to us)
Which version do people think would be best? I would say 2.4. Would
you like to start a new thread to discuss this Gary? It's best that
everyone sees this discussion as it would probably be lost in this
thread. I'll respond to the rest in a second post as this is getting
too big.
Edward
Re: r2860 - /branches/test_suite/test_suite/unit_tests/unit_test_runner.py