I almost agree ;-)
Here are my definitions
unit tests: are a test that that tests a class or function at the
lowest level and are designed to tests the limits of the functions
inputs and that the function/class/methods obey the contract they have
been designed to. As far as possible such tests test a class in
isolation if possible (even down to the use of mock objects, but that is
another story). Often unit tests are used in combination with code
coverage tools to check that all pathways through the code have at least
be exercised....
regression tests: these are generally a set of tests that you write once
you have the programming working to test that after a modification it
still produces the correct results. However, regression tests are also
often used to test that a bug that has been found doesn't occur again.
As such regression tests are often unit tests as well.... (this is
called thre find a bug write a test philosophy ;-))
functional tests functional tests are generally large scale regression
tests. You use a set of known inputs for a complete program and check
that the outputs are as expected. If the functional test fails but the
failure doesn't trigger the failure of a unit test then the unit tests
need more work and the implimentation of unit tests...
On Mon, 2006-10-30 at 11:56 +0000, Gary S. Thompson wrote:
Hi Andrew
Unit tests have been on my radar as well as I have used them under
java. As ususal having done things the wrong way round I am now
implementing unit tests for the ieee float functions. Here is my suggestion:
0. use unittest, it is a tried and tested methodology
1. if you have a file fred.py it should be accompanied by a unit test
files test_fred.py with the correct main function
2. if you have test data to go with fred.py it should be in a directory
test_data_fred
3. modify the relax test suite to scan the source directories for files
with the correct names and apply the tests...
as a general note you ought to write the unit tests first ;-) before
coding if you want to stick to the 'true philosophy' of this methodology
so the idea is
1. suggest feature
2. define relevant unit tests
3. code unit tests
4. code feature
5. check / correct / add more unit tests as needed
regards
gary
It seems to me there are a couple of different issues here.
The first is would it be useful to have unit-tests for relax. From my
point of view the answer is a clear yes, and I agree with Gary's
suggestions regarding their organisation.
The second issue raised in Andrew's post is whether the Python unit-test
mechanisms are worth using to drive the current test-suite. Again, I
think the answer is yes: the test-suite is getting big and complex
enough to justify using this mechanism. It should make coding new tests
easier, and eliminate the need for much of the repeditive code in the
current suite.
I agree +1
As to why I think these are separate issues: the current test-suite
(indeed the entire relax code base) contains not one single unit-test.
Unit-tests, as their name implies, are intended to test the smallest
possible unit of code in isolation (a single function or method, at most
a single class). The type of test coded in the test-suite is much higher
level than this, and is better termed a regression test - it is intended
to test that all 'units' in the program are working together correctly.
A complete set of regression tests should catch all bugs in relax,
This can never be the case. Catching all bugs is generally beyond the
available time and effort limits for testsing, all you can do is check
that in most cases output is right and that you haven't regressed due to
changes...
but
might not reveal much about the nature of the bugs. On the other hand,
unit tests will miss bugs of regression, but should identify more detail
of the nature and location of the bugs they do catch.
again most regression ought to be caught as unit tests... see above
Both types of
testing are important in a working package like relax.
I think its important that the distinction remains clear between the
relax test-suite, which should be a reasonably exhaustive set of
regression tests, and the relax unit-tests which will remain incomplete
for the forseeable future. Gary's suggestion above is appropriate for
the organisation of unit-tests, but the test-suite I think properly
belongs as a seperate set of tests.
I agree but feel that the unit tests should be run as part of the test
suite... and if the two don't fail together you have a problem... also
is you run unit tests as part of the test suite it means the unit tests
get exercised on lots of machines architectures etc
All of that said, there is no reason why we can't use the Unittest
module as the testing mechanism for the test-suite.
definitley
Chris
.
regards
gary
--
-------------------------------------------------------------------
Dr Gary Thompson
Astbury Centre for Structural Molecular Biology,
University of Leeds, Astbury Building,
Leeds, LS2 9JT, West-Yorkshire, UK Tel. +44-113-3433024
email: garyt@xxxxxxxxxxxxxxx Fax +44-113-2331407
-------------------------------------------------------------------
Re: unit tests vs... the current test_suite