This is quite an interesting result ;-) If you are doing float maths and
don't call any higher level functions the results from the two test
cases _ought_ to be the same if both platforms impliment ieee-754.
However, there are some caveats once you call high level functions all
bets are off as there as there can be impementation/algorithm
dependancies (certainly the c standard on first reading says nothing
about what rounding sin etc use!..?). Now there are further complcations
as some compilers use fused multiply-add instructions and other
optimisation which are not coveredby ieee-754 and are not standardised
by ieee-754 in terms of their rounding behaviour....( except in C99
which has a pragma to control this sort of thing: FP_CONTRACT)
So here my thought. What we have here are regression tests so we either
1. define a set of results for each test on each particular platform
(you have a mode there someone can run the tests on a version we believe
works, and then say e-mails them to us for inclusion) We then store them
and then use those results only for that platform
2. define a s set of result for each test which encompases worst case
performance (as long as it is reasonable), run the tests on a variety of
platforms and if it fails on some platforms decide on a case by case
basis if the result is reasonable and downgrade you regression tests
till it works everywhere.
I would go for 2. its a lot easier to work with and much more likley to
be used by the user for testing their implementation
regards
gary
Edward d'Auvergne wrote:
On 10/20/06, Alexandar Hansen <viochemist@xxxxxxxxx> wrote:
First, I tried the test-suite again, and it failed (new failures this
time
though :) )
#############################
# Results of the test suite #
#############################
Updated to revision 2654:
The model-free tests:
Constrained BFGS opt, backtracking line search {S2=0.970, te=2048,
Rex=0.149} ....... [ Failed ]
Constrained BFGS opt, backtracking line search {S2=0.970, te=2048,
Rex=0.149} ....... [ Failed ]
Constrained Newton opt, GMW Hessian mod, backtracking line search
{S2=0.970, te=2048, Rex=0.149} [ Failed ]
Constrained Newton opt, GMW Hessian mod, More and Thuente line
search
{S2=0.970, te=2048, Rex=0.149 } [ Failed ]
I've deliberately started a new thread to talk about optimisation
tests in the test suite. This originates from the post located at
https://mail.gna.org/public/relax-devel/2006-10/msg00112.html
(Message-id:
<481156b20610190725ud6bab67w1f8fbbdf849da52c@xxxxxxxxxxxxxx>).
These are new optimisation tests I have added. The problem with
setting up these types of test is that machine precision and round-off
error causes slightly different optimisation results on different
systems (different numbers of iterations, function counts, gradient
counts, etc). The model-free parameters should be, to within machine
precision, the same each time. Things which may influence this are
the CPU type, Python version, Numeric version, underlying C library,
operating system, etc.
Would you have the text output of one of these tests? Can you see if
it is a model-free parameter or optimisation static causing the
problem? I've tried the tests on Windows and exactly the same tests
fail (excluding the third one). The problem is the chi-squared value
is ~1e-24 when ~1e-27 was expected. Optimisation terminates a little
earlier on Windows (less iterations of the optimisation algorithm).
I'm wondering if testing the optimisation statistics is worthwhile in
the test suite considering the variability?
Edward
_______________________________________________
relax (http://nmr-relax.com)
This is the relax-devel mailing list
relax-devel@xxxxxxx
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-devel
.
--
-------------------------------------------------------------------
Dr Gary Thompson
Astbury Centre for Structural Molecular Biology,
University of Leeds, Astbury Building,
Leeds, LS2 9JT, West-Yorkshire, UK Tel. +44-113-3433024
email: garyt@xxxxxxxxxxxxxxx Fax +44-113-2331407
-------------------------------------------------------------------
Re: Optimisation tests in the test suite.