On Fri, 2006-10-20 at 22:25 +1000, Edward d'Auvergne wrote:
The function tolerance between iterations is set to 1e-25 (I'll get to
the importance of this in my next post).
Is this the only termination criterion? If so, why are we seeing
apparently normal optimisations terminating with chi2 >> 1e-25 ?
I guess another possibly related question is why is this only happening
for the BFGS and Newton optimisation tests - is there something special
about these algorithms that makes chi2 poorly determined in these test
cases?
No, relax uses by default the function tolerance (default of 1e-25)
and the maximum number of iterations (default of 1e7). As the tests
in the test suite use noise-free relaxation data back-calculated from
known model-free parameter values, the chi-squared value in these
cases at the minimum is zero (to infinite precision!). However as
collected relaxation data is noisy, the chi-squared value should not
be zero (an exception is when the number of parameters of the model is
greater than or equal to the number of relaxation data points).
The failure of the BFGS and Newton tests is likely to be because of
the high level functions (Numeric/LAPACK matrix operations in this
case) mentioned by Gary in his post at
https://mail.gna.org/public/relax-devel/2006-10/msg00117.html
(Message-id: <4538955F.3090300@xxxxxxxxxxxxxxx>).
After looking more closely at the tests and their output, I realise that
the BFGS and Newton optimisation tests are the only ones failing with
this chi2 problem because they are the only ones setup to produce very
small chi2. The grid search gives finite chi2 for obvious reasons, and
in all of the other tests, optimisation is failing (because max_iter is
set too low, or with OverflowError). This is apparently the intended
behaviour in these tests because the tests pass even though the
optimisation failed. As chi2 never gets small in these tests, rounding
errors etc never become significant and all of these tests pass.
I note also that the BFGS and Newton tests are set up to test that the
number of iterations, function calls, etc, are 'correct'. In fact these
also depend on architecture/implimentation issues. These tests are not
evaluated because the chi2 test fails first.
Chris
The test is to be within
'value +/- value*error' where the error is 1e-8. This equation
removes the problem of the different scaling between the model-free
parameters (the 1e12 difference between S2 and te, etc.).
This is fine for finite values like S2 and te. The issue here is that
the expected value for chi2 in these tests is 0 (assuming a perfect
optimisation). It seems to me that the best way to ensure that
optimisation is behaving correctly in these cases is to test for the
following:
1) The optimised values are correct to within some tolerance (1e-8
relative error seems about right here)
2) Termination is normal. ie. the optimiser has thrown no errors or
warnings, and has not reached its maximum number of iterations.
Good point! This currently isn't tested. Errors are caught but
optimisation warnings (in self.relax.data.res[run][i].warning) aren't.
3) Chi2 is small ( <= 1e-20 seems about right based on the few values
reported so far, but something less restrictive might be required)
That test value should work ok for this noise-free data. Would you
like to have a go changing this value?
On reflection, it is probably worth having at least some tests where we
expect a finite chi2. Testing for that value then should be much easier
to deal with.
I could add some noisy synthetic data for the tests. For testing the
ensemble of algorithms and the myriad of combinations, I don't know if
this is worth the extra effort. It could be useful for testing the
default algorithms though.
Edward
Re: Optimisation tests in the test suite.