Re: Optimisation tests in the test suite. -- October 27, 2006 - 10:40

I've now committed all the changes.  Does anyone still have a system
in which the relax test suite fails on either the 1.2 or 1.3
repository lines?

Edward


On 10/27/06, Edward d'Auvergne <edward@xxxxxxxxxxxxx> wrote:
> On 10/26/06, Chris MacRaild <c.a.macraild@xxxxxxxxxxx> wrote:
> > I'm still failing on the grid search, both BFGS and the Newton
> > +backtracking tests (though the reasons for failure are new).
> >
> > For the grid search I have:
> >
> > Parameter values: array([ 1.,  0.,  0.])
> > Function value:   3.9844117908982244
> > Iterations:       1331
> > Function calls:   1331
> > Gradient calls:   0
> > Hessian calls:    0
> > Warning:          None
> >
> > The chi-squared value of '3.9844117908982244' should be within +/-
> > 9.9999999999999995e-21 of 3.9844117908982288.
>
> I'll change this one to the error tolerance of 1e-8 as you suggest below.
>
>
> > For both BFGS tests:
> >
> > Parameter values: array([  9.70000000e-01,   2.04800000e+03,
> > 1.49000000e-01])
> > Function value:   4.1720494200991728e-21
> > Iterations:       157
> > Function calls:   699
> > Gradient calls:   164
> > Hessian calls:    0
> > Warning:          None
> >
> > The te value of '2047.9999998929304' should be within +/- 1e-08 of
> > 2048.0.
>
> I'm surprised by this difference!  I might change the te scaling to
> 1e9 (nanoseconds) and leave the error tolerances as they are.  That
> will fix this issue.
>
>
> > And for the Newton+backtracking test:
> >
> > Parameter values: array([  9.70000000e-01,   2.04800000e+03,
> > 1.49000000e-01])
> > Function value:   7.5138787163096146e-28
> > Iterations:       18
> > Function calls:   96
> > Gradient calls:   23
> > Hessian calls:    18
> > Warning:          None
> >
> > The function count of '96.0' should be greater than 55.0.
>
> Now you are at a higher precision, the previous results were lower
> precision?  Bizarrely the iteration count is identical?  I'll use
> these values instead within the test!
>
>
> > My suggestions:
> >
> > It was quite appropriate to test finite values with a relative tolerance
> > of 1e-8. This is the test that should be applied for the grid search
> > chi2 and all finite parameter values.
>
> And the optimisation algorithms which have been terminated prematurely.
>
>
> > It is clearly not appropriate to test finite chi2 values to an absolute
> > tolerance of 1e-20. Rather, use the relative 1e-8 tolerance for finite
> > chi2 and a simple chi2 < 1e-20 test for values expected to be (close to)
> > zero.
>
> I've changed all the optimisation tests to do this.  I'll commit the
> changes tomorrow.
>
>
> > Obviously the Newton failure there is a simple typo
>
> Thanks, it's fixed.
>
> Edward