Re: Precision in the test-suite -- October 25, 2007 - 09:55

Hi,

As you noted, the precision differences in the test-suite are hard
coded into relax as tolerances.  I've tried to chase down the point of
entry of these tolerances for the reduced spectral density mapping
tests but cannot find anything in the commit messages in the
repository nor in the mailing list archives.  So I'll have to explain
their origin as best I can using my slowly fading memory :)  If anyone
else can remember these discussions and this precise example, please
say.

So, I was the one who introduced these specific tolerances.  This was
preceded by many discussions with Alex, Chris, and Gary in the mailing
lists about optimisation precisions on different architectures,
operating systems, etc.  See the thread starting at
https://mail.gna.org/public/relax-devel/2006-10/msg00114.html,
specifically the post at
https://mail.gna.org/public/relax-devel/2006-10/msg00139.html for a
summary.  Now when I coded these J(w) mapping tests I had, at the
time, access to 32 bit i686 GNU/Linux and 32 bit Windows.  The
tolerances were much smaller in my pre-committed code but when I
tested it on Windows, the test-suite failed!!!  This could be because
of the Windows versions of Python or Numeric (the likely cause), or
some other Windows specific issue.  So I set these tolerances with the
idea that for a final result, this type of precision will be
reasonable.  For calculations and results prior to the final result
truncation will cause round-off error, but in this case a difference
of 1e-6 in a manuscript for values on the order of 1 will not matter.
So in the end, this compromise was just to make Windows happy!

I no longer have access to 32 bit Windows, but will sometime soon
setup a relax development environment
(http://www.nmr-relax.com/windows_devel.html) for 64 bit Windows (I
had a feeling that I would one day have a use for the Windows install
that came with my laptop :).  I have a feeling, though, that your
patches will cause the test-suite to fail under Windows.  Sorry, I
should have mentioned this prior to you making these patches.

Regards,

Edward



On 10/22/07, Sebastien Morin <sebastien.morin.1@xxxxxxxxx> wrote:
>  Hi again...
>
>  Here are two patches if the changes I proposed for the precision in the
> jw_mapping and consistency tests test-suites were to be accepted as
> relevant...
>
>  1. patch_precision_in_test-suite_1.2-r3370_jw_mapping
>
>      For jw_mapping in the main branch.
>
>  2.
> patch_precision_in_test-suite_1.2-r3370_consistency_tests
> (consistency tests branch)
>
>      For consistency tests in the consistency tests branch.
>
>  If not relevant, just discard the files..
>
>
>  Cheers
>
>
>  Séb  :)
>
>
>
>
>
>
>
>  Sebastien Morin wrote:
>  Hi again
>
>  After a look in the test_suite/consistency.py and test_suite/jw_mapping.py
> files, I realized that the 6th decimal precision behaviour was directly
> encoded.
>
>  At line ~80 in test_suite/consistency.py :
>
>                  if abs(self.relax.data.res[self.run][index].j0 - j0[index])
> > j0[index]/1e6:
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'j0 calculated value'
>                      return
>                  if abs(self.relax.data.res[self.run][index].f_eta -
> f_eta[index]) > f_eta[index]/1e6:
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'f_eta calculated value'
>                      return
>                  if abs(self.relax.data.res[self.run][index].f_r2 -
> f_r2[index]) > f_r2[index]/1e6:
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'f_r2 calculated value'
>                      return
>
>  The code specifies that the difference between the calculated value and the
> reference one should not be ore than a ppm of the reference value... I
> understand that this is to prevent differences in machine precision from
> affecting the test, but the 1/1000000 factor seems big and I feel machine
> rpecision is a lot more...
>
>  I tried (on my 32 bit x86 Linux machine) to change the code and see its
> behaviour and it seems that we could simplify this part of code to render it
> more logic and stringent :
>
>                  if abs(self.relax.data.res[self.run][index].j0 !=
> j0[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'j0 calculated value'
>                      return
>                  if abs(self.relax.data.res[self.run][index].f_eta !=
> f_eta[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'f_eta calculated value'
>                      return
>                  if abs(self.relax.data.res[self.run][index].f_r2 !=
> f_r2[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'f_r2 calculated value'
>                      return
>
>  With that change, the reference values could be erroneous to the 16th
> decimal without affecting the test. Thus, the 15th decimal is now critical,
> which is more like the other values I pointed in my first post (which must
> be related to machine precision and portability between architecture, OS,
> scientific libraries, etc.
>
>  Sorry if, again, I talk this point (We talked about this in July :
> https://mail.gna.org/public/relax-devel/2007-07/msg00050.html),
> but this seems important to me as other reference values are tested in a
> much less relaxed way...
>
>  Cheers
>
>
>  Séb
>
>
>
>
>
>
>
>  Sebastien Morin wrote:
> Hi all
>
>  As I was verifying that the consistency tests code is ready to be
> incorporated into the main stream, I found out some strange behaviour in the
> test suite (1.2, r3370, main stream and consistency tests branch).
>
>  This behaviour has something to do with, maybe, the precision of numbers we
> use in relax or maybe also machine precision (two topics I'm really not
> familiar with)...
>
>  1. test_suite/consistency_tests.py (consistency tests branch)
>
>
>  If someone changes the following lines (line ~70) :
>
>          # Correct consistency functions values:
>          j0 = [4.0958793960056238e-09, 3.7976266046729745e-09]
>          f_eta = [0.35164988964635652, 0.32556427866911447]
>          f_r2 = [2.0611470814962761e-09, 1.9117396355237641e-09]
>
>  for
>
>          # Correct consistency functions values:
>          j0 = [1.00001*4.0958793960056238e-09,
> 1.00001*3.7976266046729745e-09]
>          f_eta = [1.00001*0.35164988964635652, 1.00001*0.32556427866911447]
>          f_r2 = [1.00001*2.0611470814962761e-09,
> 1.00001*1.9117396355237641e-09]
>
>  The test-suite will not pass. However, if someone changes the lines so they
> become :
>
>          # Correct consistency functions values:
>          j0 = [1.000001*4.0958793960056238e-09,
> 1.000001*3.7976266046729745e-09]
>          f_eta = [1.000001*0.35164988964635652,
> 1.000001*0.32556427866911447]
>          f_r2 = [1.000001*2.0611470814962761e-09,
> 1.000001*1.9117396355237641e-09]
>
>  The test-suite will pass. So what's the real change in the values..? Let's
> take the first (left) value for f_eta.
>
>          FAIL : 0.35164988964635652 * 1.00001  = 0.3516534061452529835652
>          PASS : 0.35164988964635652 * 1.000001 = 0.35165024129624616635652
>
>  The real change here happens at the 6th decimal. I'm not really familiar
> with machine precision and how it's dealt with, but this 6th decimal
> shouldn't be limiting...
>
>
>
>
>  2. test_suite/jw_mapping.py (main stream)
>
>
>  The exact same behaviour applies for the values of j0, jwx and jwh in that
> file. This is quite normal has the code for consistency testing was inspired
> by the jw mapping code...
>
>
>
>
>  3. test_suite/model_free.py (main stream)
>
>
>  If someone changes the following lines (line ~910) :
>
>          # Load the relaxation data.
>          self.relax.interpreter._Relax_data.read(self.run,
> 'R1', '600', 600.0 * 1e6, 'r1.600.out', dir=path)
>
>  for :
>
>          # Load the relaxation data.
>          self.relax.interpreter._Relax_data.read(self.run,
> 'R1', '600', 1.00000000000001*600.0 * 1e6, 'r1.600.out', dir=path)
>
>  The test-suite won't pass. However, with :
>
>          # Load the relaxation data.
>          self.relax.interpreter._Relax_data.read(self.run,
> 'R1', '600', 1.000000000000001*600.0 * 1e6, 'r1.600.out', dir=path)
>
>  The test-suite will pass. So what's the real change in the values..?
>
>          FAIL : 1.00000000000001  * 600.0 = 600.000000000006  =
> 6.00000000000006  * 10^2
>          PASS : 1.000000000000001 * 600.0 = 600.0000000000006 =
> 6.000000000000006 * 10^2
>
>  Here, the change is seen at the 14th decimal... Should this be limiting in
> our calculations..?
>
>
>  For the diffusion tensor (line ~920), the 15th decimal seems critical as
> the following won't pass :
>
>          # Setup other values.
>
> self.relax.interpreter._Diffusion_tensor.init(self.run,
> 1.000000000000001*10e-9, fixed=1)
>          self.relax.interpreter._Value.set(self.run, 1.02 *
> 1e-10, 'bond_length')
>          self.relax.interpreter._Value.set(self.run, -160 *
> 1e-6, 'csa')
>
>  and the following will :
>
>          # Setup other values.
>
> self.relax.interpreter._Diffusion_tensor.init(self.run,
> 1.0000000000000001*10e-9, fixed=1)
>          self.relax.interpreter._Value.set(self.run, 1.02 *
> 1e-10, 'bond_length')
>          self.relax.interpreter._Value.set(self.run, -160 *
> 1e-6, 'csa')
>
>
>
>  For the r value (line ~920), the 13th decimal seems critical as the
> following won't pass :
>
>          # Setup other values.
>
> self.relax.interpreter._Diffusion_tensor.init(self.run,
> 10e-9, fixed=1)
>          self.relax.interpreter._Value.set(self.run,
> 1.0000000000001*1.02 * 1e-10, 'bond_length')
>          self.relax.interpreter._Value.set(self.run, -160 *
> 1e-6, 'csa')
>
>  and the following will :
>
>          # Setup other values.
>
> self.relax.interpreter._Diffusion_tensor.init(self.run,
> 10e-9, fixed=1)
>          self.relax.interpreter._Value.set(self.run,
> 1.00000000000001*1.02 * 1e-10, 'bond_length')
>          self.relax.interpreter._Value.set(self.run, -160 *
> 1e-6, 'csa')
>
>
>  For the CSA value (line ~920), the 12th decimal seems critical as the
> following won't pass :
>
>          # Setup other values.
>
> self.relax.interpreter._Diffusion_tensor.init(self.run,
> 10e-9, fixed=1)
>          self.relax.interpreter._Value.set(self.run, 1.02 *
> 1e-10, 'bond_length')
>          self.relax.interpreter._Value.set(self.run,
> 1.000000000001*-160 * 1e-6, 'csa')
>
>  and the following will :
>
>          # Setup other values.
>
> self.relax.interpreter._Diffusion_tensor.init(self.run,
> 10e-9, fixed=1)
>          self.relax.interpreter._Value.set(self.run, 1.02 *
> 1e-10, 'bond_length')
>          self.relax.interpreter._Value.set(self.run,
> 1.0000000000001*-160 * 1e-6, 'csa')
>
>
>
>  Ok... So what's the point with that ? Is it something we should care about
> ? Or is it insignificant..? Will this behaviour in precision affect the rest
> of the calculations ? Why is there variation in the way values are affected
> ?
>
>  As I said, I'm really not familiar with precision stuff... So, if my
> question is naive, don't worry with it, I'm sorry...
>
>  Cheers
>
>
>  Séb
>  --
>  ______________________________________
>  _______________________________________________
>  | |
>  || Sebastien Morin ||
>  ||| Etudiant au PhD en biochimie |||
>  |||| Laboratoire de resonance magnetique nucleaire ||||
> ||||| Dr Stephane Gagne |||||
>  |||| CREFSIP (Universite Laval, Quebec, CANADA) ||||
>  ||| 1-418-656-2131 #4530 |||
>  || ||
>  |_______________________________________________|
>  ______________________________________
>
>  ________________________________
>
> _______________________________________________
> relax (http://nmr-relax.com)
>
> This is the relax-devel mailing list
> relax-devel@xxxxxxx
>
> To unsubscribe from this list, get a password
> reminder, or change your subscription options,
> visit the list information page at
> https://mail.gna.org/listinfo/relax-devel
>
>
>  --
>  ______________________________________
>  _______________________________________________
>  | |
>  || Sebastien Morin ||
>  ||| Etudiant au PhD en biochimie |||
>  |||| Laboratoire de resonance magnetique nucleaire ||||
> ||||| Dr Stephane Gagne |||||
>  |||| CREFSIP (Universite Laval, Quebec, CANADA) ||||
>  ||| 1-418-656-2131 #4530 |||
>  || ||
>  |_______________________________________________|
>  ______________________________________
>
>
>  --
>  ______________________________________
>  _______________________________________________
>  | |
>  || Sebastien Morin ||
>  ||| Etudiant au PhD en biochimie |||
>  |||| Laboratoire de resonance magnetique nucleaire ||||
> ||||| Dr Stephane Gagne |||||
>  |||| CREFSIP (Universite Laval, Quebec, CANADA) ||||
>  ||| 1-418-656-2131 #4530 |||
>  || ||
>  |_______________________________________________|
>  ______________________________________
>
>
> Index: test_suite/jw_mapping.py
> ===================================================================
> --- test_suite/jw_mapping.py    (revision 3370)
> +++ test_suite/jw_mapping.py    (working copy)
> @@ -79,13 +79,13 @@
>                      print 'Residue',
> self.relax.data.res[self.run][index].num, 'unexpectedly not selected'
>                      return
>
> -                if abs(self.relax.data.res[self.run][index].j0 - j0[index])
> > j0[index]/1e6:
> +                if abs(self.relax.data.res[self.run][index].j0 !=
> j0[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'j0 calculated value'
>                      return
> -                if abs(self.relax.data.res[self.run][index].jwh -
> jwh[index]) > jwh[index]/1e6:
> +                if abs(self.relax.data.res[self.run][index].jwh !=
> jwh[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'jwh calculated value'
>                      return
> -                if abs(self.relax.data.res[self.run][index].jwx -
> jwx[index]) > jwx[index]/1e6:
> +                if abs(self.relax.data.res[self.run][index].jwx !=
> jwx[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'jwx calculated value'
>                      return
>
>
> Index: test_suite/consistency_tests.py
> ===================================================================
> --- test_suite/consistency_tests.py     (revision 3370)
> +++ test_suite/consistency_tests.py     (working copy)
> @@ -81,13 +81,13 @@
>                      print 'Residue',
> self.relax.data.res[self.run][index].num, 'unexpectedly not selected'
>                      return
>
> -                if abs(self.relax.data.res[self.run][index].j0 - j0[index])
> > j0[index]/1e6:
> +                if abs(self.relax.data.res[self.run][index].j0 !=
> j0[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'j0 calculated value'
>                      return
> -                if abs(self.relax.data.res[self.run][index].f_eta -
> f_eta[index]) > f_eta[index]/1e6:
> +                if abs(self.relax.data.res[self.run][index].f_eta !=
> f_eta[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'f_eta calculated value'
>                      return
> -                if abs(self.relax.data.res[self.run][index].f_r2 -
> f_r2[index]) > f_r2[index]/1e6:
> +                if abs(self.relax.data.res[self.run][index].f_r2 !=
> f_r2[index]):
>                      print 'Error in residue',
> self.relax.data.res[self.run][index].num, 'f_r2 calculated value'
>                      return
>
>
> _______________________________________________
> relax (http://nmr-relax.com)
>
> This is the relax-devel mailing list
> relax-devel@xxxxxxx
>
> To unsubscribe from this list, get a password
> reminder, or change your subscription options,
> visit the list information page at
> https://mail.gna.org/listinfo/relax-devel