Re: r22222 - /trunk/specific_analyses/relax_disp/disp_data.py -- February 21, 2014 - 11:13

Oh, I should mention that the ordering as mentioned in the
specific_analyses.relax_disp.disp_data module docstring is very
important:

"""
Ordering of data
================

The dispersion data model is based on the following concepts, in order
of importance:

    - 'exp', the experiment type,
    - 'spin', the spins of the cluster,
    - 'frq', the spectrometer frequency (if multiple field data is present),
    - 'offset', the spin-lock offsets,
    - 'point', the dispersion point (nu_CPMG value or spin-lock nu1
field strength),
    - 'time', the relaxation time point (if exponential curve data has
been collected).
"""

The reason for having the time point last is that one dispersion point
can have multiple time points (for the exponential curve).  And these
time point can be different for different curves, for example if the
HEHAHA type experiments are used to measure single peaks.

This ordering means that certain loop_*() functions are dependent on
the elements above them.  For example loop_frq_point() and
loop_frq_offset() require the experiment type to be supplied, to only
pull out the dispersion points and offsets corresponding to that
experiment.

There are some weak assumptions in these loop_*() functions, for
example it is assumed that for each experiment type, the list of
spectrometer frequencies are always the same (see how loop_frq() does
not ask for the experiment type).  It will be one of these weak
assumptions that is causing the bug you see
(http://gna.org/bugs/?21665), and the unit tests will allow you to
locate the exact function where the failure occurs.  Unit tests are
very useful when you cannot work out the location in the code where
the failure is occurring.  I hope this helps.

Regards,

Edward


P. S.  If the code in the specific_analyses.relax_disp.disp_data
module was perfect, there would be a loop_*() function for every
single data combination above, and they would all take {exp_type, frq,
offset, point, time} values as arguments to restrict the looping to
match the corresponding data.  However such complete matching would
cause the dispersion analysis to be orders of magnitude slower!  This
is the reason why there is an ordering - simply to make assumptions to
speed up the loop_*() functions and hence the whole dispersion
analysis.



On 21 February 2014 10:37, Edward d'Auvergne <edward@xxxxxxxxxxxxx> wrote:
> Hi,
>
> I saw this, and thought that the solution you found for fixing the
> unit test was too simple.  I think that if you carefully look at what
> the loop_*() function in your unit test returns, you will see that
> there will still be strange metadata combinations.  To simplify
> testing, the core of the unit test can be put into a relax script:
>
> """
> from specific_analyses.relax_disp.disp_data import
> loop_exp_frq_offset_point_time
>
> # Load the state.
> statefile = status.install_path +
> sep+'test_suite'+sep+'shared_data'+sep+'dispersion'+sep+'bug_21665.bz2'
> state.load_state(statefile, force=True)
> for exp_type, frq, offset, point, time, ei, mi, oi, di, ti in
> loop_exp_frq_offset_point_time(return_indices=True):
>     print(exp_type, frq, offset, point, time, ei, mi, oi, di, ti)
> """
>
> You could also try each of the loop_() functions one by one and see
> what they yield.  It should be logical what the results should be for
> the data in your save file.  It should be exactly what is shown in the
> GUI (open the 'bug_21665.bz2' file in the GUI and associate it with a
> new analysis).  For testing, you could create a list of lists of the
> values in the 'Spectra list' GUI element and use that in the unit
> tests, checking all elements of the loop for correctness.  You should
> also check that the number of iterations of the loop is correct (by
> maybe running the loop once first to count all, and then running a
> second time to check the values).
>
> To pin-point the problem, I would recommend creating a few more unit
> tests.  These can be easily created by copy-and-paste.  The first
> target would be the loop_exp() function (so the test could be called
> test_loop_exp()).  From your data in the GUI, this should return a
> single value of 'SQ CPMG'.
>
> Then you could copy this and try the loop_exp_frq() function (the test
> name could be test_loop_exp_frq()).  This should have two iterations
> and have the values of:
>
> [['SQ CPMG', 499862140.0],
>  ['SQ CPMG', 599890858.69999993]]
>
> Then the loop_exp_frq_offset() function (with the test name
> test_loop_exp_frq_offset()) could be created by copying and pasting
> and checking for exactly the same values - as you have no offsets set.
>  So the number of iterations is 2, and the values are as in the list
> of lists above.
>
> The same logic can be applied to each loop_*() function and a series
> of unit tests created in quick succession.  Such a network of simple
> but comprehensive unit tests will reveal exactly where the bug is!
> Checking all of the loop_*() functions will really pin-point the
> problem, as these functions all call each other for their looping.
>
> To help you get started, I'll add a first test :)  ...  Done (see
> http://article.gmane.org/gmane.science.nmr.relax.scm/20001).  Feel
> free top copy and paste this test for the other loop_*() functions.
>
> Regards,
>
> Edward
>
>
>
>
> On 20 February 2014 19:06, Troels Emtekær Linnet <tlinnet@xxxxxxxxxxxxx> 
> wrote:
> > Hi Edward.
> >
> > I now seem to have problems how to solve:
> > relax -s 
> > Relax_disp.test_bug_21665_cpmg_two_fields_two_delaytimes_fail_relax_disp
> >
> > -------
> > Traceback (most recent call last):
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/test_suite/system_tests/relax_disp.py",
> > line 295, in test_bug_21665_cpmg_two_fields_two_delaytimes_fail_relax_disp
> >     relax_disp.Relax_disp(pipe_name="compare_128_FT_R2eff",
> > pipe_bundle="cpmg_disp_sod1d90a", results_dir=self.tmpdir,
> > models=['R2eff'], grid_inc=3, mc_sim_num=5, modsel='AIC',
> > pre_run_dir=None, insignificance=1.0, numeric_only=False,
> > mc_sim_all_models=False, eliminate=True)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/auto_analyses/relax_disp.py",
> > line 116, in __init__
> >     self.run()
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/auto_analyses/relax_disp.py",
> > line 451, in run
> >     self.optimise(model=model)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/auto_analyses/relax_disp.py",
> > line 350, in optimise
> >     self.interpreter.grid_search(inc=self.grid_inc)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/prompt/uf_objects.py",
> > line 221, in __call__
> >     self._backend(*new_args, **uf_kargs)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/pipe_control/minimise.py",
> > line 152, in grid_search
> >     grid_search(lower=lower, upper=upper, inc=inc,
> > constraints=constraints, verbosity=verbosity)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py",
> > line 1093, in grid_search
> >     self.minimise(min_algor='grid', lower=lower, upper=upper, inc=inc,
> > constraints=constraints, verbosity=verbosity, sim_index=sim_index)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py",
> > line 1142, in minimise
> >     self._minimise_r2eff(min_algor=min_algor, min_options=min_options,
> > func_tol=func_tol, grad_tol=grad_tol, max_iterations=max_iterations,
> > constraints=constraints, scaling=scaling, verbosity=verbosity,
> > sim_index=sim_index, lower=lower, upper=upper, inc=inc)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py",
> > line 382, in _minimise_r2eff
> >     values.append(average_intensity(spin=spin, exp_type=exp_type,
> > frq=frq, offset=offset, point=point, time=time, sim_index=sim_index))
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/disp_data.py",
> > line 114, in average_intensity
> >     int_keys = find_intensity_keys(exp_type=exp_type, frq=frq,
> > offset=offset, point=point, time=time)
> >   File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/disp_data.py",
> > line 347, in find_intensity_keys
> >     raise RelaxError("No intensity data could be found corresponding
> > to the spectrometer frequency of %s MHz, dispersion point of %s and
> > relaxation time of %s s." % (frq*1e-6, point, time))
> > RelaxError: RelaxError: No intensity data could be found corresponding
> > to the spectrometer frequency of 499.86214 MHz, dispersion point of
> > 50.0 and relaxation time of 0.06 s
> > --------------
> >
> > I can point the error stems from:
> >
> > File 
> > "/sbinlab2/tlinnet/software/NMR-relax/relax_trunk/specific_analyses/relax_disp/api.py",
> > line 382, in _minimise_r2eff
> >     values.append(average_intensity(spin=spin, exp_type=exp_type,
> > frq=frq, offset=offset, point=point, time=time, sim_index=sim_index))
> >
> > ---------------
> >             # Loop over each spectrometer frequency and dispersion point.
> >             for exp_type, frq, offset, point in 
> > loop_exp_frq_offset_point():
> > ....
> >                 for time in cdp.relax_time_list:
> >                     values.append(average_intensity(spin=spin,
> > exp_type=exp_type, frq=frq, offset=offset, point=point, time=time,
> > sim_index=sim_index))
> >                     errors.append(average_intensity(spin=spin,
> > exp_type=exp_type, frq=frq, offset=offset, point=point, time=time,
> > error=True))
> >                     times.append(time)
> > -------------
> >
> > I think this should be solvable with something like:
> >
> >
> > ---------------
> >             # Loop over each spectrometer frequency and dispersion point.
> >             for exp_type, frq, offset, point in
> > loop_exp_frq_offset_point_time():
> > ....
> >             values.append(average_intensity(spin=spin,
> > exp_type=exp_type, frq=frq, offset=offset, point=point, time=time,
> > sim_index=sim_index))
> >             errors.append(average_intensity(spin=spin,
> > exp_type=exp_type, frq=frq, offset=offset, point=point, time=time,
> > error=True))
> >             times.append(time)
> > -------------
> >
> > What do you think ?
> >
> >
> > 2014-02-20 18:44 GMT+01:00 Troels Emtekær Linnet <tlinnet@xxxxxxxxxxxxx>:
> > > Hi Edward.
> > >
> > > I found that the main error was the return of the dispersion points
> > > from the function loop_point().
> > >
> > > It seemed that the minimal change would be to add an extra check in
> > > this function, and this
> > > extra check should be dependent on the time point.
> > >
> > > And hence i modified the loop_point() to take time as an argument, and
> > > switching the order
> > > would give me the time point.
> > >
> > > Best
> > > Troels
> > >
> > >
> > > 2014-02-20 18:27 GMT+01:00 Edward d'Auvergne <edward@xxxxxxxxxxxxx>:
> > > > Hi Troels,
> > > >
> > > > Is there a reason for switching the order?  It's useful to have that
> > > > in the commit message too.  I can see they are switched from the code
> > > > that was changed, but I cannot see why.
> > > >
> > > > Cheers,
> > > >
> > > > Edward
> > > >
> > > >
> > > >
> > > > On 20 February 2014 18:16,  <tlinnet@xxxxxxxxxxxxx> wrote:
> > > > > Author: tlinnet
> > > > > Date: Thu Feb 20 18:16:51 2014
> > > > > New Revision: 22222
> > > > >
> > > > > URL: http://svn.gna.org/viewcvs/relax?rev=22222&view=rev
> > > > > Log:
> > > > > Switched the looping over time points and dispersion points.
> > > > >
> > > > > Regarding bug #21665, (https://gna.org/bugs/?21665) - Running a CPMG 
> > > > > analysis with two fields at two delay times.
> > > > >
> > > > > Modified:
> > > > >     trunk/specific_analyses/relax_disp/disp_data.py
> > > > >
> > > > > Modified: trunk/specific_analyses/relax_disp/disp_data.py
> > > > > URL: 
> > > > > http://svn.gna.org/viewcvs/relax/trunk/specific_analyses/relax_disp/disp_data.py?rev=22222&r1=22221&r2=22222&view=diff
> > > > > ==============================================================================
> > > > > --- trunk/specific_analyses/relax_disp/disp_data.py (original)
> > > > > +++ trunk/specific_analyses/relax_disp/disp_data.py Thu Feb 20 18:16:51 
> > > > > 2014
> > > > > @@ -869,10 +869,10 @@
> > > > >          for frq, mi in loop_frq(return_indices=True):
> > > > >              # Then loop over the offset data.
> > > > >              for offset, oi in loop_offset(exp_type=exp_type, frq=frq, 
> > > > > return_indices=True):
> > > > > -                # Then the dispersion points.
> > > > > -                for point, di in loop_point(exp_type=exp_type, 
> > > > > frq=frq, offset=offset, return_indices=True):
> > > > > -                    # Finally the relaxation times.
> > > > > -                    for time, ti in loop_time(return_indices=True):
> > > > > +                # Then the relaxation times.
> > > > > +                for time, ti in loop_time(return_indices=True):
> > > > > +                    # Finally the dispersion points.
> > > > > +                    for point, di in loop_point(exp_type=exp_type, 
> > > > > frq=frq, offset=offset, time=time, return_indices=True):
> > > > >                          # Yield the data.
> > > > >                          if return_indices:
> > > > >                              yield exp_type, frq, offset, point, time, 
> > > > > ei, mi, oi, di, ti
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > relax (http://www.nmr-relax.com)
> > > > >
> > > > > This is the relax-commits mailing list
> > > > > relax-commits@xxxxxxx
> > > > >
> > > > > To unsubscribe from this list, get a password
> > > > > reminder, or change your subscription options,
> > > > > visit the list information page at
> > > > > https://mail.gna.org/listinfo/relax-commits
> > > >
> > > > _______________________________________________
> > > > relax (http://www.nmr-relax.com)
> > > >
> > > > This is the relax-devel mailing list
> > > > relax-devel@xxxxxxx
> > > >
> > > > To unsubscribe from this list, get a password
> > > > reminder, or change your subscription options,
> > > > visit the list information page at
> > > > https://mail.gna.org/listinfo/relax-devel