Re: Final model-free run with dauvergne_protocol.py -- September 23, 2012

Edward,

It seems we finally got around the relax halting problem but unfortunately I don’t know how! Logging didn’t affect anything but our last couple of trials worked fine and this sudden stop of relax could not be reproduced. Hopefully it’s not a failing hardware problem like you suggested. I haven't gotten to it yet but will definitely run relax on another workstation to rule this out.

In any case, we re-ran each DIFF_MODEL and they all completed without halting. Still, we get the same error when trying the final run:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

relax> value.set(val=-0.00017199999999999998, param='csa', spin_id='@N')

relax> pipe.create(pipe_name='local_tm - mf (Sat Sep 22 16:54:58 2012)', pipe_type='mf', bundle='mf (Sat Sep 22 16:54:58 2012)')

relax> results.read(file='results', dir='/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/local_tm/aic')
Opening the file '/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/local_tm/aic/results.bz2' for reading.

relax> pipe.create(pipe_name='sphere - mf (Sat Sep 22 16:54:58 2012)', pipe_type='mf', bundle='mf (Sat Sep 22 16:54:58 2012)')

relax> results.read(file='results', dir='/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/sphere/round_2/opt')
Opening the file '/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/sphere/round_2/opt/results.bz2' for reading.

relax> pipe.create(pipe_name='prolate - mf (Sat Sep 22 16:54:58 2012)', pipe_type='mf', bundle='mf (Sat Sep 22 16:54:58 2012)')

relax> results.read(file='results', dir='/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/prolate/round_2/opt')
Opening the file '/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/prolate/round_2/opt/results.bz2' for reading.

relax> pipe.create(pipe_name='oblate - mf (Sat Sep 22 16:54:58 2012)', pipe_type='mf', bundle='mf (Sat Sep 22 16:54:58 2012)')

relax> results.read(file='results', dir='/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/oblate/round_2/opt')
Opening the file '/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/oblate/round_2/opt/results.bz2' for reading.

relax> pipe.create(pipe_name='ellipsoid - mf (Sat Sep 22 16:54:58 2012)', pipe_type='mf', bundle='mf (Sat Sep 22 16:54:58 2012)')

relax> results.read(file='results', dir='/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/ellipsoid/round_2/opt')
Opening the file '/home/dgagne/nmrdata2/DG12-116_15SEP2012_ModelFree_Analysis_of_ECP_Free_Form/dauvergne_protocol_final/ellipsoid/round_2/opt/results.bz2' for reading.

relax> model_selection(method='AIC', modsel_pipe='final - mf (Sat Sep 22 16:54:58 2012)', bundle='mf (Sat Sep 22 16:54:58 2012)', pipes=['local_tm - mf (Sat Sep 22 16:54:58 2012)', 'sphere - mf (Sat Sep 22 16:54:58 2012)', 'prolate - mf (Sat Sep 22 16:54:58 2012)', 'oblate - mf (Sat Sep 22 16:54:58 2012)', 'ellipsoid - mf (Sat Sep 22 16:54:58 2012)'])
AIC model selection.

Global model - all diffusion tensor parameters and spin specific model-free parameters.
Data pipe Num_params_(k) Num_data_sets_(n) Chi2 Criterion
local_tm - mf (Sat Sep 22 16:54:58 2012) 233 678 374948.04349 375414.04349
sphere - mf (Sat Sep 22 16:54:58 2012) 150 678 379689.20076 379989.20076
prolate - mf (Sat Sep 22 16:54:58 2012) 155 678 379992.81969 380302.81969
oblate - mf (Sat Sep 22 16:54:58 2012) 158 678 379741.98189 380057.98189
ellipsoid - mf (Sat Sep 22 16:54:58 2012) 161 678 379888.19715 380210.19715
The model from the data pipe 'local_tm - mf (Sat Sep 22 16:54:58 2012)' has been selected.

relax> monte_carlo.setup(number=500)

relax> monte_carlo.create_data(method='back_calc')
RelaxError: The specific model has not been selected or set up.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

We went back to each individual DIFF_MODEL logs to figure out whether irregularities could be observed. Here is what shows up in each individual run (sphere, oblate, etc.) for each individual spin (we have 133 different @N spins in this pdb file, a few of those we listed in the 'unresolved' file):

Fitting to spin '#2KB5_mol1:4@N'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cannot run a grid search on a model with zero parameters, directly calculating the function value.
Storing the optimisation results, the optimised chi-squared value is lower than the current value (6753.10859718 < None).

The following message then appears:

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

relax> minimise(min_algor='newton', line_search=None, hessian_mod=None, hessian_type=None, func_tol=1e-25, grad_tol=None, max_iter=10000000, constraints=True, scaling=True, verbosity=1)

Over-fit spin deselection.

Only the model-free parameters for single spins will be used.

Fitting to spin '#2KB5_mol1:4@N'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cannot run optimisation on a model with zero parameters, directly calculating the function value.

Parameter values: []
Function value: 6701.4699098031615
Iterations: 0
Function calls: 1
Gradient calls: 0
Hessian calls: 0
Warning: No optimisation

Storing the optimisation results, the optimised chi-squared value is lower than the current value (6701.4699098 < None).

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Thanks again for your input.

Nick

Nicolas Doucet

Assistant Professor

INRS - Institut Armand-Frappier

University of Quebec

Institut Pasteur International Network

531 Boulevard des Prairies

Laval (Quebec) H7V 1B7 CANADA

Phone: (450) 687-5010 #4212

Fax: (450) 686-5501

Email: nicolas.doucet@xxxxxxxxxxx

Web: http://www.profs.inrs.ca/ndoucet/

On 2012-09-21, at 9:48 AM, Edward d'Auvergne <edward@xxxxxxxxxxxxx> wrote:

Hi,

I'll answer below:

On Friday, 21 September 2012, Doucet, Nicolas <Nicolas.Doucet@xxxxxxxxxxx> wrote:
> Edward,
>
> Using all the relaxation .out files previously created by relax in one single directory including the pdb and the dauvergne_protocol.py script, we simply ran the following command as you previously suggested:
>
> $ relax --log log dauvergne_protocol.py

I would recommend, simply as a test, to run relax without logging:

$ relax dauvergne_protocol.py

> We ran it twice identically except for the following selections:
>
> 1- With "res_name_col=3", "spin_num_col=4"
> 2- With "res_name_col-None", "spin_num_col=None"

I'm guessing that the minus is a typo here. If you have the residue number, the name is not essential. And if you have a unique spin name, then a spin number is not essential.

> In both cases the results are identical and yield the same warnings:
>
> 1- We see "Over-fit spin deselection" when "grid_search" is starting.

From your log file, I can see that protons are deselected - this is normal as you are looking at nitrogen relaxation rather than proton relaxation.

> 2- We always see the string "RelaxWarning: The spin '#2KB5_mol1:123@N' has been deselected because of missing relaxation data".

How many data points do you have for this residue 123 nitrogen?

> 3- In "fitting to spin (Grid search)", "fk" values are identical in both runs.
> 4- In "fitting to spin (Grid search)", the output message is the following: "Storing the optimisation results, the optimised chi-squared value is lower than the current value (12620.7348396 < None)". Is this what one would expect to see?

This is normal. This message makes more sense in the multiple-processor mode whereby the grid search is broken into pieces and each processor produces this message or the alternative saying that the chi-squared value is higher.

> Still, the major issue is the halting of relax in the middle of the tm8 grid search. I will try to install relax 2.1.1 on a 64-bit dual processor later today and run everything again.

This is strange! Maybe without the logging you will see more. Otherwise type 'dmesg' and see if the OS system log says more. I really have only seen such problems on failing hardware. But as this occurs in the same place each time, there must be some other problem! Your ram size of 3 Gb (https://gna.org/bugs/?20177) should be sufficient. We need to get to the bottom of this problem.

Regards,

Edward

> Nick
>
> --
> Nicolas Doucet
> Assistant Professor
> INRS - Institut Armand-Frappier
> University of Quebec
> Institut Pasteur International Network
> 531 Boulevard des Prairies
> Laval (Quebec) H7V 1B7 CANADA
> Phone: (450) 687-5010 #4212
> Fax: (450) 686-5501
> Email: nicolas.doucet@xxxxxxxxxxx
> Web: http://www.profs.inrs.ca/ndoucet/
> --
>
>
>
> -----Original Message-----
> From: edward.dauvergne@xxxxxxxxx on behalf of Edward d'Auvergne
> Sent: Fri 9/21/2012 4:35 AM
> To: Doucet, Nicolas
> Cc: relax-users@xxxxxxx
> Subject: Re: Final model-free run with dauvergne_protocol.py
>
> Hi Nick,
>
> How did you run relax? Could you copy and paste the entire command
> used? This will help me work out if you are using Gary Thompson's
> multiprocessor and how IO redirection is occurring. Do you get the
> same messages without IO redirection? It is quite peculiar that relax
> just stops like that without giving any output. I have seen this
> before, but it was due to a dying CPU (or harddrive, I don't remember
> exactly). I would guess that without the multiprocessor or IO
> redirection, you should see a much more informative error message.
>
> Cheers,
>
> Edward
>
>
>
> On 20 September 2012 21:33, Nicolas Doucet <nicolas.doucet@xxxxxxxxxxx> wrote:
>> Edward,
>>
>> We started the dauvergne_protocol.py again from scratch and we do see a few
>> warning messages in the initial local_tm run. Here is what we see right
>> before each tm grid search (the example is from the log file for tm0):
>>
>> relax> pipe.copy(pipe_from='origin - mf (Tue Sep 18 14:19:13 2012)',
>> pipe_to='tm0 - mf (Tue Sep 18 14:19:13 2012)', bundle_to='mf (Tue Sep 18
>> 14:19:13 2012)')
>>
>> relax> pipe.switch(pipe_name='tm0 - mf (Tue Sep 18 14:19:13 2012)')
>>
>> relax> model_free.select_model(model='tm0', spin_id=None)
>>
>> relax> grid_search(lower=None, upper=None, inc=11, constraints=True,
>> verbosity=1)
>>
>>
>> Over-fit spin deselection.
>>
>> RelaxWarning: The spin '#2KB5_mol1:2@N' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:3@N' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:4@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:4@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:5@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:5@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:6@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:6@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:7@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:7@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:8@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:8@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:9@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:9@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:10@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:10@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:11@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:11@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:12@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:12@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:13@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: The spin '#2KB5_mol1:13@H' has been deselected because of
>> missing relaxation data.
>> RelaxWarning: The spin '#2KB5_mol1:14@H' has been deselected because of an
>> absence of relaxation mechanisms.
>> RelaxWarning: T

Re: Final model-free run with dauvergne_protocol.py

Header

Content

Related Messages