Re: r3261 - in /branches/multi_processor: generic_fns/ multi/ prompt/ specific_fns/ -- April 12, 2007 - 18:44

Edward d'Auvergne wrote:

> > From an implementation and debugging perspective, this prepending
>
> would be quite useful.  I now understand what this is for.  In the
> final defaults the user need not see any of this, although as you
> hinted it would help in debugging MPI code in the future.


it is also useful to the user as he can see that all the processors are 
firing and that work is being distributed nicely

> As for the question about 'minimisations that are not interrelated',
> what do you mean by 'not interrelated'?  Do you mean different
> model-free models associated with different data pipes?  The new
> 'pipes' argument (previously referred to as the 'runs' argument) which
> is unrelated to the current 'run' argument will not be handled by the
> user functions themselves.  This was Chris' powerful proposal for
> executing a single user function on many different data pipes.  The
> 'pipes' argument will be caught prior to executing the user function
> code of the 'prompt' directory and be stripped from the argument list.
> This is deliberate - the 'run', 'runs', or 'pipes' argument will no
> longer be propagated throughout the program.


indeed so the question becomes one of deciding when all the relevant 
minimisations/mc runs have been added and calling run_command_queue (I 
guess we will have to wait for the code and have a look)

certainly at the end of the use function works but how much further can 
we bury it  comments chris... (before you hop on your plane to oz ;-)) 

regards
gary

> Cheers,
>
> Edward
>
>
>
> On 4/12/07, Gary S. Thompson <garyt@xxxxxxxxxxxxxxx> wrote:
>
> > Edward d'Auvergne wrote:
> >
> > >Gary,
> > >
> > >I've had a look at the changes and I'm a little alarmed by the comment
> > >'# these may need to be in c they cause an pprox 10% slowdown'.  What
> > >is the 'multi/PrependStringIO.py' module actually used for?  There
> > >isn't much in this module explaining its function.  If the slowdown is
> > >during the minimisation, do we need to prepend 'S> ' and 'E> '?
> > >
> > >
> > This merely currently a reminder to me to do some checking! I am not
> > even sure of the data as most of the overhead comes from the output
> > comes from  writing to stdout and some of the numbers I have in my head
> > may come from runs thatouput to files. The commands are quite simple
> > they prepend output from the masters and slaves with  some text that
> > indicates which stream and which processor the output came from e.g. on
> > a multiprocessor you get
> >
> > M S> script
> > M S>
> > M S>
> > M S>
> > M S>                                      relax repository checkout
> > M S>
> > M S>                           Protein dynamics by NMR relaxation data
> > analysis
> > M S>
> > M S>                              Copyright (C) 2001-2006 Edward 
> > d'Auvergne
> > M S>
> > M S> This is free software which you are welcome to modify and
> > redistribute under the conditions of the
> > M S> GNU General Public License (GPL).  This program, including all
> > modules, is licensed under the GPL
> > M S> and comes with absolutely no warranty.  For details type 'GPL'.
> > Assistance in using this program
> > M S> can be accessed by typing 'help'.
> > M S>
> > M S> script = 'test_small.py'
> > M S>
> > ---------------------------------------------------------------------------------------------------- 
> >
> > M S> # Script for model-free analysis.
> > M S>
> > M S> dataPaths = ['test_data/noe750.dat',
> > M S>              'test_data/T1_750_hgts.bs',
> > M S>              'test_data/T2_750_hgts.bs']
> > M S>
> > M S> dataTypes = [('NOE', '750', 750.8e6),
> > M S>              ('R1', '750', 750.8e6),
> > M S>              ('R2', '750', 750.8e6)]
> > M S>
> > M S> # Set the run names (also the names of preset model-free models).
> > M S> #runs = ['tm1', 'tm2', 'tm3', 'tm4', 'tm5', 'tm6', 'tm7', 'tm8', 
> > 'tm9']
> > M S> runs = ['m1', 'm2', 'm3', 'm4', 'm5', 'm6', 'm7', 'm8', 'm9']
> > M S>
> > M S> # Nuclei type
> > M S> nuclei('N')
> > M S>
> > M S> test=True
> > M S> # Loop over the runs.
> > M S> for name in runs:
> > M S>     # Create the run.
> > M S>     run.create(name, 'mf')
> > M S>
> > M S>     # Load the sequence.
> > M S>     #sequence.read(name, 'noe.500.out')
> > M S>
> > M S>     # Load a PDB file.
> > M S>     structure.read_pdb(name, 'test_data/test.pdb')
> > M S>     #pdb.read(name, 'test_data/test.pdb')
> > M S>     structure.vectors(name, proton='HN')
> > M S>
> > M S>     # Load the relaxation data.
> > M S>     for dataSet in xrange(len(dataPaths)):
> > M S>         relax_data.read(name, dataTypes[dataSet][0],
> > dataTypes[dataSet][1], dataTypes[dataSet][2], dataPaths[dataSet])
> > M S>
> > M S>     # Setup other values.
> > M S>     diffusion_tensor.init(name, 1e-8, fixed=1)
> > M S>     #diffusion_tensor.init(name, (1e-8, 1.0, 60, 290),
> > param_types=0, spheroid_type='oblate', fixed=0)
> > M S>     value.set(name, 1.02 * 1e-10, 'bond_length')
> > M S>     value.set(name, -160 * 1e-6, 'csa')
> > M S>     #value.set(name, 0.970, 's2')
> > M S>     #value.set(name, 1.0, 's2f')
> > M S>     #value.set(name, 2048e-12, 'te')
> > M S>     #value.set(name, 2048e-12, 'tf')
> > M S>     #value.set(name, 2048e-12, 'ts')
> > M S>     #value.set(name, 0.149/(2*pi*600e6)**2, 'rex')
> > M S>
> > M S>     # Select the model-free model.
> > M S>     model_free.select_model(run=name, model=name)
> > M S>     #fix(name, 'all_res')
> > M S>
> > M S>     # Minimise.
> > M S>     print '***- grid searching ' + name
> > M S>     grid_search(name, inc=11)
> > M S>     if not test:
> > M S>         print '***- minimise' + name
> > M S>         minimise('newton', run=name)
> > M S>
> > M S> if test:
> > M S>     for name in runs:
> > M S>         print '***- minimise' + name
> > M S>         minimise('newton', run=name)
> > M S>
> > M S> for name in runs:
> > M S>     # Write the results.
> > M S>     results.write(run=name, file='results', force=1)
> > M S>
> > M S> # Save the program state.
> > M S> state.save('save', force=1)
> > M S>
> > ---------------------------------------------------------------------------------------------------- 
> >
> > M S>
> > M S> relax> nuclei(heteronuc='N')
> > M S>
> > M S> relax> run.create(run='m1', run_type='mf')
> > M S>
> > M S> relax> structure.read_pdb(run='m1', file='test_data/test.pdb',
> > dir=None, model=None, load_seq=1)
> > M S> Loading all structures from the PDB file.
> > M S> Structure('test_data/test.pdb'):
> > M S>   Peptide chain BP of length 28
> > M S>
> > M S>
> > M S> Loading the sequence from the PDB file.
> > M S>
> > M S>
> > M S> relax> structure.vectors(run='m1', heteronuc='N', proton='HN',
> > res_num=None, res_name=None)
> > M S>
> > M S> Calculating the unit XH vectors from the structure.
> > M S>
> > M S> relax> relax_data.read(run='m1', ri_label='NOE', frq_label='750',
> > frq=750800000.0, file='test_data/noe750.dat', dir=None, num_col=0,
> > name_col=1, data_
> > col=2, error_col=3, sep=None)
> > M S> Opening the file 'test_data/noe750.dat' for reading.
> > M S>
> > M S> relax> relax_data.read(run='m1', ri_label='R1', frq_label='750',
> > frq=750800000.0, file='test_data/T1_750_hgts.bs', dir=None, num_col=0,
> > name_col=1, da
> > ta_col=2, error_col=3, sep=None)
> > M S> Opening the file 'test_data/T1_750_hgts.bs' for reading.
> > M S>
> > M S> relax> relax_data.read(run='m1', ri_label='R2', frq_label='750',
> > frq=750800000.0, file='test_data/T2_750_hgts.bs', dir=None, num_col=0,
> > name_col=1, da
> > ta_col=2, error_col=3, sep=None)
> > M S> Opening the file 'test_data/T2_750_hgts.bs' for reading.
> > M S>
> > M S> relax> diffusion_tensor.init(run='m1', params=1e-08,
> > time_scale=1.0, d_scale=1.0, angle_units='deg', param_types=0,
> > spheroid_type=None, fixed=1)
> > M S>
> > M S> relax> value.set(run='m1', value=1.0200000000000001e-10,
> > param='bond_length', res_num=None, res_name=None)
> > M S>
> > M S> relax> value.set(run='m1', value=-0.00015999999999999999,
> > param='csa', res_num=None, res_name=None)
> > M S>
> > M S> relax> model_free.select_model(run='m1', model='m1', res_num=None)
> > M S> ***- grid searching m1
> > M S>
> > M S> relax> grid_search(run='m1', lower=None, upper=None, inc=11,
> > constraints=1, print_flag=1)
> > M S> Only the model-free parameters for single residues will be used.
> > M S> 1 S>
> > 1 S>
> > 1 S> Fitting to residue: 29 LYS
> > 1 S> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 1 S> Unconstrained grid search size: 11 (constraints may decrease this
> > size).
> > 1 S>
> > 1 S>
> > 1 S> Grid search
> > 1 S> ~~~~~~~~~~~
> > 1 S>
> > 1 S> Searching the grid.
> > 1 S> k: 0       xk: array([
> > 0.])                                                      fk:
> > 8.6505190311418673e+200
> > 1 S> k: 1       xk: array([
> > 0.1])                                                     fk:
> > 2911.9315925220235
> > 1 S> k: 2       xk: array([
> > 0.2])                                                     fk:
> > 2169.3843091989061
> > 1 S> k: 3       xk: array([
> > 0.3])                                                     fk:
> > 1609.7386225859505
> > 1 S> k: 4       xk: array([
> > 0.4])                                                     fk:
> > 1232.9945326831564
> > 1 S> k: 5       xk: array([
> > 0.5])                                                     fk:
> > 1039.1520394905253
> > 1 S> k: 6       xk: array([
> > 0.6])                                                     fk:
> > 1028.2111430080565
> > 1 S>
> > 1 S> Parameter values: array([ 0.6])
> > 1 S> Function value:   1028.2111430080565
> > 1 S> Iterations:       11
> > 1 S> Function calls:   11
> > 1 S> Gradient calls:   0
> > 1 S> Hessian calls:    0
> > 1 S> Warning:          None
> > 1 S>
> > 1 S> 2 S>
> > 2 S>
> > 2 S> Fitting to residue: 28 GLY
> > 2 S> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 2 S> Unconstrained grid search size: 11 (constraints may decrease this
> > size).
> > 2 S>
> > 2 S>
> > 2 S> Grid search
> > 2 S> ~~~~~~~~~~~
> > 2 S>
> > 2 S> Searching the grid.
> > 2 S> k: 0       xk: array([
> > 0.])                                                      fk:
> > 2.2887432739557032e+200
> > 2 S> k: 1       xk: array([
> > 0.1])                                                     fk:
> > 5117.0688467207683
> > 2 S> k: 2       xk: array([
> > 0.2])                                                     fk:
> > 4166.2302817415612
> > 2 S> k: 3       xk: array([
> > 0.3])                                                     fk:
> > 3333.4118391834986
> > 2 S> k: 4       xk: array([
> > 0.4])                                                     fk:
> > 2618.613519046577
> > 2 S> k: 5       xk: array([
> > 0.5])                                                     fk:
> > 2021.8353213308012
> > 2 S> k: 6       xk: array([
> > 0.6])                                                     fk:
> > 1543.0772460361686
> > 2 S> k: 7       xk: array([
> > 0.7])                                                     fk:
> > 1182.3392931626797
> > 2 S> k: 8       xk: array([
> > 0.8])                                                     fk:
> > 939.62146271033373
> > 2 S> k: 9       xk: array([
> > 0.9])                                                     fk:
> > 814.92375467913234
> > 2 S> k: 10      xk: array([
> > 1.])                                                      fk:
> > 808.2461690690742
> > 2 S>
> > 2 S> Parameter values: array([ 1.])
> > 2 S> Function value:   808.2461690690742
> > 2 S> Iterations:       11
> > 2 S> Function calls:   11
> > 2 S> Gradient calls:   0
> > 2 S> Hessian calls:    0
> > 2 S> Warning:          None
> > 2 S>
> > 2 S> 3 S>
> > 3 S>
> > 3 S> Fitting to residue: 27 ALA
> > 3 S> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 3 S> Unconstrained grid search size: 11 (constraints may decrease this
> > size).
> > 3 S>
> > 3 S>
> > 3 S> Grid search
> > 3 S> ~~~~~~~~~~~
> > 3 S>
> > 3 S> Searching the grid.
> > 3 S> k: 0       xk: array([
> > 0.])                                                      fk:
> > 3.9211841976276823e+200
> > 3 S> k: 1       xk: array([
> > 0.1])                                                     fk:
> > 3619.6376359491696
> > 3 S> k: 2       xk: array([
> > 0.2])                                                     fk:
> > 2774.6135909121058
> > 3 S> k: 3       xk: array([
> > 0.3])                                                     fk:
> > 2061.2241153073624
> > 3 S> k: 4       xk: array([
> > 0.4])                                                     fk:
> > 1479.4692091349366
> > 3 S> k: 5       xk: array([
> > 0.5])                                                     fk:
> > 1029.3488723948312
> > 3 S> k: 6       xk: array([
> > 0.6])                                                     fk:
> > 710.86310508704514
> > 3 S> k: 7       xk: array([
> > 0.7])                                                     fk:
> > 524.0119072115782
> > 3 S> k: 8       xk: array([
> > 0.8])                                                     fk:
> > 468.79527876843048
> > 3 S>
> > 3 S> Parameter values: array([ 0.8])
> > 3 S> Function value:   468.79527876843048
> > 3 S> Iterations:       11
> > 3 S> Function calls:   11
> > 3 S> Gradient calls:   0
> > 3 S> Hessian calls:    0
> > 3 S> Warning:          None
> > 3 S>
> > 3 S> 4 S>
> > 4 S>
> > 4 S> Fitting to residue: 26 LYS
> > 4 S> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 4 S> Unconstrained grid search size: 11 (constraints may decrease this
> > size).
> > 4 S>
> > 4 S>
> > 4 S> Grid search
> > 4 S> ~~~~~~~~~~~
> > 4 S>
> > 4 S> Searching the grid.
> > 4 S> k: 0       xk: array([
> > 0.])                                                      fk:
> > 2.6874496103198057e+200
> > 4 S> k: 1       xk: array([
> > 0.1])                                                     fk:
> > 2589.2620896286544
> > 4 S> k: 2       xk: array([
> > 0.2])                                                     fk:
> > 2035.0724673555715
> > 4 S> k: 3       xk: array([
> > 0.3])                                                     fk:
> > 1564.7221140603995
> > 4 S> k: 4       xk: array([
> > 0.4])                                                     fk:
> > 1178.2110297431368
> > 4 S> k: 5       xk: array([
> > 0.5])                                                     fk:
> > 875.53921440378497
> > 4 S> k: 6       xk: array([
> > 0.6])                                                     fk:
> > 656.70666804234315
> > 4 S> k: 7       xk: array([
> > 0.7])                                                     fk:
> > 521.7133906588117
> > 4 S> k: 8       xk: array([
> > 0.8])                                                     fk:
> > 470.55938225319028
> > 4 S>
> > 4 S> Parameter values: array([ 0.8])
> > 4 S> Function value:   470.55938225319028
> > 4 S> Iterations:       11
> > 4 S> Function calls:   11
> > 4 S> Gradient calls:   0
> > 4 S> Hessian calls:    0
> > 4 S> Warning:          None
> > 4 S>
> > 4 S> 5 S>
> > 5 S>
> > 5 S> Fitting to residue: 25 ASP
> > 5 S> ~~~~~~~~~~~~~~~~~~~~~~~~~~
> > 5 S> Unconstrained grid search size: 11 (constraints may decrease this
> > size).
> > 5 S>
> >
> > ....
> >
> > obviously I will add some flags to squash this if the user wants to at a
> > later date, but it is currently very useful to me to see that is 
> > going on
> >
> > >In 'specific_fns/model_free.py' most of the changes where objects are
> > >no longer placed into 'self' will be eliminated in the 1.3 line.  Most
> > >class methods will probably be distributed into different modules
> > >within the directory 'specific_fns/model_free/' and converted to
> > >simple functions.  This will complicate the porting of the branch into
> > >a patch for the 1.3 line.
> > >
> > >
> > I guessed this would be the case. I expected that we will have to make a
> > new branch from the 1.3.x line when it stabalises, merge what can be
> > megered from multi and manually cross port the rest..
> >
> > >On last thing is that I think that the changes to
> > >'prompt/minimisation.py' belong elsewhere.  The call to
> > >'self.relax.processor.run_queue()' should be in the generic code
> > >(specifically 'generic_fns/minimisation.py') to allow any type of UI
> > >to have access to the MPI code.  The code in the 'prompt/' directory
> > >is solely used to set up the behaviour of the user functions
> > >accessible through the prompt and script UIs, is used to test the
> > >validity of the arguments and then execute the generic or specific
> > >code.  Then if a graphical user interface (GUI) or some new UI is
> > >added to relax, they will all automatically be MPI compatible.
> > >
> > >
> >
> > That seems fine. However, there is a problem here the run_queue method
> > needs to be called after all the   the minimisations that are not
> > interrelated have been queued. How will this work with th command pipes?
> >
> > e.g. when we have commands that take runs as opposed to run will
> > generic.minimise.minimise get called with a runs parameter instead of 
> > run?
> >
> > regards
> > gary
> >
> > >Cheers,
> > >
> > >Edward
> > >
> > >
> > >On 4/8/07, garyt@xxxxxxxxxxxxxxx <garyt@xxxxxxxxxxxxxxx> wrote:
> > >
> > >
> > >>Author: varioustoxins
> > >>Date: Sat Apr  7 22:11:21 2007
> > >>New Revision: 3261
> > >>
> > >>URL: http://svn.gna.org/viewcvs/relax?rev=3261&view=rev
> > >>Log:
> > >>Major update - monte carlo simulations supported (trivially)
> > >>also implimented: chunky queues, exception throwing on slaves, and 
> > rank
> > >>annotated output. note uniprocessor support currently broken.
> > >>
> > >>Added:
> > >>    branches/multi_processor/multi/PrependStringIO.py
> > >>Modified:
> > >>    branches/multi_processor/generic_fns/minimise.py
> > >>    branches/multi_processor/multi/commands.py
> > >>    branches/multi_processor/multi/mpi4py_processor.py
> > >>    branches/multi_processor/multi/processor.py
> > >>    branches/multi_processor/prompt/minimisation.py
> > >>    branches/multi_processor/specific_fns/model_free.py
> > >>
> > >>[This mail would be too long, it was shortened to contain the URLs 
> > only.]
> > >>
> > >>Modified: branches/multi_processor/generic_fns/minimise.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/generic_fns/minimise.py?rev=3261&r1=3260&r2=3261&view=diff 
> >
> > >>
> > >>Added: branches/multi_processor/multi/PrependStringIO.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/multi/PrependStringIO.py?rev=3261&view=auto 
> >
> > >>
> > >>Modified: branches/multi_processor/multi/commands.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/multi/commands.py?rev=3261&r1=3260&r2=3261&view=diff 
> >
> > >>
> > >>Modified: branches/multi_processor/multi/mpi4py_processor.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/multi/mpi4py_processor.py?rev=3261&r1=3260&r2=3261&view=diff 
> >
> > >>
> > >>Modified: branches/multi_processor/multi/processor.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/multi/processor.py?rev=3261&r1=3260&r2=3261&view=diff 
> >
> > >>
> > >>Modified: branches/multi_processor/prompt/minimisation.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/prompt/minimisation.py?rev=3261&r1=3260&r2=3261&view=diff 
> >
> > >>
> > >>Modified: branches/multi_processor/specific_fns/model_free.py
> > >>URL: 
> > http://svn.gna.org/viewcvs/relax/branches/multi_processor/specific_fns/model_free.py?rev=3261&r1=3260&r2=3261&view=diff 
> >
> > >>
> > >>
> > >>_______________________________________________
> > >>relax (http://nmr-relax.com)
> > >>
> > >>This is the relax-commits mailing list
> > >>relax-commits@xxxxxxx
> > >>
> > >>To unsubscribe from this list, get a password
> > >>reminder, or change your subscription options,
> > >>visit the list information page at
> > >>https://mail.gna.org/listinfo/relax-commits
> > >>
> > >>
> > >>
> > >
> > >_______________________________________________
> > >relax (http://nmr-relax.com)
> > >
> > >This is the relax-devel mailing list
> > >relax-devel@xxxxxxx
> > >
> > >To unsubscribe from this list, get a password
> > >reminder, or change your subscription options,
> > >visit the list information page at
> > >https://mail.gna.org/listinfo/relax-devel
> > >
> > >.
> > >
> > >
> > >
> >
> >
> > --
> > -------------------------------------------------------------------
> > Dr Gary Thompson
> > Astbury Centre for Structural Molecular Biology,
> > University of Leeds, Astbury Building,
> > Leeds, LS2 9JT, West-Yorkshire, UK             Tel. +44-113-3433024
> > email: garyt@xxxxxxxxxxxxxxx                   Fax  +44-113-2331407
> > -------------------------------------------------------------------
>
> .


--
-------------------------------------------------------------------
Dr Gary Thompson
Astbury Centre for Structural Molecular Biology,
University of Leeds, Astbury Building,
Leeds, LS2 9JT, West-Yorkshire, UK             Tel. +44-113-3433024
email: garyt@xxxxxxxxxxxxxxx                   Fax  +44-113-2331407
-------------------------------------------------------------------