Re: full_analysis.py -- October 16, 2006

It does seem a little bizarre that it took so long on fast machines.
I'm still not sure whether there is a problem or whether it's due to
the size of the system and the data sets.  The final stage of the
script, which generates the results file in the 'opt' directory, can
be the most computationally expensive.  This is because it is the
global model (the diffusion model together with all the model-free
models of all residues) which is being optimised.  The dimensionality
of this model is equal to the total of all parameters in your model
(sum the model-free parameters for all residues and the diffusion
parameters).  Because of this dimensionality, optimisation may take a
while.

Was the 2nd iteration much quicker to complete?  Each iteration of the
'full_analysis.py' script should complete quicker and quicker.  To
start with you have a collapse of the complexity of the global model.
This collapse should significantly speed up the grid searches and
optimisations between iterations.  Secondly, as iterations pass the
diffusion tensor gets closer and closer to the optimal values.  This
means that the optimisation of the global model will be much quicker.

If after a few iterations the calculations are still slow, there may
be a problem somewhere.  Do the results look reasonable?  If you're
using a Linux system you can type 'less -S opt/results.bz2' to view
the results file.  Can you see any issues or bizarre values in the
results files?

Edward

On 10/14/06, Sebastien Morin <sebastien.morin.1@xxxxxxxxx> wrote:

Hi

The simplest diffusion model ended the first round after 42h00 on a
Intel Pentium4 3.20GHz (Pelican)...

Now, after more than 50h00, the oblate (Intel Xeon CPU 2.80GHz, Hibou
1st CPU), prolate (Intel Xeon CPU 2.80GHz, Hibou 2nd CPU) and ellipsoid
(Intel Pentium4 CPU 3.20GHz, Pigeon) diffusion models are still running
on the first round...

Here are the last 80 lines of the log file for one of these stuck runs :

=======================================================
Entering sub-algorithm.

    Newton minimisation
    ~~~~~~~~~~~~~~~~~~~
    Line search:  Backtracking line search.
    Hessian modification:  The Gill, Murray, and Wright modified
Cholesky algorithm.

    k: 0       xk: array([  1.27705900e+04,   3.17894496e-01,
4.30908875e-01,   4.55262677e-01,
              8.50838124e-01,   2.47861916e+01,   7.92671118e-01,
2.14907851e+03,
              8.81854967e-01,   1.50334824e+01,   8.54027315e-01,
1.64578232e+03,
              8.76787298e-01,   7.92195168e-01,   1.79787560e+03,
9.87557449e-01,
              8.21228607e-01,   7.74117949e-01,   1.70184872e+03,
1.73932210e+00,
              9.49188204e-01,   8.85871296e-01,   1.04788711e+01,
9.17771174e-01,
              9.04377403e-01,   4.08815580e+02,   7.65590609e-01,
9.11933036e-01,
              8.91496184e-01,   5.56655768e+02,   1.44843220e+00,
8.90378757e-01,
              8.50622278e-01,   1.49912928e+03,   1.06027214e+00,
8.65625762e-01,
              3.88839270e+00,   8.35578101e-01,   5.25658520e+03,
1.23377507e+00,
              8.50832366e-01,   1.14258349e+01,   1.25778023e+00,
8.58099834e-01,
              8.33315784e-01,   1.04555350e+03,   8.80166810e-01,
8.42391353e-01,
              1.07328360e+03,   2.70180125e-01,   8.88393547e-01,
1.26380692e+01,
              7.51137909e-01,   7.16933985e+00,   7.01915972e-01,
1.58882726e+03,
              2.68906901e+00,   8.51688163e-01,   8.27015263e-01,
6.25535130e+02,
              1.51074548e+00,   9.03661361e-01,   8.87558977e-01,
4.80493151e+02,
              8.68556161e-01,   9.97819222e-01,   9.26383233e-01,
1.31515397e+01,
              5.99525186e-01,   8.57525619e-01,   8.30638445e-01,
1.85283345e+03,
              1.75756916e+00,   8.72986688e-01,   1.71531471e+01,
4.27254670e-01,
              8.80694475e-01,   1.81533883e+00,   8.70141488e-01,
1.60397307e+01,
              1.03406591e+00,   9.58191732e-01,   8.90603591e-01,
4.15861810e+02,
              8.30733937e-01,   1.09272538e+01,   7.23427079e-01,
8.93017099e+02,
              7.89229531e-01,   1.33440644e+01,   7.33002966e-01,
1.87268372e+03,
              1.42199321e+00,   7.53967611e-01,   1.66782443e+01,
6.61064763e-01,
              2.89157277e+03,   2.33751641e+00,   8.78699341e-01,
2.97736701e+01,
              8.38759921e-01,   3.18889549e+00,   8.28754282e-01,
8.17093600e-01,
              2.86614347e+02,   1.98926865e+00,   8.07960444e-01,
7.79624226e-01,
              8.20216424e+02,   1.69602623e+00,   8.69023915e-01,
1.29721545e+00,
              8.98814498e-01,   5.63508354e-01,   8.92301958e-01,
8.72610000e-01,
              5.83216661e+02,   1.72077954e+00,   8.31900384e-01,
7.95421005e-01,
              3.27213483e+03,   1.47608324e+00,   8.84116554e-01,
3.00188413e+01,
              8.42580844e-01,   8.13678584e-01,   1.28370304e+03,
1.52597551e+00,
              9.50467888e-01,   8.94644170e-01,   8.41203713e-01,
1.10218956e+03,
              2.26006275e+00,   9.68607682e-01,   3.25379935e+01,
8.41768913e-01,
              9.74570652e-01,   1.08978427e+00,   8.32856285e-01,
8.07695592e-01,
              2.11975345e+03,   1.95149433e+00,   8.75200671e-01,
1.18320570e+00,
              8.82565115e-01,   1.72699728e+00,   8.47796885e-01,
1.51362535e+00,
              8.69079500e-01,   5.14374767e+00,   2.29480254e+00,
9.13799620e-01,
              1.72758307e+01,   2.99204443e-01,   8.57041668e-01,
2.35031795e+00,
              8.96758624e-01,   2.18867392e+00,   8.88004177e-01,
1.55494958e+00,
              8.77947179e-01,   1.11920801e+01,   9.15181658e-01,
8.46939161e-01,
              1.83610590e+00,   8.38653701e-01,   8.18943616e-01,
3.56598335e+02,
              8.52078429e-01,   8.34895555e-01,   3.43932619e+02,
7.87542654e-01,
              8.37946629e-01,   1.24587108e+01,   1.64066609e+00,
8.40856744e-01,
              7.95194162e-01,   5.67222774e+02,   8.18578387e-01,
7.71610058e-01,
              1.65280184e+03,   1.22176221e+00,   8.50605183e-01,
7.42576359e+00,
              1.18569636e+00,   8.20365931e-01,   8.04178929e-01,
4.23814384e+02,
              1.63292525e+00,   7.87312635e-01,   7.21123309e-01,
1.55627601e+03,
              2.10657158e+00,   8.26679923e-01,   7.90596693e-01,
2.58068396e+02,
              2.69944847e-01,   8.24304505e-01,   7.88039728e-01,
3.81540776e+02,
              4.12294846e-01,   8.10539892e-01,   1.76756373e+01,
1.59567054e+00,
              8.69519794e-01,   2.27950293e+01,   1.60108929e+00,
8.46240651e-01,
              8.16750840e-01,   3.36726192e+02,   6.52545204e-01,
7.98871329e-01,
              1.67585865e+01,   1.29442868e+00,   8.72012584e-01,
1.58721788e+01,
              8.26110419e-01,   1.83079181e+03,   8.32081822e-01,
9.35711588e-01,
              4.44948915e+01,   9.47239844e-01,   8.69414580e-01,
2.60489705e+00,
              8.77732345e-01,   2.73754199e+00,   8.83674982e-01,
8.54351418e-01,
              2.31262785e+02,   3.66419461e-01,   7.86155599e-01,
1.01394952e+01,
              1.14673187e+00,   7.92655557e-01,   1.82156160e+01,
1.86619078e+00,
              8.73237261e-01,   8.49023676e-01,   3.69985402e+02,
8.63263470e-01,
              8.05115541e-01,   7.80479811e-01,   4.72043168e+02,
2.03730546e+00,
              7.71825768e-01,   7.31245168e-01,   5.45596104e+02,
8.32533217e-01,
              7.83786194e-01,   7.18560779e-01,   1.00144691e+03,
1.92514722e+00,
              8.98011263e-01,   1.61615950e+01,   8.72635732e-01,
1.94464132e+00,
              8.13134043e-01,   7.55765479e+00,   2.97987565e+00,
8.18914895e-01,
              7.86593136e-01,   2.12749118e+03,   2.53631014e+00,
8.64185295e-01,
              6.13997562e+00,   1.89991509e+00,   8.44947845e-01,   2

=======================================================

It seems the runs take a very long time making the final step of each
round (the step when they create the 'opt' directory, since all other
directories are already done).

Maybe it just the size of the dataset that slows down things... Let's be
patient !!

Cheers


Séb

Re: full_analysis.py

Header

Content

Related Messages