Re: Numerical integration of the Bloch-McConnell equations for the relaxation dispersion analysis in relax. -- July 12, 2013 - 09:55

Hi.

Very interesting with Numerical integration of the Bloch-McConnell equations !

Regarding plotting.

I find it very usefull, that xmgrace plots are produced during analysis and the
graphs are exported as stand-alone files, so it not necessary to open relax to view the files.

I have never used xmgrace before, but only gnuplot. But it seems that the GUI for
xmgrace is a little more intuitive. And they kind-of do the same things.
With the latest tweak that plots are also exported to png files, I can quickly go

through 100 residues in my folder, and modify a single residue without opening
the saved data. That's good.
But it is not good, that I probably sometimes would need to combine some data into

one graph. With the current solution, this is not easily done.

Regarding matplotlib.
I am in favor of matplotlib, because it is a plotting language that I understand (python)
and I can re-use old plot code, to get the graph I like.
And it is in development.

As relax are now, one should be able to make an external "plot file".

And then run the commands via: User functions (n-z) -> script
That is how I load my data in. I write a script file, open relax gui, and run the script.

In the file, one import matplotlib and write some line to extract data from the the current data pipe "cdp".
For example data for 1-3 residues.

I am still looking into, how to extract relevant data from the data pipe.
http://nmr-relax.kimlinnet.dk/index.php?title=Category:List_objects

And then just write some matplotlib commands to plot it.

For a future user friendly GUI solution, I imagine a menu where one can select some template

matplotlib functions. Then modify them in a wxpython text editor or OS editor, 

and then save them in the relax folder, or in $HOME/.relax/xxx

or in the current working directory. And then of course push a "execute" button.

Another possibility would also just be, to have a matplotlib script collection on the wiki.

Best

Troels Emtekær Linnet


2013/7/11 Edward d'Auvergne <edward@xxxxxxxxxxxxx>
>
> Hi Paul,
>
> Thank you for the code contribution!  First of all, I have to
> apologise for the length of this email.  I have tried to be as
> comprehensive as possible.
>
> I have attached the ZIP file to the relax task #7712
> (https://gna.org/task/?7712).  Note that I have added the [relax-devel
> att gna dot org] mailing list to the CC list to bring this out into
> the public - it will be permanently archived in multiple locations
> (see http://news.gmane.org/gmane.science.nmr.relax.devel,
> http://www.mail-archive.com/relax-devel@xxxxxxx/,
> https://mail.gna.org/public/relax-devel/,
> http://marc.info/?l=relax-devel&r=1&w=2).  This is necessary as relax
> is an open source project where, without exception, all development
> occurs in public.  It is supported by a number of developers, though
> many are now inactive
> (https://gna.org/project/memberlist.php?group=relax).  Anyone is free
> to become a relax developer and advance the project for their own
> needs/papers (http://www.nmr-relax.com/manual/relax_development.html).
>
> Below I will address each of your points.  But first, and most
> importantly, we need to address the copyright issue.  For the code to
> be added to relax, each person who has contributed code to the
> fitting_main.py file and the original MATLAB scripts needs to
> communicate in public, by having a message permanently archived on the
> [relax-devel att gna dot org] mailing list, that they agree to have
> their code licenced under the Gnu Public Licence version 3 or higher
> (GPL3+).  This does not prevent the author placing their code under a
> different copyright licence.  But this is only their code, and the
> code which was released under the GPL3 licence can never be retracted.
>  For details, see http://gplv3.fsf.org/,
> http://www.gnu.org/licenses/gpl-3.0.html, or the docs/COPYING file
> that comes with relax.  I will add a copyright statement to each file
> produced, and it will look like the start of
> http://svn.gna.org/viewcvs/relax/trunk/test_suite/system_tests/consistency_tests.py?view=markup.
>  There will be one line for each author.  This consists of the years
> the author wrote code, the full author name, and maybe a contact email
> address.  This would also include all authors of the original MATLAB
> scripts.
>
> For reference, an archived message thread on this development is:
>
> http://thread.gmane.org/gmane.science.nmr.relax.devel/3790
>
> Also for reference is the history of relaxation dispersion in relax.
> Development is happening in a relax branch rather than the main
> development line.  Therefore it will not be included in an official
> relax release until the branch is completed and merged back.  The
> branch was started by Sébastien Morin who contributed much of the
> infrastructure back in 2009.  I recently ported it to the newest relax
> code base and significantly expanded its capabilities for my own
> needs.  Troels Linnet is also working on this branch for his own paper
> and needs, and is advancing towards becoming a full relax developer.
>
> Now, to the mail:
>
>
> On 9 July 2013 08:16, Paul Schanda <paul.schanda@xxxxxx> wrote:
> > Hi Edward,
> >
> > As promised, I am sending you in the attachment the python functions that we
> > are using for fitting CPMG data, that was written essentially by Mathilde
> > Lescanne and Dominique Marion, based on my MATLAB scripts.
>
> Thank you.  As discussed at the IBS in Grenoble two weeks ago, this
> could one day also be incorporated into the NESSY project
> (http://home.gna.org/nessy/).  However that project is in a frozen
> state so it would require a significant commitment by a dedicated
> developer for this to happen.
>
> Also, the code is in a state whereby it can be directly copied and
> pasted into relax modules.  Only small modifications will be
> necessary.  This includes adding docstrings
> (http://www.python.org/dev/peps/pep-0257/) using the Epydoc format
> (http://epydoc.sourceforge.net/,
> http://epydoc.sourceforge.net/epytext.html) for the relax API
> documentation (http://www.nmr-relax.com/api/).  I will also add lots
> of comments as the relax coding style requires ~15 to 30% of comment
> lines, simply to make it easier for non-coders to understand the code
> (http://www.nmr-relax.com/manual/Comments.html).  I may ask you,
> Mathilde, about these comments later.
>
>
> > In principle, the fit program should directly work as it is, and generate
> > plots automatically. (using Matplotlib! I would strongly recommend that
> > relax also uses matplotlib. Creating plots exclusively outside of python is
> > a pain in the neck, and only useful for writing a paper, which is the step
> > that one does after having looked at tons of figures...)
>
> As I mentioned, Troels Linnet is also interested in Matplotlib as he
> has used that in his own dispersion scripts.  Matplotlib support would
> be quite useful and is welcome.  But it is also limited compared to
> the capabilities of Grace.  But the main point with matplotlib support
> would be that it be added as an optional dependency.  For the
> automated relaxation dispersion analysis in relax, matplotlib plots
> can be created when the Python package is present.  We should probably
> discuss this in a different thread.
>
>
> > You will find five different versions of fitting CPMG data. In fact we only
> > use two of them, namely an explicit numerical Bloch-McConnell treatment that
> > works for perfect 180deg pulses, and a Maple-derived solution of the
> > Bloch-McConnell treatment. The latter is numerically exact and in practice
> > the most useful version. It is being used by some of the Kay-(ex-)postdocs.
>
> These, I assume, are the functions returned by MakeFunction() when the
> a argument is set to 1 and 5 respectively.  Would you have the primary
> references for these two with DOI numbers?  This is important for the
> relax documentation (both the user manual and in the user function
> documentation).
>
>
> > The script allows to fit multiple residues and multiple fields at the same
> > time. This is all controlled via a Excel spreadsheet, where the user can put
> > the information. In relax you would do this differently, in a GUI, but the
> > logic would stay pretty much the same. The script uses a python package for
> > handling Excel files; you may need to install it.
>
> For relax, the details in the Excel spreadsheet would either be placed
> directly into a script (for the script UI) or manually input via the
> GUI.  relax supports all of these options, though in the automated
> analysis it is run a little differently.  You directly define spin
> clusters.  Then all spin clusters as well as free spins are optimised
> to all selected dispersion models simultaneously.  You are however
> free to create a non-automated, custom analysis in a script whereby
> you can do whatever you wish.  relax is designed with flexibility in
> mind and allows you to do anything you wish (and if that doesn't work,
> then it should be considered as a bug).
>
>
> > If you could incorporate this into relax it would be great.
>
> Once the copyright issues are sorted out and there are responses from
> all involved, I could then incorporate this quite quickly into relax.
> It will only be a few hours of work - mainly due to the documentation.
>  So having as much detail as possible about these numerical models
> makes my life much easier.  See the user manual (you need to compile
> it yourself) and relax_disp.select_model user function for how this
> documentation is currently set up.
>
>
> > I would also
> > greatly appreciate if you could help us getting relax to run. Our interest
> > in sending this stuff to you is that we would like to have an environment
> > where the data handling (input, output) is done properly, but which
> > basically does the fitting that we would like to do (Bloch-McConnell).
>
> For the installing the relaxation dispersion branch, Troels has
> created a very useful tutorial:
>
> http://wiki.nmr-relax.com/index.php?title=Installation_linux
>
> This includes instructions for building the relax user manual (this
> requires LaTeX).  Note that this wiki is still in a state of being set
> up, so user creation and page editing is not yet fully functional.
>
>
> > As said, it would be really great to have the program create plots
> > automatically. I don't want to use xmgrace for generating plots, or at least
> > not if I just want to get a quick overview of the fits. Could you please
> > have a look how complicated it would be to include matplotlib-plotting into
> > relax?
>
> The complication is only deciding at which level this would be added.
> For example the R2eff/R1rho plots are currently created by the
> relax_disp.plot_disp_curves user function after a given dispersion
> model has been optimised for all spin clusters and free spins (and
> Monte Carlo simulations have been performed).  This could simply be
> modified by adding an argument for selecting the graph type (set to
> either 'grace' or 'matplotlib').  If this is added, then in the
> auto-analysis I would enable both.  For this, ideas can be taken from
> the fitting_main.py script, from Troels' scripts, and from NESSY's
> matplotlib plotting.
>
> On another level, there are the grace.view and grace.write user
> functions.  These are used for plotting all of the other parameters.
> If there is interest, then this code could be copied and modified to
> the matplotlib.view and matplotlib.write user functions.
>
> Troels, what do you think?
>
> Note that, although Grace and OpenDX are the current ways of viewing
> 2D and 3D plots, the relax design is not exclusive to these softwares
> and allows other plotting capabilities to be relatively easily added
> by anyone with interest.  Matplotlib, Gnuplot, SigmaPlot, Octave
> (which uses Gnuplot), MATLAB, and Mathematica have all been considered
> at some point as a plotting output from relax.  Though most
> users/developers found Grace sufficient so these were never
> implemented due to a lack of interest.
>
>
> > Please let me know if you need any further information.
>
> We'll see what is needed as things progress.
>
> Cheers,
>
> Edward
>
>
> >
> > best -
> >
> >
> > Paul
> >
> >
> >
> > --
> > Paul Schanda, Ph.D.
> > Biomolecular NMR group
> > Institut de Biologie Structurale Jean-Pierre Ebel (IBS)
> > 41, rue Jules Horowitz
> > F-38027 Grenoble
> > France
> > +33 438 78 95 55
> > paul.schanda@xxxxxx
> > http://www.ibs.fr/groups/biomolecular-nmr-spectroscopy?lang=en