Hi Troels,
Here is a bugfixed script. I didn't realise that nmrglue was using
single and not double precision for its data. So The dtype has to be
set to float32 when converting back:
"""
import base64
import numpy as np
pipe.create(pipe_name='mf', pipe_type='mf', bundle=None)
spectrum.nmrglue_read(file='freq_real.ft2',
dir='/data/relax/branches/nmrglue/extern/nmrglue/nmrglue_0_4/tests/pipe_proc_tests',
nmrglue_id='test')
a = cdp.nmrglue_data['test']
a_shape = a.shape
s = base64.b64encode(a)
file = open('test', 'w')
file.write(s)
r = base64.decodestring(s)
q=np.frombuffer(r,dtype=np.float32)
print(q.shape)
np.reshape(q, a_shape)
print(q.shape)
"""
Regards,
Edward
On 8 December 2014 at 11:50, Edward d'Auvergne <edward@xxxxxxxxxxxxx> wrote:
Hi Troels,
If you need help with any of this XML stuff, just say. It's a rather
different concept, but the code itself is not complex. Adding classes
to the relax data store simply involves using the base classes with
to_xml() and from_xml() methods in data_store.data_classes, and then
adding something to the pipe or spin container from_xml() method to
recreate the class in the data store.
The base64 functions are insanely quick. Try the following script on
your branch to see:
"""
import base64
import numpy as np
pipe.create(pipe_name='mf', pipe_type='mf', bundle=None)
spectrum.nmrglue_read(file='freq_real.ft2',
dir='/data/relax/branches/nmrglue/extern/nmrglue/nmrglue_0_4/tests/pipe_proc_tests',
nmrglue_id='test')
a = cdp.nmrglue_data['test']
s = base64.b64encode(a)
file = open('test', 'w')
file.write(s)
r = base64.decodestring(s)
q=np.frombuffer(r,dtype=np.float64)
"""
The same logic would be used to output the string 's' into the XML
file, and convert it back. This creates a file of 8.0 Mb for a total
time of ~1 second (with half of this spent on the
spectrum.nmrglue_read user function).
Regards,
Edward
On 8 December 2014 at 10:11, Edward d'Auvergne <edward@xxxxxxxxxxxxx> wrote:
Here is the Python documentation for this:
https://docs.python.org/2/library/base64.html
Regards,
Edward
On 8 December 2014 at 10:09, Edward d'Auvergne <edward@xxxxxxxxxxxxx>
wrote:
Hi Troels,
This is a standard problem of using text files and XML for universal
compatibility. However built into the XML format is the ability to
store any text between the tags. For example in this case we probably
need Base64:
http://www.xml.com/pub/a/98/07/binary/binary.html
https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64.html
This is many orders of magnitude quicker. This is the text format
used for all binary email attachments, for example. The only problem
here is converting the nmrglue numpy format to Base64, and back again.
Maybe you could ask Jonathan about it at
https://groups.google.com/forum/#!forum/nmrglue-discuss ? Although a
quick Google search finds this very simple Python solution:
https://stackoverflow.com/questions/6485790/numpy-array-to-base64-and-back-to-numpy-array-python
The relax data store to_xml() and from_xml() methods would need to be
modified to catch this data type. The to_xml() would have to set the
XML type attribute to 'base64', or '%s, base64'%repr(value.dtype).
See lib.xml.object_to_xml(). And the from_xml() method modified to
catch this and decode the Base64 string. This will require shifting
the lib.spectrum.object.Assignment class into
data_store.nmrglue.Assignment and then writing a custom to_xml()
method (maybe by a helper function in lib.xml). The decoding of the
Base64 numpy ndarray data could be part of the general function in
lib.xml.xml_to_object(). We should not make this general for all
numpy data, as the relax data store Python and XML structures are
designed for power users to be able read, and to be able to parse and
manipulate. However Base64 can be used for certain large data
structures for speed and size efficiency, with nmrglue data being the
prime example.
Regards,
Edward
On 7 December 2014 at 23:43, <tlinnet@xxxxxxxxxxxxx> wrote:
Author: tlinnet
Date: Sun Dec 7 23:43:06 2014
New Revision: 27007
URL: http://svn.gna.org/viewcvs/relax?rev=27007&view=rev
Log:
Change to systemtest Nmrglue.test_save_state.
This now tries to save a 8MB numpy array to a state file.
The numpy array has shape 512x4096.
numpy.save store the array instantly, while saving the state, hungs up
the computer for a long time.
This shows that relax is really in-efficient in storing data.
This both accounts for simple numpy array, and for storing class
instances.
Task #7873 (https://gna.org/task/index.php?7873): Write wrapper function
to nmrglue, to read .ft2 files and process them.
Homepage: http://www.nmrglue.com/
Link to nmrglue discussion:
https://groups.google.com/forum/#!forum/nmrglue-discuss
The code is develop at Github: https://github.com/jjhelmus/nmrglue/
Documentation: http://nmrglue.readthedocs.org/en/latest/index.html
Modified:
branches/nmrglue/test_suite/system_tests/nmrglue.py
Modified: branches/nmrglue/test_suite/system_tests/nmrglue.py
URL:
http://svn.gna.org/viewcvs/relax/branches/nmrglue/test_suite/system_tests/nmrglue.py?rev=27007&r1=27006&r2=27007&view=diff
==============================================================================
--- branches/nmrglue/test_suite/system_tests/nmrglue.py (original)
+++ branches/nmrglue/test_suite/system_tests/nmrglue.py Sun Dec 7
23:43:06 2014
@@ -373,7 +373,7 @@
print("Filesize of .npy file is: %i"%(data_numpy_size) )
# Delete the large data array, for faster saving.
- cdp.nmrglue_data[sp_id] = 0
+ #cdp.nmrglue_data[sp_id] = 0
# Store the directory path, before reset of the controller.
dirpath = ds.tmpdir
_______________________________________________
relax (http://www.nmr-relax.com)
This is the relax-commits mailing list
relax-commits@xxxxxxx
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-commits
Re: r27007 - /branches/nmrglue/test_suite/system_tests/nmrglue.py