Hi Troels,
This is a standard problem of using text files and XML for universal
compatibility. However built into the XML format is the ability to
store any text between the tags. For example in this case we probably
need Base64:
http://www.xml.com/pub/a/98/07/binary/binary.html
https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64.html
This is many orders of magnitude quicker. This is the text format
used for all binary email attachments, for example. The only problem
here is converting the nmrglue numpy format to Base64, and back again.
Maybe you could ask Jonathan about it at
https://groups.google.com/forum/#!forum/nmrglue-discuss ? Although a
quick Google search finds this very simple Python solution:
https://stackoverflow.com/questions/6485790/numpy-array-to-base64-and-back-to-numpy-array-python
The relax data store to_xml() and from_xml() methods would need to be
modified to catch this data type. The to_xml() would have to set the
XML type attribute to 'base64', or '%s, base64'%repr(value.dtype).
See lib.xml.object_to_xml(). And the from_xml() method modified to
catch this and decode the Base64 string. This will require shifting
the lib.spectrum.object.Assignment class into
data_store.nmrglue.Assignment and then writing a custom to_xml()
method (maybe by a helper function in lib.xml). The decoding of the
Base64 numpy ndarray data could be part of the general function in
lib.xml.xml_to_object(). We should not make this general for all
numpy data, as the relax data store Python and XML structures are
designed for power users to be able read, and to be able to parse and
manipulate. However Base64 can be used for certain large data
structures for speed and size efficiency, with nmrglue data being the
prime example.
Regards,
Edward
On 7 December 2014 at 23:43, <tlinnet@xxxxxxxxxxxxx> wrote:
Author: tlinnet
Date: Sun Dec 7 23:43:06 2014
New Revision: 27007
URL: http://svn.gna.org/viewcvs/relax?rev=27007&view=rev
Log:
Change to systemtest Nmrglue.test_save_state.
This now tries to save a 8MB numpy array to a state file.
The numpy array has shape 512x4096.
numpy.save store the array instantly, while saving the state, hungs up the
computer for a long time.
This shows that relax is really in-efficient in storing data.
This both accounts for simple numpy array, and for storing class instances.
Task #7873 (https://gna.org/task/index.php?7873): Write wrapper function
to nmrglue, to read .ft2 files and process them.
Homepage: http://www.nmrglue.com/
Link to nmrglue discussion:
https://groups.google.com/forum/#!forum/nmrglue-discuss
The code is develop at Github: https://github.com/jjhelmus/nmrglue/
Documentation: http://nmrglue.readthedocs.org/en/latest/index.html
Modified:
branches/nmrglue/test_suite/system_tests/nmrglue.py
Modified: branches/nmrglue/test_suite/system_tests/nmrglue.py
URL:
http://svn.gna.org/viewcvs/relax/branches/nmrglue/test_suite/system_tests/nmrglue.py?rev=27007&r1=27006&r2=27007&view=diff
==============================================================================
--- branches/nmrglue/test_suite/system_tests/nmrglue.py (original)
+++ branches/nmrglue/test_suite/system_tests/nmrglue.py Sun Dec 7
23:43:06 2014
@@ -373,7 +373,7 @@
print("Filesize of .npy file is: %i"%(data_numpy_size) )
# Delete the large data array, for faster saving.
- cdp.nmrglue_data[sp_id] = 0
+ #cdp.nmrglue_data[sp_id] = 0
# Store the directory path, before reset of the controller.
dirpath = ds.tmpdir
_______________________________________________
relax (http://www.nmr-relax.com)
This is the relax-commits mailing list
relax-commits@xxxxxxx
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-commits