Hi Troels, This is a standard problem of using text files and XML for universal compatibility. However built into the XML format is the ability to store any text between the tags. For example in this case we probably need Base64: http://www.xml.com/pub/a/98/07/binary/binary.html https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64.html This is many orders of magnitude quicker. This is the text format used for all binary email attachments, for example. The only problem here is converting the nmrglue numpy format to Base64, and back again. Maybe you could ask Jonathan about it at https://groups.google.com/forum/#!forum/nmrglue-discuss ? Although a quick Google search finds this very simple Python solution: https://stackoverflow.com/questions/6485790/numpy-array-to-base64-and-back-to-numpy-array-python The relax data store to_xml() and from_xml() methods would need to be modified to catch this data type. The to_xml() would have to set the XML type attribute to 'base64', or '%s, base64'%repr(value.dtype). See lib.xml.object_to_xml(). And the from_xml() method modified to catch this and decode the Base64 string. This will require shifting the lib.spectrum.object.Assignment class into data_store.nmrglue.Assignment and then writing a custom to_xml() method (maybe by a helper function in lib.xml). The decoding of the Base64 numpy ndarray data could be part of the general function in lib.xml.xml_to_object(). We should not make this general for all numpy data, as the relax data store Python and XML structures are designed for power users to be able read, and to be able to parse and manipulate. However Base64 can be used for certain large data structures for speed and size efficiency, with nmrglue data being the prime example. Regards, Edward On 7 December 2014 at 23:43, <tlinnet@xxxxxxxxxxxxx> wrote:
Author: tlinnet Date: Sun Dec 7 23:43:06 2014 New Revision: 27007 URL: http://svn.gna.org/viewcvs/relax?rev=27007&view=rev Log: Change to systemtest Nmrglue.test_save_state. This now tries to save a 8MB numpy array to a state file. The numpy array has shape 512x4096. numpy.save store the array instantly, while saving the state, hungs up the computer for a long time. This shows that relax is really in-efficient in storing data. This both accounts for simple numpy array, and for storing class instances. Task #7873 (https://gna.org/task/index.php?7873): Write wrapper function to nmrglue, to read .ft2 files and process them. Homepage: http://www.nmrglue.com/ Link to nmrglue discussion: https://groups.google.com/forum/#!forum/nmrglue-discuss The code is develop at Github: https://github.com/jjhelmus/nmrglue/ Documentation: http://nmrglue.readthedocs.org/en/latest/index.html Modified: branches/nmrglue/test_suite/system_tests/nmrglue.py Modified: branches/nmrglue/test_suite/system_tests/nmrglue.py URL: http://svn.gna.org/viewcvs/relax/branches/nmrglue/test_suite/system_tests/nmrglue.py?rev=27007&r1=27006&r2=27007&view=diff ============================================================================== --- branches/nmrglue/test_suite/system_tests/nmrglue.py (original) +++ branches/nmrglue/test_suite/system_tests/nmrglue.py Sun Dec 7 23:43:06 2014 @@ -373,7 +373,7 @@ print("Filesize of .npy file is: %i"%(data_numpy_size) ) # Delete the large data array, for faster saving. - cdp.nmrglue_data[sp_id] = 0 + #cdp.nmrglue_data[sp_id] = 0 # Store the directory path, before reset of the controller. dirpath = ds.tmpdir _______________________________________________ relax (http://www.nmr-relax.com) This is the relax-commits mailing list relax-commits@xxxxxxx To unsubscribe from this list, get a password reminder, or change your subscription options, visit the list information page at https://mail.gna.org/listinfo/relax-commits