Re: System test "Relax_fit.check_curve_fitting" -- September 16, 2011 - 18:44

Hi Ed

I have now had a bit more of a look at this so here are some comments

On Thu, Aug 25, 2011 at 2:21 PM, Edward d'Auvergne <edward@xxxxxxxxxxxxx> wrote:

Hi,

That's great that you have time, to have this working properly is
quite important.  I didn't realise how much of an issue the merging of
the multi-processor code and GUI code would be.  I have now merged the
multi-processor branch into the main 1.3 line, as well as the new GUI
branch, and that is where the problems are.  I think there are at
least 2 problems currently occurring.  The first has to do with the
GUI tests:

$ relax --gui-test

I looks like that there is a clash of the wxPython App.MainLoop() and
how the master processors of the multi package interact.  There
appears to be race conditions, even in the uni-processor fabric.  This
only occurs in the test-suite, I have just tested manual operation of
the GUI which works, so this makes things more difficult.  The problem
was not existent in the gui_testing branch (svn co
svn+ssh://bugman@xxxxxxxxxxx/svn/relax/branches/gui_testing/@14200)
until after the multi-processor code was merged.

I presume this problem is solved and it was due to either a deadlock or calling the wx event loop outside of its main thread. Looking at the source code there seems to be quite a bit of thread specific code(?) One of the aims of the multiprocessor frame-work was to avoid directly waiting on threads and locks etc. This is why the whole design is based round queues and callbacks as they are quite simple to deal with and don't directly expose any underlying threaded model.  If you would like me to expand on this more please do say. If there are still multi threaded issues please do say and give some details of how to cause event loop problems or deadlocks

The second problem is IO redirection.  This occurs in a number of
places in relax.  These include:

1)  The --log command line flag which causes STDOUT and STDERR to be
sent to file.

2)  The --tee command line flag which causes STDOUT and STDERR to be
sent both to file and to the terminal.

3)  The test suite.  The STDOUT and STDERR streams are caught and only
sent to STDERR if there is an error or failure in a test.

4)  The relax controller (part of the GUI).  This is a window to which
STDOUT and STDERR are directed.  In the test-suite mode, the streams
also output to the terminal.

5)  The multi-processor package.  There are two parts.  The first is
essentially a pre-filter which prepends certain tokens to the IO
stream (i.e. the 'M  |', 'M  E|', and 'S 1|' text).  I cannot see how
we can do this as 4) is always set up after 5).  So I am considering
removing this part.  It will make it more difficult with debugging,
but I can see no other way.

6)  The second part for the multi-processor package, which is
currently non-functional, is the catching of the IO streams of the
slave processes to send back to the master.  I will try to mimic the
relax controller code here and store all slave text as a list with
flags specifying whether it is STDOUT or STDERR.  Then the list can be
returned to the master at which point the text can be sent to the two
streams.

The problem is that at each point here, sys.stdout and sys.stderr are
replaced and the order in which this happens is impossible to change.
Well 4) will always be last.

I think the problem here is that the whole idea of replacing the std io streams multiple times is inflexible and painful. So I have come up with a strawman multiplexed io implimentation. The idea is that you replace stdio and stderr once and then insert IO elements to deal with the needs to block the output of io streams, record them and  create Tees etc  see what you think? Should we open a couple of new mail threads to discuss these things?

regards

gary

below id the multiplexed io system

'''

Created on 16 Sep 2011

@author: garyt

'''

import unittest

import sys

class BaseIOElement:

  

  def __init__(self,name=""):

    self.name = name

  

  def get_name(self):

    return self.name

  

    

class Multiplex_IO:

    def __init__(self,stream):

      self.stream = stream

      self.stack = []

    

    def add_stream(self,stream):

      self.stack.append(stream)

    

    def write(self, string):

      

      for stream in self.stack:

        string = stream.write(string)

        if string ==  None:

          break

        

      if string != None:

        self.stream.write(string)

    

    def get_plexers(self):

      return list(self.stack) 

    

    def set_plexers(self,stack):

      self.stack =  list(stack)

      

    def flush(self):

      self.stream.flush()

    

    def __str__(self):

      names = [plexer.get_name() for plexer in self.stack]

      return " ".join(names)

      

    

      

class PrependIO (BaseIOElement):

    def __init__(self, token="", name="prepend"):

      

      self.token = token

#      self.first_time =True

      

    def write(self,string):

        string = string.replace('\n', '\n' + self.token)

        # Handle the first line of output.

#        if self.first_time == True:

#            string = '\n' + self.token + string

#            self.first_time = False

        return string

      

class CaptureIO(BaseIOElement):

    def __init__(self,name="capture"):

        self.buffer = []

    

    def write(self,string):

        self.buffer.append(string)

        return string

      

    def get_value(self):

      return ''.join(self.buffer)

    

    def clear(self):

      self.buffer = []

 

class TeeIO(BaseIOElement):       

  def __init__(self,stream, name="tee"):

    self.stream = stream

    

  def write(self,string):

    self.stream.write(string)

    return string

    

class NullIO(BaseIOElement):

  

  def __init__(self):

    self.enabled = True

  

  def set_enaled(self, state):

    self.enabled = (state == True)

    

  def write(self,string):

    return None

  

class Test:

#class Test(unittest.TestCase):

    def setUp(self):

        pass

    def tearDown(self):

        pass

    def testPassThru(self):

        sys.stderr = Multiplex_IO(sys.stderr)

        

        sys.stderr.add_stream(PrependIO("wibble> "))

        my_capture = CaptureIO()

        sys.stderr.add_stream(my_capture)

        

        print >> sys.stderr, "test"

        print >> sys.stderr, "test 2"

        

        print

        print "==capture start==",

        print my_capture.get_value()

        print "==capture end=="

        my_capture.clear()

        print "==clear capture start==",

        print my_capture.get_value()

        print "==clear capture end=="

        

        my_tee = TeeIO(sys.stdout)

        sys.stderr.add_stream(my_tee)

        

        print >> sys.stderr, "woggle"

        

        sys.stderr.add_stream(NullIO())

        

        print >> sys.stderr, "you shouldn't see this"

        

        print sys.stderr

        

if __name__ == "__main__":

    #import sys;sys.argv = ['', 'Test.testName']

#    unittest.main()

    Test().testPassThru()

Regards,

Edward

On 25 August 2011 14:51, Gary Thompson <garyt@xxxxxxxxxxxxxxx> wrote:
> On 08/25/2011 11:40 AM, Edward d'Auvergne wrote:
>>
>> Hi,
>>
>> I'm working on this at the moment.  Unfortunately the main 1.3 line is
>> severely broken at the moment!  The new multi-processor package is
>> clashing with and causing relax to die hard on the test-suite and the
>> GUI.  The main cause is IO redirection of sys.stdout and sys.stderr.
>> The multi-processor code (even in uni-processor mode) is hijacking the
>> streams and the test suite and GUI do not know what to do anymore.
>> Once I eliminate all the IO redirection of the multi package, apart
>> from the IO capture on the slave processes which is non-functional
>> anyway, then the test-suite should be back in order.  I noticed you
>> performed an svnmerge, and this is likely causing the breakages in
>> your branch.
>>
>> Regards,
>>
>> Edward
>>
>>
>
> Hi Ed
> I do at last have time to work on this, would you like me to have a look
> at what is going on?
>
> which branch do I need to look at? and what are the specific problems
> with the io redirect that need to be addressed?
>
> regards
> gary
>
>
>> On 25 August 2011 12:23, Sébastien Morin<sebastien.morin@xxxxxxxxx>
>>  wrote:
>>>
>>> Hi Ed,
>>>
>>> While working on "inversion-recovery" branch, I realized that the system
>>> tests "Relax_fit.check_curve_fitting__exp_2param_neg" and
>>> "Relax_fit.check_curve_fitting__exp_3param_inv_neg" were failing. In
>>> order
>>> to pin point the problem, I checked on the main 1.3 branch (514439) and
>>> system test "Relax_fit.check_curve_fitting" also failed, with the same
>>> error:
>>>
>>> ==========
>>> ======================================================================
>>> ERROR: check_curve_fitting (test_suite.system_tests.relax_fit.Relax_fit)
>>> Check the results of the curve-fitting.
>>> ----------------------------------------------------------------------
>>>
>>> relax>  pipe.create(pipe_name='mf', pipe_type='mf')
>>> Traceback (most recent call last):
>>>  File
>>>
>>> "/Users/semor/Documents/pse-4/collaborations/relax/relax-1.3/test_suite/system_tests/relax_fit.py",
>>> line 60, in check_curve_fitting
>>>    self.assertEqual(cdp.curve_type, 'exp')
>>> AttributeError: 'PipeContainer' object has no attribute 'curve_type'
>>>
>>> ----------------------------------------------------------------------
>>> Ran 1 test in 0.001s
>>>
>>> FAILED (errors=1)
>>> ==========
>>>
>>> I tried changing the pipe type from "mf" to "relax_fit" (as it should
>>> be).
>>> This did not solve the issue...
>>>
>>> Any idea ?
>>> Thanks !
>>>
>>>
>>> Séb   :)
>>>
>>> --
>>> Sébastien Morin, Ph.D.
>>> Postdoctoral Fellow, S. Grzesiek NMR Laboratory
>>> Department of Structural Biology
>>> Biozentrum, Universität Basel
>>> Klingelbergstrasse 70
>>> 4056 Basel
>>> Switzerland
>>>
>>>
>>> _______________________________________________
>>> relax (http://nmr-relax.com)
>>>
>>> This is the relax-devel mailing list
>>> relax-devel@xxxxxxx
>>>
>>> To unsubscribe from this list, get a password
>>> reminder, or change your subscription options,
>>> visit the list information page at
>>> https://mail.gna.org/listinfo/relax-devel
>>>
>> _______________________________________________
>> relax (http://nmr-relax.com)
>>
>> This is the relax-devel mailing list
>> relax-devel@xxxxxxx
>>
>> To unsubscribe from this list, get a password
>> reminder, or change your subscription options,
>> visit the list information page at
>> https://mail.gna.org/listinfo/relax-devel
>
>
> --
> -------------------------------------------------------------------
> Dr Gary Thompson                  [Homans Lab Research Coordinator]
>
> Astbury Centre for Structural Molecular Biology,
> University of Leeds,
> Leeds, LS2 9JT, West-Yorkshire, UK             Tel. +44-113-3433024
> email: garyt@xxxxxxxxxxxxxxx                   Fax  +44-113-3431935
> -------------------------------------------------------------------
>
>

_______________________________________________
relax (http://nmr-relax.com)

This is the relax-devel mailing list
relax-devel@xxxxxxx

To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-devel