>>SSH tunnels is probably not the best option for your system. Do you
>>know anything about MPI?
>
>I have read about MPI but have not implimented anything __YET__;-). >Also
>I have compiled some MPI based programs. It seems to a bit of a pig >and
>I don't think the low hanging fruit necessarily require that degree of
>fine grained distribution...
If this is any help, I've done what I think is some fairly exhaustive searching for python+mpi implementations recently. Note that I've never _actually_ used any of them for a project yet.
Scientific Python has an MPI interface, which is handy since it is already a relax dependancy. The drawback is that its documentation seems very geared toward those who already understand MPI reasonably well. The other drawback is that is seems to be only able to pass Numpy arrays and strings between nodes, which would mean some relax data structures would probably need to be 'repackaged' for sending via MPI.
http://starship.python.net/~hinsen/ScientificPython/ScientificPythonManual/Scientific_27.htmlAnother one is:
MYMPI -
http://peloton.sdsc.edu/~tkaiser/mympi/ (and
http://grid-devel.sdsc.edu/gridsphere/gridsphere?cid=mympi ) - syntax
intended to match C MPI API closely, and much like Scientific.MPI only
has direct support for some basic data types, not arbitrary python
objects.
Most other implementations (below) support transmission of any python object that can be pickled, and so
may take less code to implement in relax. However, sending the whole data object
when only select parts of it are required for the calculation could be
more inefficient than you would like, and so 'repackaging' and sending
just what is needed may be better anyway. I wonder which is worse in this case .. the network overhead of sending a large-ish python object, or the extra load on the 'master' node as it repackages it to smaller Numpy array ..?? Guess it all depends on whether things are carved up 'batchwise' or more fine-grained (inner loop/function level).
MMPI -
http://www.penzilla.net/mmpi/ - looks to be actively developed, good documentation with examples, including sending of python objects via pickling.
pyPar -
http://datamining.anu.edu.au/~ole/work/software/pypar/ - sends
abitrary python objects, only two GPL licensed files so would be very
easy to package directly with relax rather than make users chase
dependancies.
There are also two which are parallel python interpreters that require recompilation, and seem to work a bit differently (still getting my head around exactly how these are meant to be used).
http://www.cimec.org.ar/python/ - a parallel interpreter as well as also some MPI bindings for
python. I tested the interpreter with relax and LAM/MPI, seemed to spawn off lots of processes and run.
pyMPI -
http://pympi.sourceforge.net/index.html - a parallel python
interpreter, decent docs at (
http://heanet.dl.sourceforge.net/sourceforge/pympi/pyMPI.pdf ), seems
mature despite out of data website.
There is also:
MPY -
http://mpy.sourceforge.net/index.html (seems abandoned since 2004)
Hope this helps ...
Andrew
Re: multi processing