Module processor
source code
The processor class is the central class in the multi python
multiprocessor framework.
Overview
The framework has two main responsibilities:
-
Process management - if needed the processor can create the slave
processes it manages if they haven't been created by the operating
system. It is also responsible for reporting exceptions and
shutting down the multiprocessor in the face of errors.
-
Scheduling commands on the slave processors via an interprocess
communication fabric (MPI, PVM, threads etc) and processing
returned text and result commands.
Using the processor framework
Users of the processor framework will typically use the following
methodology:
-
At application startup determine the name of the required processor
implementation and the number of slave processors requested.
-
Create an Application_callback object. For example: relax_instance
= Relax() callbacks = Application_callback(master=relax_instance)
-
Dynamically load a processor implementation using the name of the
processor and the number of required slave processors. For
example: processor =
Processor.load_multiprocessor(relax_instance.multiprocessor_type,
callbacks, processor_size=relax_instance.n_processors)
-
Call run on the processor instance returned above and handle all
Exceptions. For example: processor.run()
-
After calling run, the processor will call back to
Application_callback.init_master from which you should call you
main program (Application_callback defaults to self.master.run()).
-
Once in the main program you should call processor.add_to_queue
with a series of multi.Slave_command objects you wish to be run
across the slave processor pool and then call processor.run_queue
to actually execute the commands remotely while blocking.
>>> example here...
-
Processor.Slave_commands will then run remotely on the slaves and
any thrown exceptions and processor.result_commands queued to
processor.return_object will be returned to the master processor
and handled or executed. The slave processors also provide
facilities for capturing the STDERR and STDOUT streams and
returning their contents as strings for display on the master's
STDOUT and STDERR streams (***more**?).
Extending the processor framework with a new interprocess communication fabric
The processor class acts as a base class that defines all the
commands that a processor implementing a new inter processor
communication fabric needs. All that is required is to implement a
subclass of processor providing the required methods (of course as
python provides dynamic typing and polymorphism 'duck typing' you can
always implement a class with the same set of method and it will also
work). Currently processor classes are loaded from the processor module
and are modules with names of the form:
>>> multi.<type>_processor.<Type>_processor
where <Type> is the name of the processor with the correct
capitalisation e.g.
>>> processor_name = 'mpi4py'
>>> callback = My_application-callback()
>>> proccesor_size = 6
>>> processor.load_multiprocessor(processor_name, callback, processor_size)
will load multi.mpi4py_processor.Mpi4py_Processor.
TODO
The following are yet to be implemented:
-
There is no ability of the processor to request command line
arguments.
-
The processor can't currently be loaded from somewhere other than
the multi directory.
Data_store
A special Processor specific data storage container.
|
Processor
The central class of the multi processor framework.
|
|
verbosity = None
|
|
__package__ = ' multi '
|
Imports:
time,
datetime,
math,
sys,
Capturing_exception,
raise_unimplemented,
Verbosity,
Threaded_result_queue,
Redirect_text,
Batched_result_command,
Null_result_command,
Result_exception,
Slave_storage_command