Module processor

The processor class is the central class in the multi python multiprocessor framework.

Overview

The framework has two main responsibilities:

1. Process management - if needed the processor can create the slave processes it manages if they haven't been created by the operating system. It is also responsible for reporting exceptions and shutting down the multiprocessor in the face of errors.
2. Scheduling commands on the slave processors via an interprocess communication fabric (MPI, PVM, threads etc) and processing returned text and result commands.

Using the processor framework

Users of the processor framework will typically use the following methodology:

1. At application startup determine the name of the required processor implementation and the number of slave processors requested.
2. Create an Application_callback object. For example: relax_instance = Relax() callbacks = Application_callback(master=relax_instance)
3. Dynamically load a processor implementation using the name of the processor and the number of required slave processors. For example: processor = Processor.load_multiprocessor(relax_instance.multiprocessor_type, callbacks, processor_size=relax_instance.n_processors)
4. Call run on the processor instance returned above and handle all Exceptions. For example: processor.run()
5. After calling run, the processor will call back to Application_callback.init_master from which you should call you main program (Application_callback defaults to self.master.run()).
6. Once in the main program you should call processor.add_to_queue with a series of multi.Slave_command objects you wish to be run across the slave processor pool and then call processor.run_queue to actually execute the commands remotely while blocking. >>> example here...
7. Processor.Slave_commands will then run remotely on the slaves and any thrown exceptions and processor.result_commands queued to processor.return_object will be returned to the master processor and handled or executed. The slave processors also provide facilities for capturing the STDERR and STDOUT streams and returning their contents as strings for display on the master's STDOUT and STDERR streams (***more**?).

Extending the processor framework with a new interprocess communication fabric

The processor class acts as a base class that defines all the commands that a processor implementing a new inter processor communication fabric needs. All that is required is to implement a subclass of processor providing the required methods (of course as python provides dynamic typing and polymorphism 'duck typing' you can always implement a class with the same set of method and it will also work). Currently processor classes are loaded from the processor module and are modules with names of the form:

>>> multi.<type>_processor.<Type>_processor

where <Type> is the name of the processor with the correct capitalisation e.g.

>>> processor_name = 'mpi4py'
>>> callback = My_application-callback()
>>> proccesor_size = 6
>>> processor.load_multiprocessor(processor_name, callback, processor_size)

will load multi.mpi4py_processor.Mpi4py_Processor.

TODO

The following are yet to be implemented:

1. There is no ability of the processor to request command line arguments.
2. The processor can't currently be loaded from somewhere other than the multi directory.

Classes

Data_store
A special Processor specific data storage container.

Processor
The central class of the multi processor framework.

Variables

verbosity = None

__package__ = 'multi'

Imports: time, datetime, math, sys, Capturing_exception, raise_unimplemented, Verbosity, Threaded_result_queue, Redirect_text, Batched_result_command, Null_result_command, Result_exception, Slave_storage_command