type Processor

Known Subclasses:

• multi_processor_base.Multi_processor
• , uni_processor.Uni_processor

The central class of the multi processor framework.

This provides facilities for process management, command queueing, command scheduling, remote execution of commands, and handling of results and error from commands. The class is abstract and should be overridden to implement new interprocess communication methods, however, even then users are encouraged to override the more full implemented multi.multi_processor.Multi_processor class. Most users should instantiate instances of this class by calling the static method Processor.load_multiprocessor.

The class is designed to be subclassed and has abstract methods that a subclass needs to override. Methods which can be overridden are clearly marked with a note annotation stating that they can be overridden.

To Do:

• It maybe a good idea to separate out the features of the class that purely deal with the interprocess communication fabric.
• The processor can't currently harvest the required command line arguments from the current command line.

Instance Methods

__init__(self, processor_size, callback)
Initialise the processor.

abort(self)
Shutdown the multi processor in exceptional conditions - designed for overriding.

add_to_queue(self, command, memo=None)
Add a command for remote execution to the queue - an abstract method.

assert_on_master(self)
Make sure that this is the master processor and not a slave.

exit(self, status=0)
Exit the processor with the given status.

anything fetch_data(self, name=None)
Fetch the data structure of the given name from the data store.

str get_intro_string(self)
Get a string describing the multi processor - designed for overriding.

str get_name(self)
Get the name of the current processor - an abstract method.

list of 2 str get_stdio_pre_strings(self)
Get the strings used prepend STDOUT and STDERR dependant on the current rank.

str get_time_delta(self, start_time, end_time)
Utility function called to format the difference between application start and end times.

bool is_queued(self)
Determine if any slave commands are queued.

master_queue_command(self, command, dest)
Slave to master processor data transfer - send the result command from the slave.

Result_command instance master_receive_result(self)
Slave to master processor data transfer - receive the result command from the slave.

post_run(self)
Method called after the application main loop has finished - designed for overriding.

pre_run(self)
Method called before starting the application main loop - designed for overriding.

int processor_size(self)
Get the number of slave processors - designed for overriding.

int rank(self)
Get the rank of this processor - an abstract method.

str rank_format_string(self)
Get a formatted string with the rank of a slave.

int rank_format_string_width(self)
Get the width of the string designating the rank of a slave process.

return_object(self, result)
Return a result to the master processor from a slave - an abstract method.

run(self)
Run the processor - an abstract method.

run_command_globally(self, command)
Run the same command on all slave processors.

run_command_queue(self, queue)
Process all commands on the queue and wait for completion.

run_queue(self)
Run the processor queue - an abstract method.

send_data_to_slaves(self, name=None, value=None)
Transfer the given data from the master to all slaves.

stdio_capture(self)
Enable capture of the STDOUT and STDERR.

stdio_restore(self)
Restore the original STDOUT and STDERR streams.

Instance Variables

NULL_RESULT
Empty result command used by commands which do not return a result (a singleton?).

_processor_size
Number of slave processors available in this processor.

callback
Callback to interface to the host application

data_store
The processor data store.

grainyness
The number of sub jobs to queue for each processor if we have more jobs than processors.

threaded_result_processing
Flag for the handling of result processing via self.run_command_queue().

Method Details

__init__(self, processor_size, callback)
(Constructor)

Initialise the processor.

Parameters:

• processor_size (int) - The requested number of __slave__processors, if the number of processors is set by the environment (e.g. in the case of MPI via mpiexec -np <n-processors> on the command line the processor is free free to ignore this value. The default value from the command line is -1, and subclasses on receiving this value either raise and exception or determine the correct number of slaves to create (e.g. on a multi-cored machine using a threaded implementation the correct number of slaves would be equal to the number of cores available).
• callback (multi.processor.Application_callback instance) - The application callback which allows the host application to start its main loop and handle exceptions from the processor.

Overrides: object.__init__

abort(self)

Shutdown the multi processor in exceptional conditions - designed for overriding.

This method is called after an exception from the master or slave has been raised and processed and is responsible for the shutdown of the multi processor fabric and terminating the application. The functions should be called as the last thing that Application_callback.handle_exception does.

As an example of the methods use see Mpi4py_processor.abort which calls MPI.COMM_WORLD.Abort() to cleanly shutdown the mpi framework and remove dangling processes.

The default action is to call the special self.exit() method.

See Also:

multi.processor.Application_callback., multi.mpi4py_processor.Mpi4py_processor.abort()., mpi4py.MPI.COMM_WORLD.Abort().

add_to_queue(self, command, memo=None)

Add a command for remote execution to the queue - an abstract method.

Parameters:

• command (? subclass instance) - A command to execute on a slave processor.
• memo (Memo subclass instance) - A place to place data needed on command completion (e.g. where to save the results) the data stored in the memo is provided to Result_commands generated by the command submitted.

See Also:

multi.processor.Slave_command, multi.processor.Result_command, multi.processor.Memo

assert_on_master(self)

Make sure that this is the master processor and not a slave.

Raises:

• Exception - If not on the master processor.

exit(self, status=0)

Exit the processor with the given status.

This default method allows the program to drop off the end and terminate as it normally would - i.e. this method does nothing.

Parameters:

• status (int) - The program exit status.

fetch_data(self, name=None)

Fetch the data structure of the given name from the data store.

This can be run on the master or slave processors.

Parameters:

• name (str) - The name of the data structure to fetch.

Returns: anything

The value of the associated data structure.

get_intro_string(self)

Get a string describing the multi processor - designed for overriding.

The string should be suitable for display at application startup and should be less than 100 characters wide. A good example is the string returned by mpi4py_processor:

>>> MPI running via mpi4py with <n> slave processors & 1 master, mpi version = <x>.<y>

Returns: str

A string describing the multi processor.

See Also: multi.processor.mpi4py_processor.Mpi4py_processor.get_intro_string.

get_name(self)

Get the name of the current processor - an abstract method.

The string should identify the current master or slave processor uniquely but is purely for information and debugging. For example the mpi implementation uses the string <host-name>-<process-id> whereas the thread implementation uses the id of the current thread as provided by python.

Returns: str

The processor identifier.

get_stdio_pre_strings(self)

Get the strings used prepend STDOUT and STDERR dependant on the current rank.

For processors with only one slave the result should be ('', '') - designed for overriding.

Returns: list of 2 str

A list of two strings for prepending to each line of STDOUT and STDERR.

Note: The defaults are ('M S|', 'M E|') and ('NN S|' , 'NN E|') for masters and slaves respectively with NN replaced by the rank of the processor.

get_time_delta(self, start_time, end_time)

Utility function called to format the difference between application start and end times.

Parameters:

• start_time (float) - The time the application started in seconds since the epoch.
• end_time (float) - The time the application ended in seconds since the epoch.

Returns: str

The time difference in the format 'hours:minutes:seconds'.

To Do: Check my format is correct.

is_queued(self)

Determine if any slave commands are queued.

Returns: bool

True if slave commands are in the queue, False otherwise.

master_queue_command(self, command, dest)

Slave to master processor data transfer - send the result command from the slave.

This is invoked by the slave processor.

Parameters:

• command (Results_command instance) - The results command to send to the master.
• dest (int) - The destination processor's rank.

master_receive_result(self)

Slave to master processor data transfer - receive the result command from the slave.

This is invoked by the master processor.

Returns: Result_command instance

The result command sent by the slave.

post_run(self)

Method called after the application main loop has finished - designed for overriding.

The default implementation outputs the application runtime to STDOUT. All subclasses should call the base method as their last action via super(). Only called on the master on normal exit from the applications run loop.

pre_run(self)

Method called before starting the application main loop - designed for overriding.

The default implementation just saves the start time for application timing. All subclasses should call the base method via super(). Only called on the master.

processor_size(self)

Get the number of slave processors - designed for overriding.

Returns: int

The number of slave processors.

rank(self)

Get the rank of this processor - an abstract method.

The rank of the processor should be a number between 0 and n where n is the number of slave processors, the rank of 0 is reserved for the master processor.

Returns: int

The rank of the processor.

rank_format_string(self)

Get a formatted string with the rank of a slave.

Only called on slaves.

Returns: str

The string designating the rank of the slave.

rank_format_string_width(self)

Get the width of the string designating the rank of a slave process.

Typically this will be the number of digits in the slaves rank.

Returns: int

The number of digits in the biggest slave processor's rank.

return_object(self, result)

Return a result to the master processor from a slave - an abstract method.

Parameters:

• result (Result_string, Result_command or Exception instance) - A result to be returned to the master processor.

See Also:

multi.processor.Result_string., multi.processor.Resulf_command.

run(self)

Run the processor - an abstract method.

This function runs the processor main loop and is called after all processor setup has been completed. It does remote execution setup and teardown (via self.pre_run() and self.post_run()) round either side of a call to Application_callback.init_master.

See Also: multi.processor.Application_callback.

run_command_globally(self, command)

Run the same command on all slave processors.

Parameters:

• command (Slave_command instance) - A slave command.

See Also: multi.processor.processor.Slave_command.

run_command_queue(self, queue)

Process all commands on the queue and wait for completion.

Parameters:

• queue (list of Command instances) - The command queue.

run_queue(self)

Run the processor queue - an abstract method.

All commands queued with add_to_queue will be executed, this function causes the current thread to block until the command has completed.

send_data_to_slaves(self, name=None, value=None)

Transfer the given data from the master to all slaves.

Parameters:

• name (str) - The name of the data structure to store.
• value (anything) - The data structure.

stdio_capture(self)

Enable capture of the STDOUT and STDERR.

This is currently used to capture the IO streams of the slaves to return back to the master.

Instance Variable Details

callback

Callback to interface to the host application

See Also: Application_callback.