Package multi :: Module processor :: Class Processor
[hide private]
[frames] | no frames]

Class Processor

source code


The central class of the multi processor framework.

This provides facilities for process management, command queueing, command scheduling, remote execution of commands, and handling of results and error from commands. The class is abstract and should be overridden to implement new interprocess communication methods, however, even then users are encouraged to override the more full implemented multi.multi_processor.Multi_processor class. Most users should instantiate instances of this class by calling the static method Processor.load_multiprocessor.

The class is designed to be subclassed and has abstract methods that a subclass needs to override. Methods which can be overridden are clearly marked with a note annotation stating that they can be overridden.


To Do:
Instance Methods [hide private]
 
__init__(self, processor_size, callback)
Initialise the processor.
source code
 
abort(self)
Shutdown the multi processor in exceptional conditions - designed for overriding.
source code
 
add_to_queue(self, command, memo=None)
Add a command for remote execution to the queue - an abstract method.
source code
 
assert_on_master(self)
Make sure that this is the master processor and not a slave.
source code
 
exit(self, status=0)
Exit the processor with the given status.
source code
anything
fetch_data(self, name=None)
Fetch the data structure of the given name from the data store.
source code
str
get_intro_string(self)
Get a string describing the multi processor - designed for overriding.
source code
str
get_name(self)
Get the name of the current processor - an abstract method.
source code
list of 2 str
get_stdio_pre_strings(self)
Get the strings used prepend STDOUT and STDERR dependant on the current rank.
source code
str
get_time_delta(self, start_time, end_time)
Utility function called to format the difference between application start and end times.
source code
bool
is_queued(self)
Determine if any slave commands are queued.
source code
 
master_queue_command(self, command, dest)
Slave to master processor data transfer - send the result command from the slave.
source code
Result_command instance
master_receive_result(self)
Slave to master processor data transfer - receive the result command from the slave.
source code
 
post_run(self)
Method called after the application main loop has finished - designed for overriding.
source code
 
pre_run(self)
Method called before starting the application main loop - designed for overriding.
source code
int
processor_size(self)
Get the number of slave processors - designed for overriding.
source code
int
rank(self)
Get the rank of this processor - an abstract method.
source code
str
rank_format_string(self)
Get a formatted string with the rank of a slave.
source code
int
rank_format_string_width(self)
Get the width of the string designating the rank of a slave process.
source code
 
return_object(self, result)
Return a result to the master processor from a slave - an abstract method.
source code
 
run(self)
Run the processor - an abstract method.
source code
 
run_command_globally(self, command)
Run the same command on all slave processors.
source code
 
run_command_queue(self, queue)
Process all commands on the queue and wait for completion.
source code
 
run_queue(self)
Run the processor queue - an abstract method.
source code
 
send_data_to_slaves(self, name=None, value=None)
Transfer the given data from the master to all slaves.
source code
 
stdio_capture(self)
Enable capture of the STDOUT and STDERR.
source code
 
stdio_restore(self)
Restore the original STDOUT and STDERR streams.
source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__

Instance Variables [hide private]
  NULL_RESULT
Empty result command used by commands which do not return a result (a singleton?).
  _processor_size
Number of slave processors available in this processor.
  callback
Callback to interface to the host application
  data_store
The processor data store.
  grainyness
The number of sub jobs to queue for each processor if we have more jobs than processors.
  threaded_result_processing
Flag for the handling of result processing via self.run_command_queue().
Properties [hide private]

Inherited from object: __class__

Method Details [hide private]

__init__(self, processor_size, callback)
(Constructor)

source code 

Initialise the processor.

Parameters:
  • processor_size (int) - The requested number of __slave__processors, if the number of processors is set by the environment (e.g. in the case of MPI via mpiexec -np <n-processors> on the command line the processor is free free to ignore this value. The default value from the command line is -1, and subclasses on receiving this value either raise and exception or determine the correct number of slaves to create (e.g. on a multi-cored machine using a threaded implementation the correct number of slaves would be equal to the number of cores available).
  • callback (multi.processor.Application_callback instance) - The application callback which allows the host application to start its main loop and handle exceptions from the processor.
Overrides: object.__init__

abort(self)

source code 

Shutdown the multi processor in exceptional conditions - designed for overriding.

This method is called after an exception from the master or slave has been raised and processed and is responsible for the shutdown of the multi processor fabric and terminating the application. The functions should be called as the last thing that Application_callback.handle_exception does.

As an example of the methods use see Mpi4py_processor.abort which calls MPI.COMM_WORLD.Abort() to cleanly shutdown the mpi framework and remove dangling processes.

The default action is to call the special self.exit() method.

See Also:
multi.processor.Application_callback., multi.mpi4py_processor.Mpi4py_processor.abort()., mpi4py.MPI.COMM_WORLD.Abort().

add_to_queue(self, command, memo=None)

source code 

Add a command for remote execution to the queue - an abstract method.

Parameters:
  • command (? subclass instance) - A command to execute on a slave processor.
  • memo (Memo subclass instance) - A place to place data needed on command completion (e.g. where to save the results) the data stored in the memo is provided to Result_commands generated by the command submitted.
See Also:
multi.processor.Slave_command, multi.processor.Result_command, multi.processor.Memo

assert_on_master(self)

source code 

Make sure that this is the master processor and not a slave.

Raises:
  • Exception - If not on the master processor.

exit(self, status=0)

source code 

Exit the processor with the given status.

This default method allows the program to drop off the end and terminate as it normally would - i.e. this method does nothing.

Parameters:
  • status (int) - The program exit status.

fetch_data(self, name=None)

source code 

Fetch the data structure of the given name from the data store.

This can be run on the master or slave processors.

Parameters:
  • name (str) - The name of the data structure to fetch.
Returns: anything
The value of the associated data structure.

get_intro_string(self)

source code 

Get a string describing the multi processor - designed for overriding.

The string should be suitable for display at application startup and should be less than 100 characters wide. A good example is the string returned by mpi4py_processor:

>>> MPI running via mpi4py with <n> slave processors & 1 master, mpi version = <x>.<y>
Returns: str
A string describing the multi processor.

See Also: multi.processor.mpi4py_processor.Mpi4py_processor.get_intro_string.

get_name(self)

source code 

Get the name of the current processor - an abstract method.

The string should identify the current master or slave processor uniquely but is purely for information and debugging. For example the mpi implementation uses the string <host-name>-<process-id> whereas the thread implementation uses the id of the current thread as provided by python.

Returns: str
The processor identifier.

get_stdio_pre_strings(self)

source code 

Get the strings used prepend STDOUT and STDERR dependant on the current rank.

For processors with only one slave the result should be ('', '') - designed for overriding.

Returns: list of 2 str
A list of two strings for prepending to each line of STDOUT and STDERR.

Note: The defaults are ('M S|', 'M E|') and ('NN S|' , 'NN E|') for masters and slaves respectively with NN replaced by the rank of the processor.

get_time_delta(self, start_time, end_time)

source code 

Utility function called to format the difference between application start and end times.

Parameters:
  • start_time (float) - The time the application started in seconds since the epoch.
  • end_time (float) - The time the application ended in seconds since the epoch.
Returns: str
The time difference in the format 'hours:minutes:seconds'.

To Do: Check my format is correct.

is_queued(self)

source code 

Determine if any slave commands are queued.

Returns: bool
True if slave commands are in the queue, False otherwise.

master_queue_command(self, command, dest)

source code 

Slave to master processor data transfer - send the result command from the slave.

This is invoked by the slave processor.

Parameters:
  • command (Results_command instance) - The results command to send to the master.
  • dest (int) - The destination processor's rank.

master_receive_result(self)

source code 

Slave to master processor data transfer - receive the result command from the slave.

This is invoked by the master processor.

Returns: Result_command instance
The result command sent by the slave.

post_run(self)

source code 

Method called after the application main loop has finished - designed for overriding.

The default implementation outputs the application runtime to STDOUT. All subclasses should call the base method as their last action via super(). Only called on the master on normal exit from the applications run loop.

pre_run(self)

source code 

Method called before starting the application main loop - designed for overriding.

The default implementation just saves the start time for application timing. All subclasses should call the base method via super(). Only called on the master.

processor_size(self)

source code 

Get the number of slave processors - designed for overriding.

Returns: int
The number of slave processors.

rank(self)

source code 

Get the rank of this processor - an abstract method.

The rank of the processor should be a number between 0 and n where n is the number of slave processors, the rank of 0 is reserved for the master processor.

Returns: int
The rank of the processor.

rank_format_string(self)

source code 

Get a formatted string with the rank of a slave.

Only called on slaves.

Returns: str
The string designating the rank of the slave.

rank_format_string_width(self)

source code 

Get the width of the string designating the rank of a slave process.

Typically this will be the number of digits in the slaves rank.

Returns: int
The number of digits in the biggest slave processor's rank.

return_object(self, result)

source code 

Return a result to the master processor from a slave - an abstract method.

Parameters:
  • result (Result_string, Result_command or Exception instance) - A result to be returned to the master processor.
See Also:
multi.processor.Result_string., multi.processor.Resulf_command.

run(self)

source code 

Run the processor - an abstract method.

This function runs the processor main loop and is called after all processor setup has been completed. It does remote execution setup and teardown (via self.pre_run() and self.post_run()) round either side of a call to Application_callback.init_master.

See Also: multi.processor.Application_callback.

run_command_globally(self, command)

source code 

Run the same command on all slave processors.

Parameters:
  • command (Slave_command instance) - A slave command.

See Also: multi.processor.processor.Slave_command.

run_command_queue(self, queue)

source code 

Process all commands on the queue and wait for completion.

Parameters:
  • queue (list of Command instances) - The command queue.

run_queue(self)

source code 

Run the processor queue - an abstract method.

All commands queued with add_to_queue will be executed, this function causes the current thread to block until the command has completed.

send_data_to_slaves(self, name=None, value=None)

source code 

Transfer the given data from the master to all slaves.

Parameters:
  • name (str) - The name of the data structure to store.
  • value (anything) - The data structure.

stdio_capture(self)

source code 

Enable capture of the STDOUT and STDERR.

This is currently used to capture the IO streams of the slaves to return back to the master.


Instance Variable Details [hide private]

callback

Callback to interface to the host application

See Also: Application_callback.