1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25 """The multi-processor package.
26
27 1 Introduction
28 ==============
29
30 This package is an abstraction of specific multi-processor implementations or fabrics such as MPI via mpi4py. It is designed to be extended for use on other fabrics such as grid computing via SSH tunnelling, threading, etc. It also has a uni-processor mode as the default fabric.
31
32
33 2 API
34 =====
35
36 The public API is available via the __init__ module. It consists of a number of functions and classes. Using this basic interface, code can be parallelised and executed via an MPI implementation, or default back to a single CPU when needed. The choice of processor fabric is up to the calling program (via multi.load_multiprocessor).
37
38
39 2.1 Program initialisation
40 --------------------------
41
42 The function multi.load_multiprocessor() is the interface for how a program can load and set up a specific processor fabric. This function returns the set up processor, which itself provides a run() method which is used to execute your application.
43
44
45 2.2 Access to the processor instance
46 ------------------------------------
47
48 The multi.Processor_box class is a special singleton object which provides access to the processor object. This is required for a number of actions:
49
50 - Queuing of slave commands and memos via Processor_box().processor.add_to_queue().
51 - Returning results (as a Results_command) from the slave processor to the master via Processor_box().processor.return_object().
52 - Determining the number of processes via Processor_box().processor.processor_size().
53 - Waiting for completion of the queued slave processors via Processor_box().processor.run_queue().
54
55
56 2.3 Slaves
57 ----------
58
59 Slave processors are created via the multi.Slave_command class. This is special base class which must be subclassed. The run() function should be overridden, this provides the code to execute on the slave processors.
60
61
62 2.4 Results handling
63 --------------------
64
65 The multi.Result_command class is a special base class which must be subclassed. The run() function should be overridden, this provides the code for the master to process the results from the slaves.
66
67 In addition, the multi.Memo should also be used. This is a special base class which must be subclassed. This is a data store used by the Results_command to help process the results from the slave on the master processor.
68
69
70
71 3 Parallelisation
72 =================
73
74 The following are the steps required to parallelise a calculation via the multi-processor package API. It is assumed that the multi.load_multiprocessor() function has been set up at the highest level so that the entire program will be executed by the returned processor's run() method.
75
76
77 3.1 Subclassing command and memo objects
78 ----------------------------------------
79
80 The first step is that the Slave_command, Result_command, and Memo classes need to be subclassed. The Slave_command.run() method must be provided and is used for running the calculations on the slave processors. The Result_command is used to unpack the results from the slave. It is initialised by the Slave_command itself with the results from the calculation as arguments of __init__(). Its run() method processes the results on the master processor. The Memo object holds data other than the calculation results required by the Result_command.run() method to process the results.
81
82
83 3.2 Initialisation and queuing
84 ------------------------------
85
86 The second step is to initialise the Slave_command and Memo and add these to the processor queue. But first access to the processor is required. The singleton multi.Processor_box should be imported, and the processor accessed with code such as::
87
88 # Initialise the Processor box singleton.
89 processor_box = Processor_box()
90
91 The slave command is then initialised and all required data by the slave for the calculation (via its run() method) is stored within the class instance. The memo is also initialised with its data required for the result command for processing on the master of the results from the slave. These are then queued on the processor::
92
93 # Queue the slave command and memo.
94 processor_box.processor.add_to_queue(command, memo)
95
96
97 3.3 Calculation
98 ---------------
99
100 To execute the calculations, the final part of the calculation code on the master must feature a call to::
101
102 processor_box.processor.run_queue().
103
104
105 4 Example
106 =========
107
108 See the script 'test_implementation.py' for a basic example of a reference, and full, implementation of the multi-processor package.
109
110
111 5 Issues
112 ========
113
114 For multi-core systems and Linux 2.6, the following might be required to prevent the master processor from taking 100% of one CPU core while waiting for the slaves:
115
116 # echo "1" > /proc/sys/kernel/sched_compat_yield
117
118 This appears to be an OpenMPI problem with late 2.6 Linux kernels.
119 """
120
121
122 __all__ = ['memo',
123 'misc',
124 'mpi4py_processor',
125 'multi_processor_base',
126 'processor',
127 'processor_io',
128 'result_commands',
129 'result_queue',
130 'slave_commands',
131 'uni_processor']
132
133
134 import sys as _sys
135 import traceback as _traceback
136
137
138 from multi.memo import Memo
139 from multi.misc import import_module as _import_module
140 from multi.misc import Verbosity as _Verbosity; _verbosity = _Verbosity()
141 from multi.result_commands import Result_command
142 from multi.slave_commands import Slave_command
143
144
145
146
147
149 """Load a multi processor given its name.
150
151 Dynamically load a multi processor, the current algorithm is to search in module multi for a
152 module called <processor_name>.<Processor_name> (note capitalisation).
153
154
155 @todo: This algorithm needs to be improved to allow users to load processors without altering the relax source code.
156
157 @todo: Remove non-essential parameters.
158
159 @param processor_name: Name of the processor module/class to load.
160 @type processor_name: str
161 @keyword verbosity: The verbosity level at initialisation. This can be changed during program execution. A value of 0 suppresses all output. A value of 1 causes the basic multi-processor information to be printed. A value of 2 will switch on a number of debugging printouts. Values greater than 2 currently do nothing, though this might change in the future.
162 @type verbosity: int
163 @return: A loaded processor object or None to indicate failure.
164 @rtype: multi.processor.Processor instance
165 """
166
167
168 if processor_name not in ['uni', 'mpi4py']:
169 _sys.stderr.write("The processor type '%s' is not supported.\n" % processor_name)
170 _sys.exit()
171
172
173 _verbosity.set(verbosity)
174
175
176 processor_name = processor_name + '_processor'
177 class_name = processor_name[0].upper() + processor_name[1:]
178 module_path = '.'.join(('multi', processor_name))
179
180
181 modules = _import_module(module_path)
182
183
184 if hasattr(modules[-1], class_name):
185 clazz = getattr(modules[-1], class_name)
186 else:
187 raise Exception("can't load class %s from module %s" % (class_name, module_path))
188
189
190 object = clazz(callback=callback, processor_size=processor_size)
191
192
193 processor_box = Processor_box()
194 processor_box.processor = object
195 processor_box.processor_name = processor_name
196 processor_box.class_name = class_name
197
198
199 return object
200
201
203 """API function for obtaining data from the Processor instance's data store.
204
205 This is for fetching data from the data store of the Processor instance. If run on the master, then the master's data store will be accessed. If run on the slave, then the slave's data store will be accessed.
206
207
208 @attention: No inter-processor communications are performed.
209
210 @keyword name: The name of the data structure to fetch.
211 @type name: str
212 @return: The value of the associated data structure.
213 @rtype: anything
214 """
215
216
217 processor_box = Processor_box()
218
219
220 return processor_box.processor.fetch_data(name=name)
221
222
224 """API function for obtaining the data store object from the Processor instance.
225
226 If run on the master, then the master's data store will be returned. If run on the slave, then the slave's data store will be returned.
227
228
229 @attention: No inter-processor communications are performed.
230
231 @return: The data store of the processor (of the same rank as the calling code).
232 @rtype: class instance
233 """
234
235
236 processor_box = Processor_box()
237
238
239 return processor_box.processor.data_store
240
241
243 """API function for sending data from the master to all slaves processors.
244
245 @attention: Inter-processor communications are performed.
246
247 @keyword name: The name of the data structure to store.
248 @type name: str
249 @keyword value: The data structure.
250 @type value: anything
251 """
252
253
254 processor_box = Processor_box()
255
256
257 processor_box.processor.send_data_to_slaves(name=name, value=value)
258
259
260
262 """Call backs provided to the host application by the multi processor framework.
263
264 This class allows for independence from the host class/application.
265
266 @note: B{The logic behind the design} the callbacks are defined as two attributes
267 self.init_master and self.handle_exception as handle_exception can be null (which is
268 used to request the use of the processors default error handling code). Note, however,
269 that a class with the equivalent methods would also works as python effectively handles
270 methods as attributes of a class. The signatures for the callback methods are documented
271 by the default methods default_init_master & default_handle_exception.
272 """
273
275 """Initialise the callback interface.
276
277 @param master: The data for the host application. In the default implementation this is an
278 object we call methods on but it could be anything...
279 @type master: object
280 """
281
282 self.master = master
283 """The host application."""
284
285 self.init_master = self.default_init_master
286 self.handle_exception = self.default_handle_exception
287
288
290 """Handle an exception raised in the processor framework.
291
292 The function is responsible for aborting the processor by calling processor.abort() as its
293 final act.
294
295 @param processor: The processor instance.
296 @type processor: multi.processor.Processor instance
297 @param exception: The exception raised by the processor or slave processor. In the case of
298 a slave processor exception this may well be a wrapped exception of type
299 multi.processor.Capturing_exception which was raised at the point the
300 exception was received on the master processor but contains an enclosed
301 exception from a slave.
302 @type exception: Exception instance
303 """
304
305
306 _traceback.print_exc(file=_sys.stderr)
307
308
309 processor.abort()
310
311
313 """Start the main loop of the host application.
314
315 @param processor: The processor instance.
316 @type processor: multi.processor.Processor instance
317 """
318
319 self.master.run()
320
321
322
324 """A storage class for the Processor instance and its attributes.
325
326 This singleton contains Processor instances and information about these Processors. Importantly
327 this container gives the calling code access to the Processor.
328 """
329
330
331 instance = None
332
333 - def __new__(self, *args, **kargs):
334 """Replacement function for implementing the singleton design pattern."""
335
336
337 if self.instance is None:
338 self.instance = object.__new__(self, *args, **kargs)
339
340
341 return self.instance
342