Author: bugman
Date: Wed Jul 23 09:59:04 2014
New Revision: 24669
URL: http://svn.gna.org/viewcvs/relax?rev=24669&view=rev
Log:
Merged revisions 24652-24664,24668 via svnmerge from
svn+ssh://bugman@xxxxxxxxxxx/svn/relax/trunk
................
r24652 | bugman | 2014-07-22 18:37:02 +0200 (Tue, 22 Jul 2014) | 3 lines
Initialized merge tracking via "svnmerge" with revisions "1-23718" from
svn+ssh://bugman@xxxxxxxxxxx/svn/relax/branches/disp_spin_speed
................
r24653 | bugman | 2014-07-22 18:50:50 +0200 (Tue, 22 Jul 2014) | 348 lines
Merged revisions 23722-23742 via svnmerge from
svn+ssh://bugman@xxxxxxxxxxx/svn/relax/branches/disp_spin_speed
........
r23722 | tlinnet | 2014-06-07 21:43:16 +0200 (Sat, 07 Jun 2014) | 6 lines
Modified profiling script to have different number of NCYC points per
frequency.
This is to complicate the data, so any errournous reshaping of data is
discovered.
It is expected, that experiments can have different number of NCYC points
per spectrometer frequency.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23723 | tlinnet | 2014-06-07 21:43:19 +0200 (Sat, 07 Jun 2014) | 55 lines
Initial try to alter the target function calc_CR72_chi2.
This is the first test to restructure the arrays, to allow for higher
dimensional computation.
All numpy arrays have to have same shape to allow to multiply together.
The dimensions should be [ei][si][mi][oi][di]. [Experiment][spins][spec.
frq][offset][disp points].
This is complicated with number of disp point can change per spectrometer
frequency.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
This implementation brings a high overhead.
The first test shows no winning of time.
The creation of arrays takes all the time.
Checked on MacBook Pro
2.4 GHz Intel Core i5
8 GB 1067 Mhz DDR3 RAM.
Python Distribution -- Python 2.7.3 |EPD 7.3-2 (32-bit)|
Timing for:
3 fields, [600. * 1E6, 800. * 1E6, 900. * 1E6]
('sfrq: ', 600000000.0, 'number of cpmg frq', 15)
('sfrq: ', 800000000.0, 'number of cpmg frq', 20)
('sfrq: ', 900000000.0, 'number of cpmg frq', 22)
iterations of function call: 1000
Timed for simulating 1 or 100 clustered spins.
For TRUNK
1 spin:
ncalls tottime percall cumtime percall filename:lineno(function)
3000 0.267 0.000 0.313 0.000 cr72.py:100(r2eff_CR72)
1000 0.056 0.000 0.434 0.000
relax_disp.py:456(calc_CR72_chi2)
3000 0.045 0.000 0.061 0.000 chi2.py:32(chi2)
100 spins:
ncalls tottime percall cumtime percall filename:lineno(function)
300000 26.315 0.000 30.771 0.000 cr72.py:100(r2eff_CR72)
1000 5.498 0.005 42.660 0.043
relax_disp.py:456(calc_CR72_chi2)
300000 4.438 0.000 6.021 0.000 chi2.py:32(chi2)
TESTING
1 spin:
ncalls tottime percall cumtime percall filename:lineno(function)
19013 0.278 0.000 0.278 0.000
{numpy.core.multiarray.array}
1000 0.191 0.000 0.777 0.001
relax_disp.py:457(calc_CR72_chi2)
1000 0.147 0.000 0.197 0.000 cr72.py:101(r2eff_CR72)
3000 0.044 0.000 0.061 0.000 chi2.py:32(chi2)
100 spins:
ncalls tottime percall cumtime percall filename:lineno(function)
1504904 25.215 0.000 25.215 0.000
{numpy.core.multiarray.array}
1000 17.261 0.017 51.180 0.051
relax_disp.py:457(calc_CR72_chi2)
300000 4.637 0.000 6.310 0.000 chi2.py:32(chi2)
........
r23724 | tlinnet | 2014-06-07 21:43:21 +0200 (Sat, 07 Jun 2014) | 4 lines
Temporary changed the lib/dispersion/cr72.py function to unsafe state.
This change turns-off all the safety measures, since they have to be
re-implemented
for higher dimensional structures.
........
r23725 | tlinnet | 2014-06-07 22:36:00 +0200 (Sat, 07 Jun 2014) | 71 lines
Altered profiling script to report cumulative timings and save to
temporary files.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
This indeed shows that the efficiency has gone down:
-----
Checked on MacBook Pro
2.4 GHz Intel Core i5
8 GB 1067 Mhz DDR3 RAM.
Python Distribution -- Python 2.7.3 |EPD 7.3-2 (32-bit)|
Timing for:
3 fields, [600. * 1E6, 800. * 1E6, 900. * 1E6]
('sfrq: ', 600000000.0, 'number of cpmg frq', 15)
('sfrq: ', 800000000.0, 'number of cpmg frq', 20)
('sfrq: ', 900000000.0, 'number of cpmg frq', 22)
iterations of function call: 1000
Timed for simulating 1 or 100 clustered spins.
For TRUNK
1 spin:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.434 0.434 <string>:1(<module>)
1 0.001 0.001 0.434 0.434 pf:354(single)
1000 0.002 0.000 0.432 0.000 pf:340(calc)
1000 0.007 0.000 0.429 0.000
relax_disp.py:908(func_CR72_full)
1000 0.054 0.000 0.416 0.000
relax_disp.py:456(calc_CR72_chi2)
3000 0.256 0.000 0.300 0.000 cr72.py:100(r2eff_CR
100 spins:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 42.035 42.035 <string>:1(<module>)
1 0.002 0.002 42.035 42.035 pf:378(cluster)
1000 0.004 0.000 41.957 0.042 pf:340(calc)
1000 0.011 0.000 41.953 0.042
relax_disp.py:908(func_CR72_full)
1000 5.378 0.005 41.928 0.042
relax_disp.py:456(calc_CR72_chi2)
300000 25.942 0.000 30.276 0.000 cr72.py:100(r2eff_CR72)
300000 4.362 0.000 5.903 0.000 chi2.py:32(chi2)
TESTING
1 spin:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.792 0.792 <string>:1(<module>)
1 0.001 0.001 0.792 0.792 pf:354(single)
1000 0.002 0.000 0.789 0.001 pf:340(calc)
1000 0.011 0.000 0.787 0.001
relax_disp.py:966(func_CR72_full)
1000 0.187 0.000 0.769 0.001
relax_disp.py:457(calc_CR72_chi2)
26013 0.290 0.000 0.290 0.000
{numpy.core.multiarray.array}
1000 0.147 0.000 0.198 0.000 cr72.py:101(r2eff_CR72)
2001 0.002 0.000 0.086 0.000 numeric.py:167(asarray)
3000 0.045 0.000 0.062 0.000 chi2.py:32(chi2)
100 spins:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 46.821 46.821 <string>:1(<module>)
1 0.002 0.002 46.821 46.821 pf:378(cluster)
1000 0.004 0.000 46.744 0.047 pf:340(calc)
1000 0.017 0.000 46.740 0.047
relax_disp.py:966(func_CR72_full)
1000 15.422 0.015 46.708 0.047
relax_disp.py:457(calc_CR72_chi2)
1511904 23.475 0.000 23.475 0.000
{numpy.core.multiarray.array}
300000 4.175 0.000 5.668 0.000 chi2.py:32(chi2)
2001 0.004 0.000 4.804 0.002 numeric.py:167(asarray)
1000 0.010 0.000 2.438 0.002 fromnumeric.py:1774(amax)
1000 0.006 0.000 2.428 0.002 fromnumeric.py:32(_wrapit)
300000 0.353 0.000 1.493 0.000 fromnumeric.py:1379(sum)
1000 1.281 0.001 1.444 0.001 cr72.py:101(r2eff_CR72)
........
r23726 | tlinnet | 2014-06-07 23:18:15 +0200 (Sat, 07 Jun 2014) | 3 lines
Added print out of chi2 to profile script.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23727 | tlinnet | 2014-06-07 23:18:18 +0200 (Sat, 07 Jun 2014) | 3 lines
Moved the creation of special numpy structures outside target function.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23728 | tlinnet | 2014-06-08 13:14:34 +0200 (Sun, 08 Jun 2014) | 5 lines
Modified profiling script to calculate correct values when setting up
R2eff values.
This is to test, that the return of chi2 gets zero.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23729 | tlinnet | 2014-06-08 13:14:36 +0200 (Sun, 08 Jun 2014) | 31 lines
Removing looping over exp and offset indicies in calc_chi2. They are
always 0 anyway.
This brings a little speed.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
----
Profiling.
1000 iterations
100 spins
3 sfrq
('sfrq: ', 600000000.0, 'number of cpmg frq', 15, array([ 2., 6.,
10., 14., 18., 22., 26., 30., 34., 38., 42.,
46., 50., 54., 58.]))
('sfrq: ', 800000000.0, 'number of cpmg frq', 20, array([ 2., 6.,
10., 14., 18., 22., 26., 30., 34., 38., 42.,
46., 50., 54., 58., 62., 66., 70., 74., 78.]))
('sfrq: ', 900000000.0, 'number of cpmg frq', 22, array([ 2., 6.,
10., 14., 18., 22., 26., 30., 34., 38., 42.,
46., 50., 54., 58., 62., 66., 70., 74., 78., 82., 86.]))
('chi2 cluster:', 0.0)
TRUNK
ncalls tottime percall cumtime percall filename:lineno(function)
1000 5.221 0.005 53.578 0.054
relax_disp.py:456(calc_CR72_chi2)
BEFORE
ncalls tottime percall cumtime percall filename:lineno(function)
1000 14.871 0.015 43.084 0.043
relax_disp.py:494(calc_CR72_chi2)
AFTER removing looping over exp and offset indicies. They are always 0.
ncalls tottime percall cumtime percall filename:lineno(function)
1000 12.831 0.013 38.248 0.038
relax_disp.py:494(calc_CR72_chi2)
........
r23730 | tlinnet | 2014-06-08 13:14:38 +0200 (Sun, 08 Jun 2014) | 5 lines
In profiling script, moved up the calculation of values one level.
This is to better see the output of the profiling iterations for cr72.py.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23731 | tlinnet | 2014-06-08 19:48:23 +0200 (Sun, 08 Jun 2014) | 6 lines
Fix for calculation of the larmor frequency per spin in profiling script.
The frq loop should also be up-shifted.
It was now extracted as 0.0.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23732 | tlinnet | 2014-06-08 19:48:25 +0200 (Sun, 08 Jun 2014) | 7 lines
Re-inserted safety checks in lin/dispersion/cr72.py file.
This is re-inserted for the rank_1 cases.
This makes the unit-tests pass again.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23733 | tlinnet | 2014-06-08 19:48:27 +0200 (Sun, 08 Jun 2014) | 8 lines
Important fix for extracting the correct shape to create new arrays.
If using just one field, or having the same number of dispersion points,
the shape would extend to the dispersion number.
It would report [ei][si][mi][oi][di] when calling ndarray.shape.
Shape always has to be reported as: [ei][si][mi][oi].
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23734 | tlinnet | 2014-06-08 19:48:29 +0200 (Sun, 08 Jun 2014) | 3 lines
Made it easier to switch between single and cluster reporting in
profiling script.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23735 | tlinnet | 2014-06-08 19:48:31 +0200 (Sun, 08 Jun 2014) | 9 lines
Important fix for the creation of the multi dimensional pA numpy array.
It should be created as numpy.zeros([ei][si][mi][oi]) instead of
numpy.ones([ei][si][mi][oi]).
This allows for rapid testing of all dimensions with np.allclose(pA,
numpy.ones(dw.shape)).
pA can have missing filled out values, when the number of dispersion
points are different
per spectrometer frequency.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23736 | tlinnet | 2014-06-08 19:48:33 +0200 (Sun, 08 Jun 2014) | 18 lines
Added unit tests demonstrating edge cases 'no Rex' failures of the model
'CR72 full', for a clustered multi dimensional calculation.
This is implemented for one field.
This is to implement catching of math domain errors, before they occur.
These tests cover all parameter value combinations which result in no
exchange:
- dw = 0.0,
- pA = 1.0,
- kex = 0.0,
- dw = 0.0 and pA = 1.0,
- dw = 0.0 and kex = 0.0,
- pA = 1.0 and kex = 0.0,
- dw = 0.0, pA = 1.0, and kex = 0.0.
- kex = 1e5,
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23737 | tlinnet | 2014-06-08 19:48:35 +0200 (Sun, 08 Jun 2014) | 7 lines
Re-implemented safety checks in lib/dispersion/cr72.py.
This is now implemented for both rank-1 float array and of higher
dimensions.
This makes the unit tests pass for multi dimensional computing.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23738 | tlinnet | 2014-06-08 19:48:37 +0200 (Sun, 08 Jun 2014) | 17 lines
Added unit tests demonstrating edge cases 'no Rex' failures of the model
'CR72 full', for a clustered multi dimensional calculation.
This is implemented for three fields.
This is to implement catching of math domain errors, before they occur.
These tests cover all parameter value combinations which result in no
exchange:
- dw = 0.0,
- pA = 1.0,
- kex = 0.0,
- dw = 0.0 and pA = 1.0,
- dw = 0.0 and kex = 0.0,
- pA = 1.0 and kex = 0.0,
- dw = 0.0, pA = 1.0, and kex = 0.0.
- kex = 1e5,
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23739 | tlinnet | 2014-06-08 19:48:38 +0200 (Sun, 08 Jun 2014) | 5 lines
Changed that special numpy structure is also created for "CR72".
This makes most system tests pass.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23740 | tlinnet | 2014-06-08 22:22:22 +0200 (Sun, 08 Jun 2014) | 5 lines
Critical fix for the slicing of values in target function.
This makes system test: Relax_disp.test_sod1wt_t25_to_cr72 pass.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23741 | tlinnet | 2014-06-08 22:22:24 +0200 (Sun, 08 Jun 2014) | 6 lines
Added self.has_missing keyword in initialization of the Dispersion class.
This is to test once, per spin or cluster.
This saves a looping over the dipsersion points, when collection the data.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23742 | tlinnet | 2014-06-08 22:22:26 +0200 (Sun, 08 Jun 2014) | 6 lines
Created multi dimensional error and value numpy arrays.
This is to calculate the chi2 sum much faster.
Reordered the loop over missing data points, so it is only iniatiated if
missing points is detected.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
................
r24654 | bugman | 2014-07-22 18:52:58 +0200 (Tue, 22 Jul 2014) | 118 lines
Merged revisions 23745-23752,23754-23758 via svnmerge from
svn+ssh://bugman@xxxxxxxxxxx/svn/relax/branches/disp_spin_speed
........
r23745 | tlinnet | 2014-06-08 23:44:44 +0200 (Sun, 08 Jun 2014) | 8 lines
Swith the looping from spin->frq to frq->spin.
Since the number of dispersion points are the same for all spins, this
allows to move the calculation of pA and kex array one level up.
This saves alot of computation.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23746 | tlinnet | 2014-06-08 23:44:45 +0200 (Sun, 08 Jun 2014) | 3 lines
Changed all the creation of special numpy arrays to be of float64 type.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23747 | tlinnet | 2014-06-08 23:54:41 +0200 (Sun, 08 Jun 2014) | 5 lines
Moved the data filling of special numpy array errors and values, to
initialization of Dispersion class.
These values does not change, and can safely be stored outside.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23748 | tlinnet | 2014-06-08 23:56:36 +0200 (Sun, 08 Jun 2014) | 3 lines
Just a tiny little more speed, by removing temporary storage of chi2
calculation.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
r23749 | tlinnet | 2014-06-09 00:09:59 +0200 (Mon, 09 Jun 2014) | 25 lines
Changed all calls to numpy np.X functions to just the numpy function.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
Timing is now showing, 17% loss per single spin, but but 277 % gain on
100 spin.
3 fields, 1000 iterations.
1 spin
1 0.000 0.000 0.677 0.677 <string>:1(<module>)
1 0.001 0.001 0.677 0.677 pf:419(single)
1000 0.002 0.000 0.671 0.001 pf:405(calc)
1000 0.009 0.000 0.669 0.001
relax_disp.py:979(func_CR72_full)
1000 0.102 0.000 0.655 0.001
relax_disp.py:507(calc_CR72_chi2)
1003 0.160 0.000 0.365 0.000 cr72.py:101(r2eff_CR72)
23029 0.188 0.000 0.188 0.000
{numpy.core.multiarray.array}
4003 0.119 0.000 0.182 0.000 numeric.py:1862(allclose)
100 spin
1 0.000 0.000 19.783 19.783 <string>:1(<module>)
1 0.002 0.002 19.783 19.783 pf:441(cluster)
1000 0.004 0.000 19.665 0.020 pf:405(calc)
1000 0.013 0.000 19.661 0.020
relax_disp.py:979(func_CR72_full)
1000 6.541 0.007 19.634 0.020
relax_disp.py:507(calc_CR72_chi2)
916108 11.127 0.000 11.127 0.000
{numpy.core.multiarray.array}
1300 1.325 0.001 2.026 0.002 cr72.py:101(r2eff_CR72)
4300 0.495 0.000 0.634 0.000 numeric.py:1862(allclose)
........
r23750 | tlinnet | 2014-06-09 00:49:14 +0200 (Mon, 09 Jun 2014) | 3 lines
Removed unused import of numpy.
Task #7807 (https://gna.org/task/index.php?7807): Speed-up of dispersion
models for Clustered analysis.
........
[... 4304 lines stripped ...]
r24669 - in /branches/zooming_grid_search: ./ lib/dispersion/ lib/linear_algebra/ specific_analyses/relax_disp/ target_funct...