python - Multiprocessing incompatible with NumPy -
this question has answer here:
i trying run simple test using multiprocessing. test works until import numpy (even though not used in program). here code:
from multiprocessing import pool import time import numpy np #this problematic line def costlyfunc(n): """""" tstart = time.time() x = 0 in xrange(n): j in xrange(n): if % 2: x += 2 else: x -= 2 print "costlyfunc : elapsed time %f s" % (time.time() - tstart) return x #serial application resultlist0 = [] starttime = time.time() in xrange(3): resultlist0.append(costlyfunc(5000)) print "elapsed time (serial) : ", time.time() - starttime #multiprocessing application starttime = time.time() pool = pool() asyncresult = pool.map_async(costlyfunc, [5000, 5000, 5000]) resultlist1 = asyncresult.get() print "elapsed time (multiporcessing) : ", time.time() - starttime if don't import numpy result is:
costlyfunc : elapsed time 2.866265 s costlyfunc : elapsed time 2.793213 s costlyfunc : elapsed time 2.794936 s elapsed time (serial) : 8.45455098152 costlyfunc : elapsed time 2.889815 s costlyfunc : elapsed time 2.891556 s costlyfunc : elapsed time 2.898898 s elapsed time (multiporcessing) : 2.91595196724 the total elapsed time similar time required 1 process, meaning computation has been parallelized. if import numpy result becomes :
costlyfunc : elapsed time 2.877116 s costlyfunc : elapsed time 2.866778 s costlyfunc : elapsed time 2.860894 s elapsed time (serial) : 8.60492110252 costlyfunc : elapsed time 8.450145 s costlyfunc : elapsed time 8.473006 s costlyfunc : elapsed time 8.506402 s elapsed time (multiporcessing) : 8.55398178101 the total time elapsed same both serial , multiprocessing methods because 1 core used. clear problem comes numpy. possible have incompatibility between versions of multiprocessing , numpy?
i using python2.7, numpy 1.6.2 , multiprocessing 0.70a1 on linux
(first post sorry if not formulated or alligned)
you can stop numpy use multithreading seting mkl_num_threads 1
under debian used:
export mkl_num_threads=1 source related stackoverflow post: python: how stop numpy multithreading?
result:
user@pc:~/tmp$ python multi.py costlyfunc : elapsed time 3.847009 s costlyfunc : elapsed time 3.253226 s costlyfunc : elapsed time 3.415734 s elapsed time (serial) : 10.5163660049 costlyfunc : elapsed time 4.218424 s costlyfunc : elapsed time 5.252429 s costlyfunc : elapsed time 4.862513 s elapsed time (multiporcessing) : 9.11713695526 user@pc:~/tmp$ export mkl_num_threads=1 user@pc:~/tmp$ python multi.py costlyfunc : elapsed time 3.014677 s costlyfunc : elapsed time 3.102548 s costlyfunc : elapsed time 3.060915 s elapsed time (serial) : 9.17840886116 costlyfunc : elapsed time 3.720322 s costlyfunc : elapsed time 3.950583 s costlyfunc : elapsed time 3.656165 s elapsed time (multiporcessing) : 7.399310112 i not sure if helps because guess want numpy run in parallel maybe try adjust number of threads numpy machine.
Comments
Post a Comment