Avoiding parallel threading in py-pde

[ArvinSubramaniam]

Hi,

My question is regarding the use of parallel threading in py-pde.

From what I can tell, it turns out that py-pde is using all cores for simulations (see aattached photo of CPU usage on my linux computer - all 8 cores are used), although it is not clear that this is necessary (since no parallel computing is used). This drastically slows down simulations on my device, and also the computing cluster that I use.

An upon adding the following lines:

from pde.tools import
numba.NUMBA_PARALLEL = False

the problem still remains (see second photo - granted they look the same but trust me they were taken from before and after)

The code is thus:

from pde import ScalarField,UnitGrid,PDE,FieldCollection,CartesianGrid,MemoryStorage,plot_kymograph,plot_kymographs

from pde.tools import *
numba.NUMBA_PARALLEL = False

n_grid = 256
t_start = time.time()
grid = CartesianGrid([[0,1],[0,1]],[n_grid,n_grid],periodic=True) ### Time step has to be smaller here

K1 = 2.1

K = K1/(n_grid**(2))

init_in = np.random.normal(0.3,0.2,(n_grid,n_grid))
init_ = ScalarField(grid,init_in)
init_.plot()

storage = MemoryStorage()
num_kym = 50
eq=PDE({"u":f"-{K}*laplace(laplace(u)) - laplace(u - u**(3))"})# define the pde
 

tend = 1
result2=eq.solve(init_, t_range=tend,tracker=[storage.tracker(tend/num_kym)])
result2.plot(title=r'$T={}$'.format(tend))

Is there a neat way enforce that a single core usage on the py-pde source code? Any help is much appreciated.

[david-zwicker]

I found a bug in py-pde, which prevented disabling multiprocessing. The current master branch should fix this. You should now be able to disable multiprocessing using the following construct

from pde.tools import numba
numba.NUMBA_PARALLEL = False

Could you please test whether this fixes the problem?

[david-zwicker]

Thanks for investigating this further. I think the culprit might be the function scipy.intergrate.solve_ivp, which is used for time-stepping when no fixed time step is supplied. I don't know how to control multi-processing in scipy, but you can switch to a Numba-compiled explicit time stepper by supplying a fixed time step using the dt argument in the call to eq.solve.

[ArvinSubramaniam]

Thanks for this. Indeed this helps.

Another (potneitally interesting finding) is that, even with dt specified, the parallelism seems to depend on the exact grid size, L. I've inlcuded belwo CPU usage for L=213 vs L=256. The former uses one core whereas the latter has heavy parallelism. Interestingly, paraellism for larger L=1024 is actually less than 256, for reasons I haven't figured out.

L=213

L=256

So to conclude, the best bet seems to be modest grid sizes L<256 that are not powers of 2.

[david-zwicker]

Yes, py-pde disables parallelism when evaluating differential operators on small grids. To test whether the multiprocessing is actually disabled, you thus need to look at grids that are large enough.

[david-zwicker]

Just as a quick follow-up. I changed the way settings are handled in the latest version of py-pde, so multi-processing should now be disabled using the following construct:

from pde import config
config['numba.parallel'] = False

Note that the old way might still work for about half a year, but it will be removed eventually.

[ArvinSubramaniam]

Thanks!