Numba JIT causes up to 15s import times

[fluffbucket]

Compiling sGrid and whatever else uses Numba takes up to 15seconds every time causing long bootup times, consider using AOT instead with Numba.

Measured with cProfile around importing pysheds grid and raster

[mdbartos]

Hi @fluffbucket, all of the functions use numba's caching option to store the compiled code:

pysheds/pysheds/_sgrid.py

Line 11 in 32fb2ef

cache=True)

It should just compile once the first time you use it and then not compile again unless modified.

[fluffbucket]

Indeed that's what it does, however even loading in the already cached methods takes that long which is why I reported it. Try measuring it.

…

On Wed, 21 Jun 2023 at 18:01, Matt Bartos ***@***.***> wrote: Hi @fluffbucket <https://github.com/fluffbucket>, all of the functions use numba's caching option to store the compiled code: https://github.com/mdbartos/pysheds/blob/32fb2efba94c5da4ee8e2556d484a218d352226b/pysheds/_sgrid.py#L11 It should just compile once the first time you use it and then not compile again unless modified. — Reply to this email directly, view it on GitHub <#222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATH2XB7WO6N4GC4TVLELOKDXMMLFNANCNFSM6AAAAAAZOIZ7HY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[wiesnerfriedman]

Hi @fluffbucket I'm wondering if you're using Windows? I'm having issues using from pysheds.grid import Grid and it seems to crash in _sgrid.py around def _d8_catchment_iter_numba. I'm thinking it's a Windows thing maybe?

[mdbartos]

I've noticed numba sometimes has problems on Windows with the functions that use numba implementations of linked lists and dictionaries. You may want to try using algorithm='recursive' in your call to grid.catchment and see if it's more stable.

I haven't been able to reproduce the 15+ second loading times though. I suspect that you may have a caching issue that is causing everything to be re-compiled each time you import.