-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lazy_parallellize having trouble with function context? #164
Comments
Thanks for the issue report. The reason for dill/cloudpickle is that there are quite a few types that python can't pickle but that those can. I think the solution here is probably making it easy to specify which pickler (including a no-op "pickler"). I'd be open to a PR that implements this and leaves the current defaults. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Just getting back to this one, found a very nasty bug / interaction with Spark with Python 3.7 due to pyfunctional loading dill. I will work on a PR to specify the pickler. Can you expand on how a no-op pickler option would work? I quickly hacked together import pickle as serializer. What kind of tests would you suggest to make sure that other picklers work well? Thank you. |
I'm using a function defined in the current file in pseq, and seems it errors out not being able to find other referenced functions or even simple types like Dict. This works fine when using seq.
I think the problem is with pickling the target function in lazy_parallelize:
I executed on my own the function with pool.imap and works fine.
Wouldn't it be better not to use pickling to avoid these kind of problems?
The text was updated successfully, but these errors were encountered: