Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share with workers a private copy of data structures set by main thread, stored on the namespace (ns) object #252

Open
eldipa opened this issue Jun 5, 2022 · 0 comments
Labels
enhancement something nice to have but it is not neither critical nor urgent

Comments

@eldipa
Copy link
Collaborator

eldipa commented Jun 5, 2022

Describe the feature you'd like
The namespace or ns object is used to communicate or share data structures between the main thread and the workers and between the workers.
These data structures need to be created by the sharer in order to be, well, really shared and if they are mutable, they must also be protected with a kind of a lock (RLock), also created by sharer.

This allows a communication back and forward between the main thread and the workers and between the workers.

But in some cases it is needed something simpler: a main thread share information with its worker in one way fashion and the workers don't need to communicate between themselves. This scenario makes a fully N-to-N shared data structure and the requirement of a lock totally unnecessary.

In this case byexample could allow to create non-shared data-structures like plain list, dict or int in the main thread and make them visible in the workers as private copies.

To ensure that this will work in threading and in multiprocessing environments, the private objects must be pickle-able and the copy process should be (take main's object)->pickle->(get the bytes)->unpickle->(get the copy object)

Additional context (optional)
The documentation of byexample does not explain what happen if ns is used to store objects not created by sharer. In principle we could forbid them or we could make it private. In any case it may break some 3rd party plugin but the impact should be quite low.

@eldipa eldipa added the enhancement something nice to have but it is not neither critical nor urgent label Jun 5, 2022
@eldipa eldipa modified the milestone: 11.0.0 Jun 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement something nice to have but it is not neither critical nor urgent
Projects
None yet
Development

No branches or pull requests

1 participant