-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run stages in parallel #34
Comments
I managed that in one of my forked branches: develop...ainar:synpp:multiprocessing I encompassed each stage execution in a multiprocessing's Process. The processes are launched as soon as enough resources are available and the executions of their dependencies are done. In my fork, the total amount of resources and the amount of resources needed for each stage still need to be configurable. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Often, the tree structure of the pipelines allows to run things in parallel. Right now the pipeline runs one stage at a time. To make use of parallel computing power, a couple of steps are necessary:
parallel()
context, it should not already be in a child process! Therefore, we need to put some thoughts and intelligent management of the process pool. (In particular, it would need to be managed centrally by the pipeline instead of perParallelMasterContext
object).The text was updated successfully, but these errors were encountered: