-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Irregular behavior with large numbers of files and quick running modules #226
Comments
Thanks for your feedback @jean997. Any chance you could share a small example DSC that generates the error on your end? |
Thanks Peter. I'll see if I can come up with something simple. |
@jean997 indeed something simple but concrete would help. After you increase it to 1000 instances per job, how many jobs in total do you still end up having? I would expect less errors when you increase it for per job ... We would like to review your other configurations and I can try on my end too if we have a simpler example. |
I've noticed that when I have a module like an evaluation module that does something very quick like subtracting two numbers, DSC tends to hang up or fail. The two most common things that I see are an error message that a job was killed or DSC just stopping and hanging until I kill it with no error message.
In the configuration file, I have tried setting the small modules to run with 100 or 500 instances per job to reduce the amount of jobs. I got more errors when I increased this to 1000 (these were mostly errors saying that a job had been killed).
Is there something I can do to improve this? I've currently modified my pipeline so that each analysis step also does the simple evaluation step but this isn't ideal because it requires lots of copying code.
Thanks!
Jean
The text was updated successfully, but these errors were encountered: