Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Irregular behavior with large numbers of files and quick running modules #226

Open
jean997 opened this issue May 18, 2022 · 3 comments
Open

Comments

@jean997
Copy link

jean997 commented May 18, 2022

I've noticed that when I have a module like an evaluation module that does something very quick like subtracting two numbers, DSC tends to hang up or fail. The two most common things that I see are an error message that a job was killed or DSC just stopping and hanging until I kill it with no error message.

In the configuration file, I have tried setting the small modules to run with 100 or 500 instances per job to reduce the amount of jobs. I got more errors when I increased this to 1000 (these were mostly errors saying that a job had been killed).

Is there something I can do to improve this? I've currently modified my pipeline so that each analysis step also does the simple evaluation step but this isn't ideal because it requires lots of copying code.

Thanks!
Jean

@pcarbo
Copy link
Member

pcarbo commented May 18, 2022

Thanks for your feedback @jean997. Any chance you could share a small example DSC that generates the error on your end?

@jean997
Copy link
Author

jean997 commented May 18, 2022

Thanks Peter. I'll see if I can come up with something simple.

@gaow
Copy link
Member

gaow commented May 19, 2022

@jean997 indeed something simple but concrete would help. After you increase it to 1000 instances per job, how many jobs in total do you still end up having? I would expect less errors when you increase it for per job ... We would like to review your other configurations and I can try on my end too if we have a simpler example.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants