Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add feature to override harikiri shutdown on verdi job execution #20

Open
pymonger opened this issue Jan 18, 2019 · 0 comments
Open

add feature to override harikiri shutdown on verdi job execution #20

pymonger opened this issue Jan 18, 2019 · 0 comments

Comments

@pymonger
Copy link
Contributor

Currently:

  1. harikiri.py will see that no jobs came in for the last 10 minutes and flags the instance to terminate
  2. harikiri.py does a random sleep to smear out thundering herd api calls to AWS
  3. during the sleep, a job is queued that the verdi worker is pulling from and start processing that job
  4. after waking from the sleep, harikiri.py gracefully shuts down all docker images and supervisord
  5. The shut down of docker images is caught by verdi and thus results in a job-failed with exit code 143

Task is to add feature at start of each job to check for and cancel harikiri if it's in the middle of termination.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant