Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH - Downtime recovery (persistance, crash recovery, process restart) #203

Open
Leikt opened this issue Jun 7, 2023 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@Leikt
Copy link

Leikt commented Jun 7, 2023

Is your feature request related to a problem? Please describe.
We use schedulers in production systems. During the patching operations (or any other downtime of the server), the schedulers are shutted down. Consider the following scenario:

  • 08:03: the task "X" runs according to the schedule daily between 08:00 and 18:00
  • 09:00: the Rocketry process is shutted down for patching (server reboot)
  • 11:00: the Rocketry process is restarted after the patching
  • 11:03: the task "X" runs again

The problem: if there is any downtime, the tasks will run again.

Describe the solution you'd like
We need to be able to stop and restart the Rocketry process (not only the session) at any time. Therefore, we should be able to export/import session to a persistent file/database.
Something like

# On clean shutdown
session_data = session.export()
Path('session_state.json').write_text(json.dump(session_data))
# On start
session_data = json.loads(Path('session_state.json').read_text())
session.import(session_data)

That is just a quick though to give the idea, but a better way to handle this may exists

Describe alternatives you've considered
I considered writing a custom export and import parsing the logs to retrieve the last_run values of the tasks, store them in a file. Then, before app.run(), loading the file and set the tasks last_run values.
The problem with this is that it requires to dive deep into the Rocketry code and weaken my code (it would break easily with a change like #147 )

Additional context
I'm an infrastructure engineer with a tendency for development. I'm considering creating a scheduling tool to replace our current one. It's not bendy, not user friendly and certainly expensive. Before i officially submit the project to the managers, i'm working on a prototype (just selecting the packages for now)

PS: Thanks for the amazing work you've done on this package!

@Leikt Leikt added the enhancement New feature or request label Jun 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant