Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flytekit/async] Merge loop usage & do proper signal handling #6068

Open
2 tasks done
wild-endeavor opened this issue Dec 2, 2024 · 0 comments
Open
2 tasks done

[flytekit/async] Merge loop usage & do proper signal handling #6068

wild-endeavor opened this issue Dec 2, 2024 · 0 comments
Labels
backlogged For internal use. Reserved for contributor team workflow. Epic: Async Flytekit Enable asynchronous behavior in flytekit flytekit FlyteKit Python related issue

Comments

@wild-endeavor
Copy link
Contributor

wild-endeavor commented Dec 2, 2024

Describe the issue

Multiple loops and threads are used in the course of flytekit's async handling. Eager tasks themselves rely on a worker to launch executions of sub-flyte-entities against the control plane. Currently this latter loop is run outside of the main loop manager that flytekit uses. We should consider merging, and then implement proper loop lifecycle management for all loops, running on any thread. (i.e. calling .cancel() on all running tasks of each loop and handling canceled errors correctly in the relevant coroutines).

What if we do not do this?

Signal handling and async loop lifecycle management remains messy.

Are you sure this issue hasn't been raised already?

  • Yes

Have you read the Code of Conduct?

  • Yes
@wild-endeavor wild-endeavor added housekeeping Issues that help maintain flyte and keep it tech-debt free flytekit FlyteKit Python related issue Epic: Async Flytekit Enable asynchronous behavior in flytekit backlogged For internal use. Reserved for contributor team workflow. and removed housekeeping Issues that help maintain flyte and keep it tech-debt free labels Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backlogged For internal use. Reserved for contributor team workflow. Epic: Async Flytekit Enable asynchronous behavior in flytekit flytekit FlyteKit Python related issue
Projects
None yet
Development

No branches or pull requests

1 participant