[flytekit/async] Context manager & general thread/coroutine safety

[wild-endeavor]

Overview

The flytekit sdk has been traditionally developed without much regard for thread-safety and simultaneous access of objects. With the introduction of async support and the ability to run multiple tasks concurrently if not in parallel, we should revisit these assumptions, fix and test bugs and limitations.

This is also likely going to be a problem with the new async workflows effort that will be done.

The context manager is a known issue but we should find others as well.

Context Manager

The context manager probably is the biggest offender here. This is a global list of context states and is appended to every time a new task is run, among other operations. This breaks in the new eager model for instance. If you have the following series of tasks.

@task
def t1():
    print("normal task")

@eager
def eg_1():
    some_other_task()

@eager
def eg_p():
    t1()
    eg_1()

because each of eg_p, t1, and eg_1 will try to set context by appending something to the list, the view of what is FlyteContextManager.current_context() will not make sense.

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes