Union Types

[EngHabu]

from @maximsmol's RFC:

2 Motivation

Currently any type can take none values (see this comment in Propeller's sources). This creates a few unwanted outcomes:

• The type system does not enforce required workflow parameters as always having a valid value
• Collections are not in practice homogeneous since they may contain none values
  • Example: [1, 2, 3, null] is a valid list int value since int is implicitly nullable
• In Python the use of typing.Optional is a no-op and furthermore not supported by default as there is no type transformer for typing.Optional since it would be useless
• Examining the types of workflow parameters is not enough to determine whether the parameter is intended to be optional
  • This particular point affects Latch as we generate workflow interfaces based on type information
  • Collections are the most troublesome here as it is unclear whether list int is intended to take none-values (and thus whether the interface should allow them)

a very common example for needing full Union for us is mutually-exclusive parameters something like

 @task
 def mageck_mle(
   design_matrix_file: Optional[FlyteFile] = None, # CSV of labels
   day0_label: Optional[List[str]] = None, # labels themselves
   ...):

where this pipeline (called MAGECK) supports both getting experimental labels from a file and as a direct list (they are processed somewhat differently)
so we can have them encoded in this way with Optionals and somehow enforce separately that they are not both provided together (and that at least one is provided)
or we can use this instead:

 @task
 def mageck_mle(
   labels: Union[FlyteFile, List[str]], # a CSV or just a list of labels
   ...):

in the latter case the type system enforces that there is exactly one place for us to get the labels

[EngHabu]

To provide a bit of additional context about the current state of the system:

• While it’s true that, at runtime in propeller, we do not validate whether the value is really a Void or whether it matches the target type, in practice there is only one scenario where that is possible…
  • If your task’s signature returns type Foo and the implementation returns None, the SDK will complain at runtime
  • If you have your own SDK, then you can generate Void values
  • If you run a map task with the option success_ratio < 1, that’s the only real possibility of generating Void values since we try to fill in Voids in place of the missing/failed values…
  • So outside of a couple of known users, I think we can safely assume that when a task returns/accepts Foo, people actually do return/accept instances of type Foo…

Implementing typing.Optional will be fairly straight forward.

• We will add a bool is_optional to the LiteralType
• The compiler will validate that when it matches signatures (only Optional Inputs can accept Optional Outputs)
• Flytekit will run that assertion again when transforming inputs to python types… and when generating literals for outputs from python types

[EngHabu]

@wild-endeavor @cosmicBboy @eapolinario @kumare3 Please chime in...

[EngHabu]

@maximsmol, @wild-endeavor, @cosmicBboy, @eapolinario, @kumare3 and myself met today to discuss the proposal moving forward. Here are the arguments for and with Sum Types I've heard today (for documentation purposes):

Pros:

1. It's very ergonomic in Python to receive (and even return) Union types to keep APIs tolerant to multiple different types that share some common behavior (e.g. in your example, a file path or list of strings represent the same thing in different mediums)
2. It's a super-type of Optional and therefore, can be used to implement Optional (as you indicated earlier).
3. This is one way of implementing interface evolution (allowing newer versions of tasks to add optional params and maintain backward compatibility with older workflows)
4. This is one way of implementing offloaded literals consistently across the platform.

Cons:

1. Not all languages (e.g. JAVA) don't have native support for SumTypes.
   I believe we can find creative ways to expose the same semantics to consume tasks that take SumTypes to flytekit-java.
2. Additional complexity in the typing system that might not be solving the real underlying concern (offloaded types.. etc.)

Looking at both arguments, we believe there is a stronger argument for adding this support across the system. Thank you for writing up the RFC. Please open a PR so we can start leaving comments on the details listed there..