Sane Overridable Defaults #3731
Replies: 6 comments 4 replies
-
@katrogan thank you for this proposal. This is an amazing RFC and will really simply the user lives. +1 |
Beta Was this translation helpful? Give feedback.
-
Love it! A quick question regarding update. What do you think we make it closer to how kubectl works, either client-side or server-side apply, instead of having our own way relying on |
Beta Was this translation helpful? Give feedback.
-
I think it is a great idea to simplify this! We do have project/domain specific pod templates but don't override further defaults. You said that doing so is error-prone, difficult to holistically reason about, and requires understanding about the data model. I couldn't have put it into these words but I would say that when I did look at this part of the documentation in the past, "it didn't look inviting to try this unless one really has to". One other important reason, apart from complexity/UX, why I'd be a bit hesitant to use this feature in case one of our platform users suggested to have defaults overridden for their project, is that we try to stay away from any platform config (Flyte and in general) that we cannot put into declarative infrastructure as code that we can version control. As I'm reluctant to use I'm aware that in some situations where the documentation tells the user to use |
Beta Was this translation helpful? Give feedback.
-
@katrogan so I think considering the feedback from the community, feel free to kickoff the process for a new RFC! |
Beta Was this translation helpful? Give feedback.
-
@davidmirror-ops thanks, opened #3749 |
Beta Was this translation helpful? Give feedback.
-
Closing this as the real RFC #3749 has already been accepted. |
Beta Was this translation helpful? Give feedback.
-
Sane overridable defaults
Background
Flyte comes with a rich set of overridable defaults (matchable resources) from everything to default task resource requests, project resource quota, kubernetes labels and annotations for executions and default service accounts.
These can be configured to have default levels across a Flyte installation, on a per-domain basis, on a per-project basis and event at a per-workflow basis.
Managing these requires setting the defaults at a level of specifity using a variety of message types. Furthermore it's difficult to holistically reason about what overrides exist at say, the project level. Updating these overrides is error-prone because there is no existing merge logic and requires understanding the data model to even know what specific matchable resource to specifically update.
Proposal
Using
kubectl apply
as an inspiration, we should be able to make it simple to view, edit and update defaults for domains, projects and workflows. It should be difficult to unintentionally overwrite changes and easy to understand what is currently applied.Overview
Proposed User Experience
It should be simple to view and edit overridable defaults
For
Proposed document structure
A single call to get all overrides should return the collated document for all overrides set at a level (e.g. per project)
Note a few details
Implementation Proposal
Define a new set of APIs for returning overridable defaults
FlyteAdmin changes
Update overrides
In implementing the endpoint handlers above, FlyteAdmin will need to create a new table to store document version for each scope of overrides (that is at the per project or per domain level).
So that the call to update overrides can only be applied when its version matches what is stored in the database. This prevents stale data from being committed and possibly overwriting a newer set of overrides that would need to be merged.
A call to update overrides should consist of a transaction to
inherited
in the document.i) To support override deletion, any overrides in the database which do not exist in the document should be removed.
Get Overrides
Getting overrides at a super specific level, for example at the workflow level, should use the same fallback logic at execution time to determine applicable overrides.
That is, the workflow-level document should include all applicable workflow overrides, any project overrides and any domain overrides in one document with inherited overrides marked as such.
Migration Strategy
Eventually we should store the entire document as part of the overrides table and move away from the confusing resources table. This allows us to do a finite number of gets at execution time to fetch the entire list of realized overrides for the global, domain, project and workflow level for all overridable attributes
To get there, we can create a new flyteadmin binary that runs a migration from the resources table and flattens it as an entry in the overrides table.
Beta Was this translation helpful? Give feedback.
All reactions