Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add driver and executor pod template params to Spark config #1863

Closed
wants to merge 2 commits into from

Conversation

andrewwdye
Copy link
Contributor

@andrewwdye andrewwdye commented Sep 29, 2023

TL;DR

Add driver and executor pod template params to Spark config

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

This change adds pairs of pod template and pod template name params for the driver and executor pods. This mirrors the interface we provide for container tasks generally.

TODO: once we get consensus on the api I will populate the SparkJob proto and add tests

Tracking Issue

flyteorg/flyte#4105

@andrewwdye andrewwdye requested a review from hamersaw September 29, 2023 23:15
@codecov
Copy link

codecov bot commented Sep 29, 2023

Codecov Report

Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (660f5aa) 54.33% compared to head (56cdde0) 54.85%.
Report is 12 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1863      +/-   ##
==========================================
+ Coverage   54.33%   54.85%   +0.52%     
==========================================
  Files         288      294       +6     
  Lines       21934    22209     +275     
  Branches     3344     2171    -1173     
==========================================
+ Hits        11917    12183     +266     
- Misses       9855     9880      +25     
+ Partials      162      146      -16     
Files Coverage Δ
...ugins/flytekit-spark/flytekitplugins/spark/task.py 86.74% <100.00%> (+0.84%) ⬆️
...ins/flytekit-spark/flytekitplugins/spark/models.py 68.22% <53.84%> (-4.62%) ⬇️

... and 36 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@wild-endeavor wild-endeavor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this make any sense?

Signed-off-by: Andrew Dye <[email protected]>
@andrewwdye
Copy link
Contributor Author

Closing until we get consensus on the proto interface for multi-role pod template overrides. See comment here.

@andrewwdye andrewwdye closed this Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants