Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for running dbt tasks in AWS EKS #944

Merged
merged 24 commits into from
May 22, 2024
Merged

Support for running dbt tasks in AWS EKS #944

merged 24 commits into from
May 22, 2024

Conversation

VolkerSchiewe
Copy link
Contributor

@VolkerSchiewe VolkerSchiewe commented May 7, 2024 •
edited

Description

We are using MWAA in combination with EKS so that all our dags in airflow are running in our EKS. We would like to use the same setup with cosmos.

What changes?

  • New AwsEksOperator classes (inheriting from KubernetesOperators) - Based on the original EksOperator
  • Tests
  • Adjusted documentation

Related Issue(s)

Breaking Change?

No - only an additional feature

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@VolkerSchiewe VolkerSchiewe requested a review from a team as a code owner May 7, 2024 08:12
@dosubot dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label May 7, 2024
@netlify Netlify
Copy link

netlify bot commented May 7, 2024 •
edited

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name Link
🔨 Latest commit 42fa4c4
🔍 Latest deploy log https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/664de8fe8711f20008ea8fa5
😎 Deploy Preview https://deploy-preview-944--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@dosubot dosubot bot added area:docs Relating to documentation, changes, fixes, improvement area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:testing Related to testing, like unit tests, integration tests, etc labels May 7, 2024
@tatiana tatiana added this to the 1.5.0 milestone May 7, 2024
tatiana
tatiana reviewed May 7, 2024
View reviewed changes
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, @VolkerSchiewe !
Please, could you address the tests currently failing?

@VolkerSchiewe
Copy link
Contributor Author

Thanks for the contribution, @VolkerSchiewe ! Please, could you address the tests currently failing?

Hi @tatiana thanks for getting back to me! I was already working on it, but I ran into this issue: apache/airflow#39103

Seems to be affecting the latest version of the amazon airflow provider. I already tried with pinning an older version of xmlsec1 with no success :/

@tatiana tatiana added the cosmos label May 10, 2024
@codecov Codecov
Copy link

codecov bot commented May 21, 2024 •
edited

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.72%. Comparing base (007325a) to head (42fa4c4).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #944      +/-   ##
==========================================
+ Coverage   95.67%   95.72%   +0.05%     
==========================================
  Files          59       60       +1     
  Lines        2890     2926      +36     
==========================================
+ Hits         2765     2801      +36     
  Misses        125      125

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tatiana
tatiana reviewed May 22, 2024
View reviewed changes
cosmos/constants.py Outdated Show resolved Hide resolved
pankajkoti
pankajkoti reviewed May 22, 2024
View reviewed changes
Copy link
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great contribution @VolkerSchiewe . I think we're close and soon can merge this.

Would it be possible to also paste a snapshot of a successful DAG run using AWS EKS operator?

cosmos/operators/eks.py Outdated Show resolved Hide resolved
pyproject.toml Outdated Show resolved Hide resolved
@VolkerSchiewe
Copy link
Contributor Author

Would it be possible to also paste a snapshot of a successful DAG run using AWS EKS operator?

Hope this is what you mean @pankajkoti (the operator naming is still before the renaming though)
image

pankajkoti
pankajkoti approved these changes May 22, 2024
View reviewed changes
Copy link
Contributor

@pankajkoti pankajkoti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Appreciate your contribution @VolkerSchiewe 👏🏽

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label May 22, 2024
tatiana
tatiana approved these changes May 22, 2024
View reviewed changes
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks a lot for iterating and improving it so quickly, @VolkerSchiewe !

@tatiana tatiana changed the title Support for EKS operator Support for AWS EKS operator May 22, 2024
@tatiana tatiana changed the title Support for AWS EKS operator Support for running dbt tasks in AWS EKS May 22, 2024
pankajastro
pankajastro approved these changes May 22, 2024
View reviewed changes
Copy link
Contributor

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great 👍

docs/getting_started/execution-modes.rst Outdated Show resolved Hide resolved
@pankajkoti pankajkoti merged commit cb2a27a into astronomer:main May 22, 2024
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pankajkoti pankajkoti pankajkoti approved these changes

@tatiana tatiana tatiana approved these changes

@pankajastro pankajastro pankajastro approved these changes

Assignees

@pankajkoti pankajkoti

@VolkerSchiewe VolkerSchiewe

Labels
area:docs Relating to documentation, changes, fixes, improvement area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:testing Related to testing, like unit tests, integration tests, etc cosmos epic-assigned lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files. triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Milestone
Cosmos 1.5.0
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@VolkerSchiewe @tatiana @pankajkoti @pankajastro