-
Notifications
You must be signed in to change notification settings - Fork 691
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AirflowDagRunner fails with RuntimeParameter for scheduled runs #4389
Comments
I didn't realize about this. I also expected this to return an empty dictionary. I have to admit we have not been using this a lot. That said, I'm sure a PR can be made and tested :D Would be happy to review and tag folks from Google to help review! |
Thanks for the info, I would love to contribute a patch but unfortunately it's very difficult on my end to get approval given the CLA requirement. |
Are you still looking for a resolution? We are planning on prioritising the issues based on the community interests. Please let us know if this issue still persists with the latest TFX 1.13 release so that we can work on fixing it. Thank you for your contributions. |
This is not a priority for us. |
This is not a contribution.
TFX Version: 1.2
Airflow version 1.14
Describe the current behavior
When using a RuntimeParameter with the AirflowDagRunner the dag will fail for scheduled runs where the dag_run.conf is missing.
Specifically this error is thrown:
As far as I can tell this occurs due to this line which assumes that
dag_run.conf
is defined, whereas it appears to beNone
in some situations (see apache/airflow#5388). The template probably needs to be adapted todag_run.conf.get("{prop_name}", {default}) if dag_run.conf else {default}
Note that this has been running successfully where we manually trigger the dags through the airflow UI.
@casassg how do you normally get around this?
The text was updated successfully, but these errors were encountered: