Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional change added for using Croniter Library for Dag Dependency Handling #1357

Open
wants to merge 41 commits into
base: main
Choose a base branch
from
Open
Changes from 38 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
d6126d6
Added the Composer Dependency Management Example
tshashankrock May 27, 2024
172c130
Update README.md
tshashankrock May 27, 2024
8d5fcd2
Update Cloud Composer Example README.md
tshashankrock May 27, 2024
3caba14
Added new line in the .py files
tshashankrock May 27, 2024
5fa06f1
Updated Composer Dependency Management README.md
tshashankrock May 27, 2024
d31f6b2
Restructure the code base
tshashankrock May 28, 2024
dd70dac
Updated README.md
tshashankrock May 28, 2024
ead937d
Update README.md
tshashankrock May 28, 2024
f17a69d
Update README.md
tshashankrock May 28, 2024
ce3e875
Update product_catalog_refresh.py
tshashankrock Jun 4, 2024
30b54da
Update company_cal_refresh.py
tshashankrock Jun 4, 2024
9e7bd3a
Update product_inventory_refresh.py
tshashankrock Jun 4, 2024
00c33b8
Update weekly_summary_report.py
tshashankrock Jun 4, 2024
377f8cb
Update company_cal_refresh.py removed template_searchpath
tshashankrock Jun 4, 2024
596cf83
Update product_catalog_refresh.py removed template_searchpath
tshashankrock Jun 4, 2024
fdefd83
Update product_inventory_refresh.py removed template_searchpath
tshashankrock Jun 4, 2024
78e1a81
Update weekly_summary_report.py removed template_searchpath
tshashankrock Jun 4, 2024
0d73055
Update README.md
tshashankrock Jun 4, 2024
841f6b3
Update README.md
tshashankrock Jun 4, 2024
486bb87
Update README.md
tshashankrock Jun 5, 2024
61ec994
Merge branch 'GoogleCloudPlatform:main' into main
tshashankrock Jun 5, 2024
2a02020
Update README.md
sguruprasad-google Jun 5, 2024
3ca0562
Update README.md
sguruprasad-google Jun 5, 2024
0f239d7
Update README.md
sguruprasad-google Jun 5, 2024
89b19c1
Update README.md
sguruprasad-google Jun 5, 2024
6cef5c4
Update README.md
sguruprasad-google Jun 5, 2024
46ce6e0
Update README.md
sguruprasad-google Jun 5, 2024
b819578
Update README.md
sguruprasad-google Jun 5, 2024
46ad68f
Update README.md
sguruprasad-google Jun 5, 2024
852523a
Update README.md
tshashankrock Jun 5, 2024
14a73a2
Updated README.md file
tshashankrock Jun 5, 2024
df41a8b
Revoked the earlier README.md changes
tshashankrock Jun 5, 2024
61912b6
cosmetic changes in the readme file
naitik1995 Jun 5, 2024
6f84670
Merge branch 'main' into main
agold-rh Jun 5, 2024
e9dbc08
Formatted the files and removed unused import
tshashankrock Jun 5, 2024
6f34b1e
Merge branch 'GoogleCloudPlatform:main' into main
tshashankrock Sep 24, 2024
24b914e
Added an update for using croniter library to get the dag instance ex…
naitik1995 Sep 24, 2024
2ae0b22
Merge branch 'main' into main
agold-rh Oct 7, 2024
ddd2f3d
commented use of unrequired libraries
naitikgadaorica Oct 7, 2024
fa0c5ca
Merge branch 'main' of https://github.com/tshashankrock/professional-…
naitik1995 Oct 7, 2024
9219415
Merge branch 'main' into main
naitik1995 Oct 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
from datetime import datetime, timedelta
from airflow.operators.dummy_operator import DummyOperator
from airflow.sensors.external_task_sensor import ExternalTaskSensor
from croniter import croniter
import pytz

default_dag_args = {
"depends_on_past": False,
Expand Down Expand Up @@ -53,6 +55,16 @@ def execution_delta_dependency(logical_date, **kwargs):
elif schedule_frequency == "daily":
parent_dag_poke = (dt).replace(hour=0, minute=0, second=0, microsecond=0)
print(parent_dag_poke)

#OPTIONAL : In case your dag frequency does not fall in any of the above options or if you want to test with varying dynamic frequencies
# such as 5th minute every hour and so on, consider uncommenting the below code and comment / delete the above code.
# the below code uses the croniter library to get the previous execution instance of the dag based on the date and the scheduled frequency

# iter = croniter(schedule_frequency, dt)
# prev_instance = iter.get_prev(datetime)
# parent_dag_poke = pytz.utc.localize(prev_instance)
# return prev_instance

return parent_dag_poke

with airflow.DAG('product_catalog_refresh',
Expand Down