You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ADF has proven to be a reliable service for orchestrating pipelines, however it does have its limitations. Fortunately, there's no need to worry. The advent of managed airflow brings a promising solution, empowering us to overcome these limitations through the power of coding.
In this challenge, you will get hands on with Airflow and you will know how to orchestrate ADF.
Description
In this challenge, you will properly set up a managed airflow instance and connect it to azure data factory.
Please configure the managed airflow instance with the following specifications:
Use basic auth.
Create airflow folder in your ADLS containing 2 sub-folders : dags and plugins.
You will have to create 2 pipelines like so: one pipeline will sum 2 random numbers
The second pipeline will calculate the average of these 2 numbers
Write a DAG that will generate these two pipelines randomly – use random library in python
Success Criteria
To complete this challenge successfully, you should be able to:
Verify that the ADF has a configured airflow instance.
Verify that in the managed airflow UI you got the connection with ADF configured properly.
Demonstrate that airflow can trigger pipelines using the power of coding and see triggered pipelines in ADF monitor.
sallydabbahmsft
changed the title
Challenge 061- Orchestrating ADF pipelines with managed airflow
Orchestrating ADF pipelines with managed airflow
Aug 5, 2023
Topic
Orchestrating ADF with Managed Apache Airflow
Abstract/Learning Objectives
Pre-requisites (Optional)
Basic knowledge of ADF pipelines
Basic Knowledge in Apache airflow
Workspace in ADF and Storage account in ADLS.
Introduction
ADF has proven to be a reliable service for orchestrating pipelines, however it does have its limitations. Fortunately, there's no need to worry. The advent of managed airflow brings a promising solution, empowering us to overcome these limitations through the power of coding.
In this challenge, you will get hands on with Airflow and you will know how to orchestrate ADF.
Description
In this challenge, you will properly set up a managed airflow instance and connect it to azure data factory.
Use basic auth.
Create airflow folder in your ADLS containing 2 sub-folders : dags and plugins.
You will have to create 2 pipelines like so: one pipeline will sum 2 random numbers
The second pipeline will calculate the average of these 2 numbers
Write a DAG that will generate these two pipelines randomly – use random library in python
Success Criteria
To complete this challenge successfully, you should be able to:
Verify that the ADF has a configured airflow instance.
Verify that in the managed airflow UI you got the connection with ADF configured properly.
Demonstrate that airflow can trigger pipelines using the power of coding and see triggered pipelines in ADF monitor.
Learning Resources
How does Managed Airflow work? - Azure Data Factory | Microsoft Learn
airflow.operators.python — Airflow Documentation (apache.org)
airflow.providers.microsoft.azure — apache-airflow-providers-microsoft-azure Documentation
Advanced Challenges (Optional)
Too comfortable? Eager to do more? Try these additional challenges!
Create an alert if DAG fails (send a mail to a specific user)
Write a DAG which sends parameters to an existing pipeline in ADF.
Delivery Date
20/09/2023
Authors
No response
Other
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: