Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orchestrating ADF pipelines with managed airflow #697

Open
1 task done
sallydabbahmsft opened this issue Aug 5, 2023 · 0 comments
Open
1 task done

Orchestrating ADF pipelines with managed airflow #697

sallydabbahmsft opened this issue Aug 5, 2023 · 0 comments
Labels
proposal Proposal for a new hack

Comments

@sallydabbahmsft
Copy link

Topic

Orchestrating ADF with Managed Apache Airflow

Abstract/Learning Objectives

Pre-requisites (Optional)

Basic knowledge of ADF pipelines

Basic Knowledge in Apache airflow

Workspace in ADF and Storage account in ADLS.

Introduction

ADF has proven to be a reliable service for orchestrating pipelines, however it does have its limitations. Fortunately, there's no need to worry. The advent of managed airflow brings a promising solution, empowering us to overcome these limitations through the power of coding.

In this challenge, you will get hands on with Airflow and you will know how to orchestrate ADF.

Description

In this challenge, you will properly set up a managed airflow instance and connect it to azure data factory.

  1. Please configure the managed airflow instance with the following specifications:

Use basic auth.

Create airflow folder in your ADLS containing 2 sub-folders : dags and plugins.

  1. You will have to create 2 pipelines like so: one pipeline will sum 2 random numbers

    The second pipeline will calculate the average of these 2 numbers

  2. Write a DAG that will generate these two pipelines randomly – use random library in python

Success Criteria

To complete this challenge successfully, you should be able to:

Verify that the ADF has a configured airflow instance.

Verify that in the managed airflow UI you got the connection with ADF configured properly.

Demonstrate that airflow can trigger pipelines using the power of coding and see triggered pipelines in ADF monitor.

Learning Resources

How does Managed Airflow work? - Azure Data Factory | Microsoft Learn

airflow.operators.python — Airflow Documentation (apache.org)

airflow.providers.microsoft.azure — apache-airflow-providers-microsoft-azure Documentation

Advanced Challenges (Optional)

Too comfortable? Eager to do more? Try these additional challenges!

Create an alert if DAG fails (send a mail to a specific user)

Write a DAG which sends parameters to an existing pipeline in ADF.

Delivery Date

20/09/2023

Authors

No response

Other

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@sallydabbahmsft sallydabbahmsft added the proposal Proposal for a new hack label Aug 5, 2023
@sallydabbahmsft sallydabbahmsft changed the title Challenge 061- Orchestrating ADF pipelines with managed airflow Orchestrating ADF pipelines with managed airflow Aug 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposal Proposal for a new hack
Projects
None yet
Development

No branches or pull requests

1 participant