Dagster 🪢 Azure Key Vault

This repo is an example of using Azure Key Vault as a Dagster resource.

I am open sourcing this both to get feedback and learn ... and to save you a lot of frustration (this took waaay too long for me to figure out)

If you have any questions, feedback, bugs, or improvements, create an issue - or contact me on the Dagster Slack, my name is Martin Carlsson, or directly on [email protected]

And don't forget to hit the ⭐️ button.

Setup repo on your local machine

Download repo

Start by forking this repo, hit the star icon ⭐️, and download the repo locally.

Install development environment

Use Remote development in Containers or install requirements directly pip install -r requirements.txt

Run pytest

Set environment variables:

export AZURE_KEY_VAULT_URI=
export AZURE_KEY_VAULT_SERVICE_PRINCIPAL_CLIENT_SECRET=
export AZURE_KEY_VAULT_SERVICE_PRINCIPAL_CLIENT_ID=
export AZURE_TENANT_ID=

Execute pytest:

Just run pytest in the terminal.

Run Dagit

cd dagster_azure_key_vault
dagit

Config:

ops:
  get_secret_from_azure_key_vault:
    config:
      azure_key_vault_secret_name: "ForTestingKeyVault"
resources:
  key_management:
    config:
      azure_key_vault_service_principal_client_id:
      azure_key_vault_service_principal_client_secret:
      azure_key_vault_uri:
      azure_tenant_id:

Setup and connect to Azure Key Vault

Create an Azure Key Vault

Create a key vault using the Azure portal

Create a secret

Open Key Vaults in Azure portal:

Select the Key Vault you just created:

Copy Vault URI:

You will need it when connecting to Azure Key Vault from Dagster.

Click on Secrets:

Click on +Generate/Import:

For testing, create the following secret:

• Name: ForTestingKeyVault
• Value: TestValidated

Create Service Principal

We will connect to Azure Key Vault via a Service Principal.

Register an application with Azure AD and create a service principal

Give the Service Principal access to Azure Key Vault:

Give Service Principal access to Azure Key Vault

Create a new application secret for Service Principal

Create a new application secret