Skip to content

This repository contains application code for the Wizeline Data Engineering Bootcamp (DEB) 2023. It is one of two repositories for the DEB. The other houses the infrastructure code.

Notifications You must be signed in to change notification settings

uche-madu/deb-application

Repository files navigation

🗜️⚙️ DEB Application Repository 🔩🧰

Airflow Image Build Push Pyspark Scripts to GCS

This is one of two repositories with code for the entire DEB Project. While this repository focuses on the application code such as Airflow DAGs, the DEB Infrastructure repository focuses on provisioning cloud resources. This separation of concerns via separate repositories aims to follow GitOps Principles.

CI/CD Workflow

  • The build-push.yaml Github Actions workflow builds a custom airflow image and pushes it to Google Artifact Registry. It then checks out the DEB Infrastructure repository to update the airflow-helm/values-dev.yaml file with the new image and tag. For this to work, create a repository-scoped Personal Access Token for your Github account and store it as a secret in this repository. Visit Github Docs for details about creating tokens.
  • Also create Github secrets for your Github email and Username

About

This repository contains application code for the Wizeline Data Engineering Bootcamp (DEB) 2023. It is one of two repositories for the DEB. The other houses the infrastructure code.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published