Skip to content

Workspace for Airflow training, inlcuding docker and docker compose

Notifications You must be signed in to change notification settings

godatadriven/airflow_workspace

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Codespaces Airflow training environment

This repository contains the code for the Airflow training environment.

Getting started

  1. Check that you have enough memory allocated to your codespaces instance. In the terminal (ctrl + ` ) check you have more than 4GB of allocated memory:

    docker run --rm "debian:bullseye-slim" bash -c 'numfmt --to iec $(echo $(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE))))'
    

    An no, 3.9 is not enough. If you see 3.9 means that you did not selected the right machine in step 2.

  2. Run database migrations (ONLY RUN ONCE)

    You need to run database migrations and create the first user account. It is all defined in the docker compose file so just run:

    docker compose up airflow-init
    
  3. Now you can start all services:

    docker compose up
    

    This will make Airflow available at: http://localhost:8080

  4. Login to Airflow:

    image

Running Airflow in a Python Environment

If for some reason its not possible to you to run Airflow using docker, you can also do it using python.

  1. Install airflow

    pip install apache-airflow
    

    or

    poetry add apache-airflow
    

    Make sure you have a virtual environment activated in which you can isolate your code/dependencies

  2. initialize the database

    airflow db init
    
  3. Create a new user

    airflow users create --username airflow --password airflow --firstname anon --lastname nymus --role Admin --email [email protected] 
    
  4. Copy your dags to the dags/ folder

    cp dags/mydag.py ~/airflow/dags/
    
  5. In a termnial initialize the webserver

    airflow webserver -p 8080
    
  6. In a second terminal initialize the scheduler

    airflow scheduler
    

About

Workspace for Airflow training, inlcuding docker and docker compose

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published