This is an advanced example for managing CI/CD across multiple Prefect workspaces.
The GitHub Actions workflows in this repo:
- create mirrored deployments across staging and prod workspaces
- which workspace flows are deployed to is determined by the branch that changes happen on:
- opening a pull request on any branch, as well as commits to an open pull request, will update deployments in the staging workspace
- a merge to
mainwill update deployments in the prod workspace
- which workspace flows are deployed to is determined by the branch that changes happen on:
- build and push docker images that do not contain flow code, and are tagged like
:<commit-hash>-<environment>to tie the code version and branch/env to image versions - cache both docker image layers and pip dependencies for ultra fast rebuilds
- push code to S3 (via the
pushstep of eachprefect.yaml), where:- each environment gets its own bucket
- each project gets its own folder in that bucket
This example demonstrates a monorepo approach with multiple projects, each project having:
- a
prefect.yaml - a
Dockerfile - a
requirements.txt - its own flow(s)
- its own independently triggered GitHub Actions workflow
The configuration relies heavily on templating to make copy/pasting files when starting new projects or adding deployments easier.