Airflow Deployment on Openshift Failed for CeleryExecutor and KubernetesExecutor

[yogeshwariingale]

I have been trying to deploy airflow on OpenShift 4. For 'LocalExecutor' It works fine. But if i try to deploy for celery or Kubernetes executor it does not start Pod gives CrashloopBackoff error for every pod other than postgres pod. Below is the command that i am running and attached is values.yaml file used
helm repository:
airflow-stable https://airflow-helm.github.io/charts

command Run to install Helm on OpenShift:
helm install "apache-helm" airflow-stable/airflow --namespace "apache-helm" --version "8.X.X" --values ./values.yml

Values.yaml:

uid: 1001020000
gid: 1001020000
airflowHome: /opt/airflow
defaultAirflowRepository: apache/airflow
defaultAirflowTag: "2.1.3"
airflowVersion: "2.1.3"

executor: "CeleryExecutor"

dags:
persistence:
enabled: false
size: 1Gi
accessMode: ReadWriteMany
storageClassName:
existingClaim:

gitSync:
enabled: true
repo: https://github.com/gitrepoPath/airflow_dag.git
branch: main
subPath: ""
rev: HEAD
depth: 1
maxFailures: 3
credentialsSecret: git-credentials
wait: 60
images:
airflow:
repository: ~
tag: ~
pullPolicy: IfNotPresent
pod_template:
repository: ~
tag: ~
pullPolicy: IfNotPresent
flower:
repository: ~
tag: ~
pullPolicy: IfNotPresent
statsd:
repository: apache/airflow
tag: airflow-statsd-exporter-2021.04.28-v0.17.0
pullPolicy: IfNotPresent
redis:
repository: redis
tag: 6-buster
pullPolicy: IfNotPresent
pgbouncer:
repository: apache/airflow
tag: airflow-pgbouncer-2021.04.28-1.14.0
pullPolicy: IfNotPresent
pgbouncerExporter:
repository: apache/airflow
tag: airflow-pgbouncer-exporter-2021.04.28-0.5.0
pullPolicy: IfNotPresent
gitSync:
repository: k8s.gcr.io/git-sync/git-sync
tag: v3.6.0
pullPolicy: IfNotPresent

allowPodLaunching: true

env: []
secret: []

workers:
replicas: 3
serviceAccount:
create: true
name: anyuid
args:
- "bash"
- "-c"
- |-
exec
airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery worker" "worker" }}

persistence:
enabled: true
size: 100Gi
storageClassName:

resources: {}

terminationGracePeriodSeconds: 600
safeToEvict: true
extraVolumes: []
extraVolumeMounts: []

scheduler:
replicas: 1
serviceAccount:
create: false
name: anyuid

args: ["bash", "-c", "exec airflow scheduler"]
podDisruptionBudget:
enabled: false
config:
maxUnavailable: 1
safeToEvict: true
extraVolumes: []
extraVolumeMounts: []
uid: 1001020000

postgresql:
enabled: true
auth:
postgresPassword: postgres
postgresqlUsername: postgres
uid: 1001020000
data:
metadataConnection:
user: postgres
pass: postgres
protocol: postgresql
host: ~
port: 5432
db: postgres
sslmode: disable

createUserJob:
serviceAccount:
create: false
name: anyuid

migrateDatabaseJob:
serviceAccount:
create: false
name: anyuid

webserver:
serviceAccount:
create: false
name: anyuid
allowPodLogReading: true
replicas: 1
args: ["bash", "-c", "exec airflow webserver"]

defaultUser:
enabled: true
role: Admin
username: admin
email: [email protected]
firstName: admin
lastName: user
password: admin

service:
type: ClusterIP
ports:
- name: airflow-ui
port: "{{ .Values.ports.airflowUI }}"

redis:
enabled: true
serviceAccount:
create: false
name: anyuid
persistence:
enabled: true
size: 1Gi
password: ~
ports:
flowerUI: 5555
airflowUI: 8080
workerLogs: 8793
redisDB: 6379
statsdIngest: 9125
statsdScrape: 9102
pgbouncer: 6543
pgbouncerScrape: 9127

labels: {}

multiNamespaceMode: false

logs:
persistence:
enabled: false
size: 100Gi
storageClassName:
existingClaim:

triggerer:
replicas: 1
args: ["bash", "-c", "exec airflow triggerer"]
serviceAccount:
create: false
name: anyuid

flower:
enabled: true
serviceAccount:
create: false
name: anyuid
username: ~
password: ~
args:
- "bash"
- "-c"
- |-
exec
airflow {{ semverCompare ">=2.0.0" .Values.airflowVersion | ternary "celery flower" "flower" }}
service:
type: ClusterIP
ports:
- name: flower-ui
port: "{{ .Values.ports.flowerUI }}"

statsd:
enabled: true
serviceAccount:
create: false
name: anyuid
uid: 1001020000

pgbouncer:
enabled: false
serviceAccount:
create: false
name: anyuid

metadataPoolSize: 10
resultBackendPoolSize: 5
maxClientConn: 100
uid: 1001020000

cleanup:
enabled: true
schedule: "*/15 * * * *"
nodeSelector: {}
affinity: {}
tolerations: []
serviceAccount:
create: false
name: anyuid

elasticsearch:
enabled: false

connection: {}

[thesuperzapper]

@yogeshwariingale First, I think your values file looks wrong, take a look at "step 3" of the quickstart guide for info on creating your values file.

If you are still having problems after updating your values, I have two recommendations:

1. Take a quick look at the pod events (and logs) for the pods which are in CrashLoopBackoff
2. Openshift is known to be deployed with lots of strange security requirements, especially around which IDs are allowed for securityContext settings like fsGroup and runAsGroup, so I bet you will have to change from the defaults in the chart

If you still have a problem, please share your error logs/events and we will take a look.

[yogeshwariingale]

Hi, Thank you for your support. I am able to deploy airflow on OpenShift