Define daemonset for the scheduler/web when under heavy load?

[el-aasi]

Is there a way to basically scale scheduler/web on each node as the node number increases?

[thesuperzapper]

@el-aasi I don't understand why you would want to run an instance of the scheduler/webserver for each node in your cluster. In all but the most extreme cases, you can just figure out how many replicas are required at your peak load, and just run that 24/7.

If you REALLY need to dynamically scale the Web/Scheduler, you need to look into setting up your own HorizontalPodAutoscaler for those Deployments (which would need to be managed outside the chart).

---

TIP: ensure you are setting resource requests for each Pod (otherwise Kubernetes will assume they are literally using 0 resources when deciding which node to schedule them on)

---

PS: the thing you probably do want to scale dynamically is the number of celery workers, and I am actively working on a task-aware autoscaler which I hope to release in the next version of the chart (#339)

[el-aasi]

@thesuperzapper yes, you are completely right, there is no reason to do that.

I realized that quickly after writting that text, should have deleted it.

I went with a solution where the (kubernetes) executors run on separate nodes than the scheduler/UI - which works well for bursts of load. But will most probably look into either having more replicas of the UI or hpa.

Thank you.