Feature request: downscale stabilization

[xificurk]

Hello, I'm trying to minimize frequent fluctuations in the replicas count in our setup. The rate of incoming messages is fluctuating quite a lot. The usual scenario is: sudden peak in incoming messages, this leads to a peak in queue length, autoscaler kicks in and starts scaling up every interval, the messages are quickly processed, the scaling up repeats upto eval-intervals, after that autoscaler starts to scale down... until another peak comes in and the cycle repeats.

The size of fluctuations can be somewhat mitigated by increase-limit and decrease-limit, but it does not help with the rate at which the replica count is constantly changing.

Increasing eval-intervals does not help either, because then a very short-lived but big peak in queue length leads to scaling up long after the messages have been processed.

I think a feature similar to --horizontal-pod-autoscaler-downscale-stabilization from standard HPA could help.

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

Finally, just before HPA scales the target, the scale recommendation is recorded. The controller considers all recommendations within a configurable window choosing the highest recommendation from within that window. This value can be configured using the --horizontal-pod-autoscaler-downscale-stabilization flag, which defaults to 5 minutes. This means that scaledowns will occur gradually, smoothing out the impact of rapidly fluctuating metric values.

What do you think? Does it make sense to add something like this to kube-amqp-autoscale?