Document the way the Kubernetes scheduler mode impacts Karpenter

[stevehipwell]

Description

What problem are you trying to solve?
I have the following hypothesis.

With the default Kubernetes scheduler configuration of LeastAllocated, Karpenter is statistically more likely to disrupt pods and/or require consolidation than if the scheduler was running in MostAllocated mode.

I'd like to see this tested and documented. I suspect that in some cases, especially where eviction on some pods is disabled, that this would provide a significant price reduction.

How important is this feature to you?
I think having this officially documented would help users understand how to get the most out of Karpenter and would add weight to requests to cloud providers to support customising the scheduler.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[stevehipwell]

CC @jonathan-innis (this is what I spoke to you at KubeCon EU about)

[jonathan-innis]

I'd like to see this tested and documented. I suspect that in some cases, especially where eviction on some pods is disabled, that this would provide a significant price reduction

Agreed. This sounds like a good feature to explore. Obviously, for us to give a recommendation for those that have direct control of their SchedulerConfiguration and the control plane, but also platforms like EKS should explore this alternative option to see if they get increased performance with Karpenter from this other setting.

I suspect that you are right about the performance, at least that we would be able to act more aggressively with consolidation since we'd have "less cluster churn" disrupting our current consolidation decision-making.

[jonathan-innis]

/triage accepted