Have the ability to configure CPU and Memory related parameters on K8s

[EladDolev]

Some K8s clusters are running with CFS quota disabled (--cpu-cfs-quota=false) so setting pods QOS class to Guaranteed by setting requests == limits comes without paying any price, so having the flexibly of defining how much memory to use may lead to better utilization with more reliability (reducing chance of OOM)

Some K8s clusters are running with static policy for CPU management, so having the ability to set Redpanda affinity to only several vCPU and/or NUMA domains should increase performance and allow for better isolation.
Currently with static policy Redpanda would crash with the following message when CPUs to use are not being set

redpanda::main - application.cc:158 - Failure during startup: cluster::configuration_invariants_changed (Configuration invariants changed. Expected: { version: 0, node_id: 2, core_count: 4 }, current: { version: 0, node_id: 2, core_count: 1 })

Also K8s 1.21 introduce an alpha feature to use a static policy for memory so more chances for optimizations there

[dimitriscruz]

Thanks, @EladDolev!

For reference, we got a couple of improvements merged that are related to this discussion
#1416 - sets --smp and --memory based on the resource request
#1556 - sets the init container resources (to what is provided in the CR) as done for the Redpanda container