-
Notifications
You must be signed in to change notification settings - Fork 701
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubeadm: keep dns resourceRequirements on upgrade #3033
Comments
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:
Please see the group list for a listing of the SIGs, working groups, and committees available. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/transfer kubeadm |
@maxl99 the unwritten kubeadm policy about coredns is the following, roughly - if you'd like customizations you can deploy your own coredns and manage it. that said we did add the feature to preserve the deployment replica count, so maybe we can also preserve other aspects of the existing deployment. but not such that would block the coredns migrator code (like "image" IIRC). your PR comes 6 days before code freeze for 1.30 so i think it's a problem merging it without discussion: |
/remove-kind bug |
@neolit123 In my opinion it would be nice to preserve some specs in the deployment/read them from the existing k8s deplyoment. It is really bad to hardcode mem limits like in coredns (170M) without an option to change it. |
can we understand better what else can be preserved from the deployment and handle it all in a single pr? |
In our env we touched only the resourceRequirements and replicas(already handled by kubernetes/kubernetes#85837). But I imagine that there are also some other points for other people. |
IMO, if we supported every feature(replica、resource、label、annotation ...) this way, it would make the code difficult to maintain. |
Can we support kube-proxy and coredns patches like control plane components patches in https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-Patches? BTW, can we skip dns upgrade like etcd?
|
kube-proxy is a DS so it's controlled by a single config (which we create on init), arguably doesn't need patches.
sadly our dns code is already a mess because of the coredns migration integration. is "coredns" as a patch target OK, or should it be something like "corednsdeployment"? @maxl99 |
any concerns about adding this patch target?
best to be explicit about the object type |
With kubernetes/kubernetes#124820, you can configure coredns resource requirements in the following way: $ mkdir patches
$ vim ./patches/corednsdeployment+strategic.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
spec:
template:
spec:
containers:
- name: coredns
resources:
limits:
memory: 2Gi
cpu: 2
requests:
memory: 1Gi
cpu: 1
$ kubeadm upgrade apply <version> --patches ./patches |
closing as completed. 1.31 has the new feature to patch corednsdeployment |
What happened?
After an upgrade with kubeadm the resourceRequirements will be reset.
Currently the memory limit is hardcoded to 170M in the template.
In our big prod cluster (1000 nodes,5k services, 20k pods) coreDNS needs more than 170M.
So we increased the mem limit in the deployment after an outage (all coredns pods got OOM killed)
After kubeadm upgrade the deployment gets updated to 170M
-> Due to that all coredns pods rotate(during startup the probes are sucessful) and will get oomkilled after loading/caching all dns entries
What did you expect to happen?
keep resourceRequests after kubeadm upgrade
How can we reproduce it (as minimally and precisely as possible)?
Anything else we need to know?
No response
Kubernetes version
Cloud provider
OS version
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
The text was updated successfully, but these errors were encountered: