Restarting k8s: webhook not ready in time

[9Rune5]

I stopped and then started my k8s cluster.

My pods started initializing long before the hook. As a result there were no AZURE_xxx environment variables in play and all my pods failed.

After running kubectl delete and kubectl apply on each pod everything was working again.

How can I make my deployment wait for the hook to initialize, or encourage the hook to redeploy my pods?

[aramase]

@9Rune5

1. You can configure the webhook failurePolicy to Fail instead of Ignore
   

   azure-workload-identity/deploy/azure-wi-webhook.yaml

   Line 253 in df6362a

   failurePolicy: Ignore

2. Label all the workloads that need to use the workload identity webhook (ref: https://azure.github.io/azure-workload-identity/docs/topics/service-account-labels-and-annotations.html#labels-1).
3. Add a labelSelector in the mutating webhook configuration to only mutate pods with the "azure.workload.identity/use": "true" label.

Note: we are going to make the above changes default in the upcoming v1.0.0 release (xref: #601)

[9Rune5]

v1.0.0, is that approximately 23Q1?

[mikerains]

I think I have the same issue, and resolve it by "restarting" the pods - by setting the workload replicas to 0, then back to a positive number.

ie:
kubectl scale deployment <deployment_name> replicas=0
kubectl scale deployment <deployment_name> replicas=1

[mikerains]

@aramase you wrote "3. Add a labelSelector in the mutating webhook configuration to only mutate pods with the "azure.workload.identity/use": "true" label."

I've configured AKS to use Workload Identity following MSLearn Docs Deploy and configure cluster, ie:
az aks create -g myResourceGroup -n myAKSCluster --node-count 1 --enable-oidc-issuer --enable-workload-identity

I am assuming --enable-workload-identity installed the Mutating Admission Webhook

From that starting point, how would I implement step 3? ie:

1. pull the helm chart into my own Azure Container Registry using it as a local Helm chart registry, like this
2. modify the MutatingWebhookConfiguration in config/webhook/manifests.yaml to add a labelSelector
3. install from my modified registry

What would the labelSelector be and where does it go in the MutatingWebhookConfiguration ? ie:

webhooks:
  objectSelector:
    matchLabels:
      azure.workload.identity/use: true

[mikerains]

Hi @9Rune5 --Thanks. That link you provided is the same as the link I gave in "Mutating Admission Webhook" above; and using --enable-workload-identity did provide the 2 WI pods you mention. Hence my surmise that --enable-workload-identity is installing the admission controller.

That part of my post was just context of how I installed in case that became important to the answer. But the question in my post was asking how I would perform step 3 (and pertains to step 1 as well) that @aramase gave in his answer. I don't know how I would modify the Workload Identity helm package in those steps and was looking for advice.

[mikerains]

Correction: --enable-workload-identity did provide 2 azure-wi-webhook-controller-manager pods, but they are in kube-system namespace.

I have to assume that installing the Mutating Admission Webhook is not required when using --enable-workload-identity, because otherwise the documented approach in MSLearn's Deploy and configure cluster would be completely missing that step.

I also can only assume the namespace is not important, other than it would be better to put the WI pods into their own namespace rather than polluting the kube-system namespace.

[9Rune5]

Thanks @mikerains, that looks like an easier way of deploying the webhooks. (but... will updating them be as easy?)

[mikerains]

@9Rune5 --- that's a good point!

[mikerains]

Actually, I think I could make the modifications that @aramase provided (steps 1 and 3 above) on the already-installed AKS cluster by
obtaining the MutatingWebhookConfiguration, modifying it and re-applying it.

kubectl get MutatingWebhookConfiguration azure-wi-webhook-mutating-webhook-configuration -o yaml > mycopy.yaml
<edit mycopy.yaml>
kubectl apply -n kube-system -f mycopy.yaml

One could even use kubectl edit but that's a little to risky for me. I would save a copy of the existing yaml as well in order to be able to revert to original.