Make ASG Warmpool depend on ASG Lifecycle hook #16583
Conversation
Fixes: kubernetes#16582 When creating a new ASG with a Warmpool using Lifecycle hooks, the instances that first join the Warmpool when the ASG is created could come up before the Lifecycle hook is in effect. This can lead to problems such as those instances not calling the Lifecycle hook notification when pressed into service, causing the ASG to terminate them approximately 10 mins after they've been performing work in the cluster. Setting a dependency on for the Warmpool to take into account any Lifecycle hooks means that the Warmpool won't be created until the hooks are ready. Signed-off-by: Jim Barber <[email protected]>
k8s-ci-robot
added
cncf-cla: yes
|
Hi @jim-barber-he. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/S
|
Question: is this race specific to the instances in the WarmPool, or can it apply to the (main) ASG instances also? I'm wondering if we should be setting LifecycleHookSpecificationList in CreateAutoScalingGroup... |
It happens to all the instances that are created at the time the ASG is created and before its lifecycle hook is in effect. So all hosts are affected, but only the Warmpool ones are reaped when they later go into service, because by that time the Lifecycle hook is in place on the ASG but they don't seem to attempt to call it.
I attempted to fix it for all hosts using the Since the first instances that skip the Warmpool when the ASG is created are safe from being reaped by the ASG and it was only the Warmpool instances at risk I figured I'd raise this PR as it gets us out of hot water. |
|
Thanks @jim-barber-he , this makes a lot of sense and is a good fix for warmpools, even if we end up (somehow) also precreating the hooks with LifecycleHookSpecificationList (for non-warmpools). Thanks also for the detailed comments on the issue (#16582), I'm going to look that over and see if I have any ideas! /approve |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: justinsb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
|
Are you okay if I raise a PR to cherry pick this back to kOps 1.29 as well? |
That would be wonderful, thank you! If you haven't seen them, there is a script you can use described here: https://github.com/kubernetes/kops/blob/master/docs/contributing/proposing-a-cherry-pick.md |
…f-#16583-upstream-release-1.29 Automated cherry pick of #16583: Make ASG Warmpool depend on ASG Lifecycle hook
Fixes: #16582
When creating a new ASG with a Warmpool using Lifecycle hooks, the instances that first join the Warmpool when the ASG is created could come up before the Lifecycle hook is in effect.
This can lead to problems such as those instances not calling the Lifecycle hook notification when pressed into service, causing the ASG to terminate them approximately 10 mins after they've been performing work in the cluster.
Setting a dependency on for the Warmpool to take into account any Lifecycle hooks means that the Warmpool won't be created until the hooks are ready.