You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. What kops version are you running? The command kops version, will display
this information.
1.28
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.28
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
kops1.28.4 replace --force -f /path/to/kops.yaml
5. What happened after the commands executed?
Error: error replacing cluster: spec.cloudProvider.aws.nodeTerminationHandler.enableScheduledEventDraining: Forbidden: scheduled event draining cannot be disabled in Queue Processor mode
6. What did you expect to happen?
I would expect to be able to have enabledScheduledEventDraining disabled in the config while in SQS mode. The kops validation is running this code which is problematic:
funcvalidateNodeTerminationHandler(cluster*kops.Cluster, spec*kops.NodeTerminationHandlerSpec, fldPath*field.Path) (allErrs field.ErrorList) {
ifspec.IsQueueMode() {
ifspec.EnableSpotInterruptionDraining!=nil&&!*spec.EnableSpotInterruptionDraining {
allErrs=append(allErrs, field.Forbidden(fldPath.Child("enableSpotInterruptionDraining"), "spot interruption draining cannot be disabled in Queue Processor mode"))
}
ifspec.EnableScheduledEventDraining!=nil&&!*spec.EnableScheduledEventDraining {
allErrs=append(allErrs, field.Forbidden(fldPath.Child("enableScheduledEventDraining"), "scheduled event draining cannot be disabled in Queue Processor mode"))
}
if!fi.ValueOf(spec.EnableRebalanceDraining) &&fi.ValueOf(spec.EnableRebalanceMonitoring) {
allErrs=append(allErrs, field.Forbidden(fldPath.Child("enableRebalanceMonitoring"), "rebalance events can only drain in Queue Processor mode"))
}
}
returnallErrs
}
Based on the AWS Node Termination Handler documention, enableScheduledEventDraining is only applicable in IMDS mode . While performing kops and kubernetes upgrades of our cluster, we ran into the error above.
Looking at the AWS Node Termination Handler source code, we can see that scheduled event draining is only used when !imdsDisabled (or when imds is enabled)
if!imdsDisabled&&nthConfig.EnableScheduledEventDraining {
//will retry 4 times with an interval of 2 seconds.pollCtx, cancelPollCtx:=context.WithTimeout(context.Background(), 8*time.Second)
err=wait.PollUntilContextCancel(pollCtx, 2*time.Second, true, func(context.Context) (donebool, errerror) {
err=handleRebootUncordon(nthConfig.NodeName, interruptionEventStore, *node)
iferr!=nil {
log.Warn().Err(err).Msgf("Unable to complete the uncordon after reboot workflow on startup, retrying")
}
returnfalse, nil
})
iferr!=nil {
log.Warn().Err(err).Msgf("All retries failed, unable to complete the uncordon after reboot workflow")
}
cancelPollCtx()
}
We should be able to disable Scheduled Event Draining while in SQS mode since it has no impact @johngmyers. Maybe I'm missing something here?
7. Please provide your cluster manifest.
This is the relevant part:
8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
9. Anything else do we need to know?
The text was updated successfully, but these errors were encountered:
/kind bug
1. What
kops
version are you running? The commandkops version
, will displaythis information.
1.28
2. What Kubernetes version are you running?
kubectl version
will print theversion if a cluster is running or provide the Kubernetes version specified as
a
kops
flag.1.28
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
5. What happened after the commands executed?
6. What did you expect to happen?
I would expect to be able to have
enabledScheduledEventDraining
disabled in the config while in SQS mode. The kops validation is running this code which is problematic:Based on the AWS Node Termination Handler documention,
. While performing kops and kubernetes upgrades of our cluster, we ran into the error above.
enableScheduledEventDraining
is only applicable in IMDS modeLooking at the AWS Node Termination Handler source code, we can see that scheduled event draining is only used when
!imdsDisabled
(or when imds is enabled)We should be able to disable Scheduled Event Draining while in SQS mode since it has no impact @johngmyers. Maybe I'm missing something here?
7. Please provide your cluster manifest.
This is the relevant part:
8. Please run the commands with most verbose logging by adding the
-v 10
flag.Paste the logs into this report, or in a gist and provide the gist link here.
9. Anything else do we need to know?
The text was updated successfully, but these errors were encountered: