Kubeadm: check health status of all control plane components in wait-control-plane phase for inti #119598

tommas1988 · 2023-07-26T16:37:25Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fixed kubeadm init command of wait-control-plane phase dose not check health status of all control plane components

Which issue(s) this PR fixes:

Fixes kubernetes/kubeadm#2907

Special notes for your reviewer:

Does this PR introduce a user-facing change?

kubeadm: 
Introduce a new feature: `WaitForAllControlPlaneComponents` to support waiting for all control plane components  to be ready, when creating a control plane node with `init` or `join` sub command

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2023-07-26T16:37:27Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2023-07-26T16:37:29Z

Please note that we're already in Test Freeze for the release-1.28 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.28.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Wed Jul 26 16:09:42 UTC 2023.

linux-foundation-easycla · 2023-07-26T16:37:32Z

The committers listed above are authorized under a signed CLA.

✅ login: tommas1988 / name: Tommas Yuan (b30ec06, 823dc89, 3f4ad62, a00df8d, b041dcc)

k8s-ci-robot · 2023-07-26T16:37:33Z

Welcome @tommas1988!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2023-07-26T16:37:33Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2023-07-26T16:37:34Z

Hi @tommas1988. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2023-07-26T16:37:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tommas1988
Once this PR has been reviewed and has the lgtm label, please assign pacoxu for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

cmd/kubeadm/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

neolit123

the PR has one blocker about imports / code organization.
but let's continue the discussion on the ticket, to decide if we want this.

also we are in code freeze for 1.28.

neolit123 · 2023-07-26T19:07:39Z

cmd/kubeadm/.import-restrictions

@@ -3,3 +3,4 @@ rules:
  - selectorRegexp: k8s[.]io/kubernetes
    allowedPrefixes:
      - k8s.io/kubernetes/cmd/kubeadm
+      - k8s.io/kubernetes/pkg/probe


kubeadm must not import kubernetes/pkg

Why kubeadm is limited from importing kubernetes/pkg

Please see kubernetes/kubeadm#1600

kubernetes/pkg contains tools that must not be imported by external projects.
at the same time, we want to move kubeadm outside of kubernetes/kubernetes one day.

neolit123 · 2023-07-26T19:08:38Z

cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go

+	"k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm"
+	"k8s.io/kubernetes/cmd/kubeadm/app/phases/controlplane"
+	"k8s.io/kubernetes/pkg/probe"
+	httpprobe "k8s.io/kubernetes/pkg/probe/http"


as mentioned above kubeadm must not import kubernetes/pkg.

note, we use a few import groups separated by new line:

go stdlib

external (e.g. github)

k8s.io (non kubeadm)

k8s.io kubeadm

neolit123

thanks for updating the PR. i will comment on the ticket.

neolit123 · 2023-08-20T14:21:18Z

/ok-to-test

adding test runs, but note that this needs more discussion. i think.

neolit123 · 2024-01-03T13:54:59Z

cmd/kubeadm/app/cmd/phases/join/kubelet.go

+	if cfg.ControlPlane != nil && features.Enabled(initCfg.FeatureGates, features.WaitForAllControlPlaneComponents) {
+		fmt.Println("[kubelet-start] Wait for control plane components")
+		timeout := 40 * time.Second
+		err := controlplane.WaitForControlPlaneComponents(


after the kubelet start phase there is one called "control-plane-join"
can we have this logic there instead?

kubernetes/cmd/kubeadm/app/cmd/join.go

Lines 215 to 219 in 114aca5

joinRunner.AppendPhase(phases.NewPreflightPhase())

joinRunner.AppendPhase(phases.NewControlPlanePreparePhase())

joinRunner.AppendPhase(phases.NewCheckEtcdPhase())

joinRunner.AppendPhase(phases.NewKubeletStartPhase())

joinRunner.AppendPhase(phases.NewControlPlaneJoinPhase())

i don't recall if we had this discussion already.

No, we haven`t discuss this before.

There are three subphases in "control-plane-join" phase, which subphase should contain the WaitForControlPlaneComponents code? MarkControlPlane phase ?
Or create a new subphase ?

i think a new Phase{} that is hidden is best here.
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/phases/workflow/phase.go#L46C9-L46C13

perhaps wait-control-plane

neolit123 · 2024-01-03T14:03:58Z

cmd/kubeadm/app/cmd/phases/join/kubelet.go

@@ -217,6 +220,21 @@ func runKubeletStartJoinPhase(c workflow.RunData) (returnErr error) {
 		return err
 	}

+	if cfg.ControlPlane != nil && features.Enabled(initCfg.FeatureGates, features.WaitForAllControlPlaneComponents) {
+		fmt.Println("[kubelet-start] Wait for control plane components")
+		timeout := 40 * time.Second


and following on the prev comment, there should be possible to use clusterConfig.APIServer.TimeoutForControlPlane.Duration instead of 40s.

FTR, we have this pending:
#122529

but my proposal is to still have a single timeout value for any CP component - 4m.

…command

SataQiu · 2024-01-25T06:56:24Z

cmd/kubeadm/app/cmd/init.go

 	initRunner.AppendPhase(phases.NewUploadConfigPhase())
 	initRunner.AppendPhase(phases.NewUploadCertsPhase())
+	initRunner.AppendPhase(phases.NewWaitControlPlanePhase())


I'm a little confused, why change the sequence of phases?
I'm afraid executing UploadConfigPhase before the control plane is ready will fail.

kubelet needs cluster-admins role binding exists when requesting pods from kubelet. And initData.Client() will make sure the role binding exists before returning api client.

I have reverted init.go back, and create a new commit to move this logic into controlplane/components.go

…ontrolPlaneJoin phase

…om kubelet

k8s-ci-robot · 2024-01-27T22:35:35Z

@tommas1988: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-linter-hints	`b041dcc`	link	false	`/test pull-kubernetes-linter-hints`
pull-kubernetes-verify-lint	`b041dcc`	link	false	`/test pull-kubernetes-verify-lint`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

neolit123 · 2024-02-04T13:04:45Z

cmd/kubeadm/app/phases/controlplane/components.go

+// ControlPlaneComponents contains all components in control plane
+var ControlPlaneComponents = []string{
+	kubeadmconstants.KubeAPIServer,
+	kubeadmconstants.KubeControllerManager,
+	kubeadmconstants.KubeScheduler,
+}


let's make this a function instead that returns a private list:

func ControlPlaneComponents() []string { ... }

neolit123 · 2024-02-04T13:09:49Z

cmd/kubeadm/app/phases/controlplane/components.go

+
+	if config.Authorization.Mode == kubeletconfig.KubeletAuthorizationModeWebhook {
+		// make sure cluster admins role binding is created, thus request to kubelet will pass server authorization
+		if _, err := kubeconfig.EnsureAdminClusterRoleBinding(kubeadmconstants.KubernetesDir, nil); err != nil {


isn't the RB already created at this point?
it is one of the first things that happen during kubeadm init and for joining CP nodes it should be there already.

isn't the RB already created at this point? it is one of the first things that happen during kubeadm init and for joining CP nodes it should be there already.

This RB is created in function initData.Client() during kubeadm init, and initData.Client() is ever not called till WaitControlPlanePhase is reached

if the CRB is required earlier perhaps we can call a _ = initData.Client() instead of calling kubeconfig.EnsureAdminClusterRoleBinding. that is because we want this Client() function to set the right vars / state. getKubeletEndpoint() should just return the endpoint and not perform CRB creation.

in runWaitControlPlanePhase there is a client, err := data.ClientWithoutBootstrap(), but this client doesn't have permissions.

but, in this discussion we talked about using the apiserver client cert/key
https://github.com/kubernetes/kubernetes/pull/119598/files#r1430504568
and the code in this PR is still using it. i'm not sure i understand why we need to bootstrap the admin.conf client?

i.e.

// make sure cluster admins role binding is created, thus request to kubelet will pass server authorization

kubeconfig.EnsureAdminClusterRoleBinding() ensures that the kubeadm:cluster-admins group (which admin.conf is part of) is grated the cluster-admin privilege. it seems unrelated to getting the pods with the apiserver client cert/key. that user should already be able to get the pods from the kubelet, no?

but, in this discussion we talked about using the apiserver client cert/key
https://github.com/kubernetes/kubernetes/pull/119598/files#r1430504568
and the code in this PR is still using it. i'm not sure i understand why we need to bootstrap the admin.conf client?

previously waitconrolpane,go use data.Client() to get client which is changed by this commit.

kubeconfig.EnsureAdminClusterRoleBinding() ensures that the kubeadm:cluster-admins group (which admin.conf is part of) is grated the cluster-admin privilege. it seems unrelated to getting the pods with the apiserver client cert/key. that user should already be able to get the pods from the kubelet, no?

kubelet server will install auth filter with Authorization configuration, and when Authorization.Mode = Webhook (which is default), kubelet server will create SubjectAccessReview to authorize requests to kubelet.

if the CRB is required earlier perhaps we can call a _ = initData.Client() instead of calling kubeconfig.EnsureAdminClusterRoleBinding. that is because we want this Client() function to set the right vars / state. getKubeletEndpoint() should just return the endpoint and not perform CRB creation.

Why put this code in getKubeletEndpoint() is only use _ = initData.Client() is not clear why request to kubelet will be failed without kubeadm:cluster-admins RB is created (I did spend some time to find why request to kubelet return athorize error).

Now, I need to revert client, err := data.ClientWithoutBootstrap() to client, err := data.Client() or _ = data.Client() when this feature gate is enabled in waitcontrolplane?

Now, I need to revert client, err := data.ClientWithoutBootstrap() to client, err := data.Client() or _ = data.Client() when this feature gate is enabled in waitcontrolplane?

there might be a "chicken and egg" problem, because data.ClientWithoutBootstrap was used because data.Client() does not work yet. i.e. the CRB cannot be performed due to apiserver pod not ready. you can try it, of course.

kubelet server will install auth filter with Authorization configuration, and when Authorization.Mode = Webhook (which is default), kubelet server will create SubjectAccessReview to authorize requests to kubelet.

i see, so it's because of this:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/phases/certs/certlist.go#L294
apiserver-kubelet-client needs the admin credentials, to get pods or do other actions.

if it's possible to create the Client() earlier that might be best, but if it's not, perhaps we can:

during init use the credentials from super-admin.conf

during join just use admin.conf or apiserver-kubelet-client. at that point the CRB should already exist.

neolit123 · 2024-02-04T13:12:10Z

cmd/kubeadm/app/phases/controlplane/components.go

+
+		resp, err := client.Get(kubeletEndpoint)
+		if err != nil {
+			fmt.Printf("[kubelet client] Error getting pods [%v]\n", err)


instead of printing these errors we can store the last error as a lastError and return it?

err := wait.PollUntilContextTimeout... if err != nil { return lastError }

neolit123 · 2024-02-04T13:16:37Z

cmd/kubeadm/app/phases/controlplane/components.go

+					}
+				}
+
+				comp.touched = true


on a quick look it's not obvious why we the code is tracking labels and touched state. can you elaborate why status.Ready is not enough?

on a quick look it's not obvious why we the code is tracking labels and touched state. can you elaborate why status.Ready is not enough?

component.touched: bool is dealing with the case: when pods info returned from kubelet cannot be found with this component.labels.
I will add some comments on component struct

ok, i will check the comments later.
currently, i still don't get why we need to the labels and touched flags.

neolit123 · 2024-02-04T13:20:05Z

cmd/kubeadm/app/phases/controlplane/components.go

+		components[i] = &component{name, labels, false}
+	}
+
+	return wait.PollUntilContextTimeout(context.Background(), 5*time.Second, timeout, false, func(ctx context.Context) (bool, error) {


can we use the same retry interval as the legacy waiter for API server:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/util/apiclient/wait.go#L82
or it's too short?

neolit123 · 2024-02-04T13:20:37Z

cmd/kubeadm/app/phases/controlplane/components.go

+			return false, nil
+		}
+
+		defer resp.Body.Close()


unless i'm mistaken the linter in this repo will complain that the return value is not checked.

unless i'm mistaken the linter in this repo will complain that the return value is not checked.

Yes, the linter did complaint with this, but it is not a required check, and i also check some old code that did not check the returned err with closing a response body, so I just levea is as it is.

I will check this returned error in new commit

neolit123 · 2024-02-04T13:28:21Z

cmd/kubeadm/app/cmd/phases/join/controlplanejoin.go

+func newWaitControlPlaneComponentsSubphase() workflow.Phase {
+	return workflow.Phase{
+		Name:   "wait-control-plane-components",
+		Hidden: true,


a new hidden phases seems ok.
i guess on join we are not calling WaitForAPI or have a dedicated phase to wait for the apiserver to come up?

a new hidden phases seems ok. i guess on join we are not calling WaitForAPI or have a dedicated phase to wait for the apiserver to come up?

Do you mean we need to add legacy WaitForAPI code when this feature gate is disabled in this hidden phase

no, i'm just trying to remember what happens on join.
i guess we are not even calling WaitForAPI on join, so adding the new hidden phase seems OK.
later if we decide we can make it non-hidden.

neolit123

give this a try:
https://github.com/kubernetes/kubernetes/pull/119598/files#r1477736950

if it doesn't work i can try downloading the PR locally and check what can be done.

tommas1988 · 2024-02-06T06:55:09Z

give this a try: https://github.com/kubernetes/kubernetes/pull/119598/files#r1477736950

if it doesn't work i can try downloading the PR locally and check what can be done.

Given the credential data from super-admin.conf also dosen`t work.

And after reading code of kubelet and kube-apiserver, it seems that kubelet will use a RequestAttributes that contains nodes/proxy resource to ask apiserver to check request authorization. And this RequestAttributes is only allowed for RABC roles of cluster-admin and system:kubelet-api-admin by apiserver.

by the way, is that reasonable for creating cluster-admin rolebinding when waiting for controlplane components, and when it failed, which it just means that apiserver is failed to start?

neolit123 · 2024-02-06T10:49:53Z

give this a try: https://github.com/kubernetes/kubernetes/pull/119598/files#r1477736950
if it doesn't work i can try downloading the PR locally and check what can be done.

Given the credential data from super-admin.conf also dosen`t work.

it doesn't?
super-admin.conf uses system:masters group which bypasses rbac, so it should work.

And after reading code of kubelet and kube-apiserver, it seems that kubelet will use a RequestAttributes that contains nodes/proxy resource to ask apiserver to check request authorization. And this RequestAttributes is only allowed for RABC roles of cluster-admin and system:kubelet-api-admin by apiserver.

i am surprised if system:masters doesn't work for this. one of the reasons it exists is to create initial rbac rules on bootstrap.

by the way, is that reasonable for creating cluster-admin rolebinding when waiting for controlplane components, and when it failed, which it just means that apiserver is failed to start?

not sure I understand the question.
we could try creating the crb before waiting for the components. i will try your pr locally later today or tomorrow.

but please tell me if you have more ideas.

neolit123 · 2024-02-06T17:49:41Z

@tommas1988
i remember an earlier version of this PR used probe code from /pkg, kubeadm should not use /pkg code but we could still check for status 200 at /healthz of these components. i wonder why you moved away from that idea?
this doesn't have the authz problem as the kubeadm setup already uses the same method for probes on the static pods.

see this patch. i tested it locally and it works with go routines at /healthz. the code is relatively simple:

diff --git a/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go b/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go
index 8b746f288c3..f0b0a9587ce 100644
--- a/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go
+++ b/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go
@@ -112,7 +112,7 @@ func runWaitControlPlanePhase(c workflow.RunData) error {
 	}
 
 	waiter.SetTimeout(data.Cfg().Timeouts.ControlPlaneComponentHealthCheck.Duration)
-	if err := waiter.WaitForAPI(); err != nil {
+	if err := waiter.WaitForControlPlaneComponents(); err != nil {
 		return handleError(err)
 	}
 
diff --git a/cmd/kubeadm/app/phases/upgrade/staticpods_test.go b/cmd/kubeadm/app/phases/upgrade/staticpods_test.go
index 94ba22b66ba..017b28c7077 100644
--- a/cmd/kubeadm/app/phases/upgrade/staticpods_test.go
+++ b/cmd/kubeadm/app/phases/upgrade/staticpods_test.go
@@ -98,6 +98,11 @@ func NewFakeStaticPodWaiter(errsToReturn map[string]error) apiclient.Waiter {
 	}
 }
 
+// WaitForControlPlaneComponents just returns a dummy nil, to indicate that the program should just proceed
+func (w *fakeWaiter) WaitForControlPlaneComponents() error {
+	return nil
+}
+
 // WaitForAPI just returns a dummy nil, to indicate that the program should just proceed
 func (w *fakeWaiter) WaitForAPI() error {
 	return nil
diff --git a/cmd/kubeadm/app/util/apiclient/wait.go b/cmd/kubeadm/app/util/apiclient/wait.go
index da122b553dd..2b396e36394 100644
--- a/cmd/kubeadm/app/util/apiclient/wait.go
+++ b/cmd/kubeadm/app/util/apiclient/wait.go
@@ -18,6 +18,7 @@ package apiclient
 
 import (
 	"context"
+	"crypto/tls"
 	"fmt"
 	"io"
 	"net/http"
@@ -28,6 +29,7 @@ import (
 	v1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	utilerrors "k8s.io/apimachinery/pkg/util/errors"
 	netutil "k8s.io/apimachinery/pkg/util/net"
 	"k8s.io/apimachinery/pkg/util/wait"
 	clientset "k8s.io/client-go/kubernetes"
@@ -37,6 +39,8 @@ import (
 
 // Waiter is an interface for waiting for criteria in Kubernetes to happen
 type Waiter interface {
+	// WaitForControlPlaneComponents waits for all control plane components to report "ok" on /healthz
+	WaitForControlPlaneComponents() error
 	// WaitForAPI waits for the API Server's /healthz endpoint to become "ok"
 	WaitForAPI() error
 	// WaitForPodsWithLabel waits for Pods in the kube-system namespace to become Ready
@@ -72,6 +76,69 @@ func NewKubeWaiter(client clientset.Interface, timeout time.Duration, writer io.
 	}
 }
 
+// WaitForControlPlaneComponents waits for all control plane components to report "ok" on /healthz
+func (w *KubeWaiter) WaitForControlPlaneComponents() error {
+	fmt.Printf("[control-plane-check] Waiting for healthy control plane components."+
+		" This can take up to %v\n", w.timeout)
+
+	type component struct {
+		name string
+		port int
+	}
+	var components = []component{
+		{name: "kube-api-server", port: 6443},
+		{name: "kube-controller-manager", port: 10257},
+		{name: "kube-scheduler", port: 10259},
+	}
+	var errs []error
+	errChan := make(chan error, len(components))
+
+	for _, comp := range components {
+		go func(comp component) {
+			tr := &http.Transport{
+				TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
+			}
+			client := &http.Client{Transport: tr}
+			start := time.Now()
+			var lastError error
+
+			err := wait.PollUntilContextTimeout(
+				context.Background(),
+				constants.KubernetesAPICallRetryInterval,
+				w.timeout,
+				true, func(ctx context.Context) (bool, error) {
+					resp, err := client.Get(fmt.Sprintf("https://127.0.0.1:%d/healthz", comp.port))
+					if err != nil {
+						lastError = errors.WithMessagef(err, "%s /healthz check failed", comp.name)
+						return false, nil
+					}
+
+					defer resp.Body.Close()
+					if resp.StatusCode != http.StatusOK {
+						lastError = errors.Errorf("%s /healthz check failed with status: %d", comp.name, resp.StatusCode)
+						return false, nil
+					}
+
+					return true, nil
+				})
+			if err != nil {
+				fmt.Printf("[control-plane-check] %s is not healthy after %v\n", comp.name, time.Since(start))
+				errChan <- lastError
+				return
+			}
+			fmt.Printf("[control-plane-check] %s is healthy after %v\n", comp.name, time.Since(start))
+			errChan <- nil
+		}(comp)
+	}
+
+	for i := 0; i < len(components); i++ {
+		if err := <-errChan; err != nil {
+			errs = append(errs, err)
+		}
+	}
+	return utilerrors.NewAggregate(errs)
+}
+
 // WaitForAPI waits for the API Server's /healthz endpoint to report "ok"
 func (w *KubeWaiter) WaitForAPI() error {
 	fmt.Printf("[api-check] Waiting for a healthy API server. This can take up to %v\n", w.timeout)
diff --git a/cmd/kubeadm/app/util/dryrun/dryrun.go b/cmd/kubeadm/app/util/dryrun/dryrun.go
index b1c4e7a4712..ae34d2bdee9 100644
--- a/cmd/kubeadm/app/util/dryrun/dryrun.go
+++ b/cmd/kubeadm/app/util/dryrun/dryrun.go
@@ -88,6 +88,11 @@ func NewWaiter() apiclient.Waiter {
 	return &Waiter{}
 }
 
+// WaitForControlPlaneComponents just returns a dummy nil, to indicate that the program should just proceed
+func (w *Waiter) WaitForControlPlaneComponents() error {
+	return nil
+}
+
 // WaitForAPI just returns a dummy nil, to indicate that the program should just proceed
 func (w *Waiter) WaitForAPI() error {
 	fmt.Println("[dryrun] Would wait for the API Server's /healthz endpoint to return 'ok'")

output:

[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] kube-controller-manager is healthy after 2.460923064s
[control-plane-check] kube-scheduler is healthy after 3.831072293s
[control-plane-check] kube-api-server is healthy after 6.006457944s

EDIT: one note here, apiserver has /readyz which is better but the other components don't have it, thus i used /healthz for all of them. also the old WaitForAPI code uses /healthz anyhow.

IMPORTANT: kubeadm does not support very well --annonymous-auth=false on the kube-apiserver, but if some user decides to turn it off they will face this problem:
#43784
(long discussion)

but i guess our WaitForAPI already has this problem.

in any case, i think what we can do the following:

use /healthz checks the way i show in the diff.
if the feature gate is enabled ~~expose the phase as non-hidden and skipable, so that users can skip this health check completely.~~ on init i think we can just replace WaitForAPI in wait-control-plane. on join we should add a new phase that is skippable and non-hidden if the FG is enabled and hidden if the FG is not enabled. (we could try this approach, at least)
if the feature gate is disabled the new phases should be hidden
one the feature gate goes GA we can expose the phases as non-hidden and users can skip them.

WDYT?

pacoxu · 2024-02-07T01:30:07Z

/kind feature
/release-note-edit

Kubeadm: added an alpha feature gate WaitForAllControlPlaneComponents to support waiting for all control plane components to be ready for `kubeadm init` or `kubeadm join`.

@tommas1988 the release note block should started with

```release-note

k8s-ci-robot · 2024-02-07T01:30:09Z

@pacoxu: /release-note-edit must be used with a single release note block.

In response to this:

/kind feature
/release-note-edit

Kubeadm: added an alpha feature gate WaitForAllControlPlaneComponents to support waiting for all control plane components to be ready for `kubeadm init` or `kubeadm join`.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

neolit123 · 2024-02-07T06:45:37Z

the release note block should started with

https://raw.githubusercontent.com/kubernetes/kubernetes/master/.github/PULL_REQUEST_TEMPLATE.md

neolit123 · 2024-02-16T14:44:22Z

i opened a new PR for the same feature
#123341

neolit123

closing in favor of #123341

neolit123 · 2024-02-27T07:01:31Z

/close

k8s-ci-robot · 2024-02-27T07:01:37Z

@neolit123: Closed this PR.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Jul 26, 2023

k8s-ci-robot added the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 26, 2023

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 26, 2023

k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 26, 2023

k8s-ci-robot added area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 26, 2023

k8s-ci-robot requested review from neolit123 and SataQiu July 26, 2023 16:38

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 26, 2023

neolit123 reviewed Jul 26, 2023

View reviewed changes

tommas1988 mentioned this pull request Jul 27, 2023

graduate WaitForAllControlPlaneComponents to GA kubernetes/kubeadm#2907

Open

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 28, 2023

tommas1988 requested a review from neolit123 August 6, 2023 21:28

neolit123 reviewed Aug 20, 2023

View reviewed changes

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Aug 20, 2023

k8s-ci-robot removed the do-not-merge/contains-merge-commits Indicates a PR which contains merge commits. label Dec 26, 2023

neolit123 reviewed Jan 3, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 15, 2024

tommas1988 added 3 commits January 25, 2024 00:26

support waiting for all control plane components to be ready in init …

b30ec06

…command

support waiting for all control plane components to be ready in join …

823dc89

…command

fix pull-kubernetes-verify error

3f4ad62

tommas1988 force-pushed the fix_control_plane_status_check branch from 9d779ff to c37d1e8 Compare January 24, 2024 16:36

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 24, 2024

tommas1988 force-pushed the fix_control_plane_status_check branch from c37d1e8 to e432781 Compare January 24, 2024 16:45

SataQiu reviewed Jan 25, 2024

View reviewed changes

tommas1988 added 2 commits January 27, 2024 08:10

move WaitForControlPlaneComponents logic from KubeletStart phase to C…

a00df8d

…ontrolPlaneJoin phase

make sure cluster-admins role binding exists, when requesting pods fr…

b041dcc

…om kubelet

tommas1988 force-pushed the fix_control_plane_status_check branch from e432781 to b041dcc Compare January 27, 2024 21:59

neolit123 reviewed Feb 4, 2024

View reviewed changes

neolit123 reviewed Feb 5, 2024

View reviewed changes

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Feb 7, 2024

neolit123 mentioned this pull request Feb 7, 2024

kubeadm: make a control-plane's kubelet talk to the local API Server on kubeadm join. kubernetes/enhancements#4471

Open

5 tasks

neolit123 mentioned this pull request Feb 16, 2024

kubeadm: introduce the WaitForAllControlPlaneComponents feature gate #123341

Merged

neolit123 reviewed Feb 27, 2024

View reviewed changes

k8s-ci-robot closed this Feb 27, 2024

	joinRunner.AppendPhase(phases.NewPreflightPhase())
	joinRunner.AppendPhase(phases.NewControlPlanePreparePhase())
	joinRunner.AppendPhase(phases.NewCheckEtcdPhase())
	joinRunner.AppendPhase(phases.NewKubeletStartPhase())
	joinRunner.AppendPhase(phases.NewControlPlaneJoinPhase())

Kubeadm: check health status of all control plane components in wait-control-plane phase for inti #119598

Kubeadm: check health status of all control plane components in wait-control-plane phase for inti #119598

Conversation

tommas1988 commented Jul 26, 2023 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jul 26, 2023

k8s-ci-robot commented Jul 26, 2023

linux-foundation-easycla bot commented Jul 26, 2023 • edited Loading

k8s-ci-robot commented Jul 26, 2023

k8s-ci-robot commented Jul 26, 2023

k8s-ci-robot commented Jul 26, 2023

k8s-ci-robot commented Jul 26, 2023

neolit123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 Jul 26, 2023 • edited Loading

Choose a reason for hiding this comment

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 commented Aug 20, 2023

Choose a reason for hiding this comment

tommas1988 Jan 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 Feb 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tommas1988 Feb 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

neolit123 left a comment

Choose a reason for hiding this comment

tommas1988 commented Feb 6, 2024

neolit123 commented Feb 6, 2024 • edited Loading

neolit123 commented Feb 6, 2024 • edited Loading

pacoxu commented Feb 7, 2024 • edited Loading

k8s-ci-robot commented Feb 7, 2024

neolit123 commented Feb 7, 2024

neolit123 commented Feb 16, 2024

neolit123 left a comment

Choose a reason for hiding this comment

neolit123 commented Feb 27, 2024

k8s-ci-robot commented Feb 27, 2024

tommas1988 commented Jul 26, 2023 •

edited

Loading

linux-foundation-easycla bot commented Jul 26, 2023 •

edited

Loading

neolit123 Jul 26, 2023 •

edited

Loading

tommas1988 Jan 3, 2024 •

edited

Loading

neolit123 Feb 4, 2024 •

edited

Loading

tommas1988 Feb 5, 2024 •

edited

Loading

neolit123 commented Feb 6, 2024 •

edited

Loading

neolit123 commented Feb 6, 2024 •

edited

Loading

pacoxu commented Feb 7, 2024 •

edited

Loading