Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Upgrade v1.2.1->v1.2.2 stuck at Pre-drained node due to certificates expiration #5828

Open
W0n9 opened this issue May 17, 2024 · 11 comments
Open

[BUG] Upgrade v1.2.1->v1.2.2 stuck at Pre-drained node due to certificates expiration #5828

W0n9 opened this issue May 17, 2024 · 11 comments
Assignees
@starbops
Labels
kind/bug Issues that are defects reported by users or that we know have reached a real release require/doc Improvements or additions to documentation severity/needed Reminder to add a severity label and to remove this one
Projects
Community Issue Review

Comments

@W0n9
Copy link

W0n9 commented May 17, 2024 •
edited

Describe the bug

image
image
image
image
image

To Reproduce
Steps to reproduce the behavior:

  1. Upgrade harvester from v1.2.1 to v1.2.2

Expected behavior

Cluster should be upgraded successfully

Support bundle

hvst-upgrade-hjw2p-upgradelog-archive-2024-05-17T07-59-36Z.zip

Environment

  • Harvester ISO version: v1.2.2
  • Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): Baremetal with Dell PowerEdge R730

Additional context
Add any other context about the problem here.
@W0n9 W0n9 added kind/bug Issues that are defects reported by users or that we know have reached a real release reproduce/needed Reminder to add a reproduce label and to remove this one severity/needed Reminder to add a severity label and to remove this one labels May 17, 2024
@rebeccazzzz rebeccazzzz added this to New in Community Issue Review via automation May 17, 2024
@irishgordo
Copy link

irishgordo commented May 17, 2024 •
edited

@ibrokethecloud noticing:

Already installed harvester-seeder
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis harvester-seeder --namespace harvester-system
2024/05/17 07:20:28 Release 'harvester-seeder' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2024/05/17 07:20:28 Get release 'harvester-seeder' latest version.
2024/05/17 07:20:28 Check release 'harvester-seeder' for deprecated or removed APIs...
2024/05/17 07:20:28 Finished checking release 'harvester-seeder' for deprecated or removed APIs.
2024/05/17 07:20:28 Release 'harvester-seeder' has no deprecated or removed APIs.
2024/05/17 07:20:28 Map of release 'harvester-seeder' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading harvester-seeder'
+ shift 1
Upgrading harvester-seeder
+ helm_v3 upgrade --repo http://harvester-cluster-repo.cattle-system.svc/charts --version 0.1.0 harvester-seeder harvester-seeder --values /config/values-01_HelmChart.yaml
Error: chart "harvester-seeder" version "0.1.0" not found in http://harvester-cluster-repo.cattle-system.svc/charts repository

Specifically, the Error: chart "harvester-seeder" version "0.1.0" not found in http://harvester-cluster-repo.cattle-system.svc/charts repository about 6/7 times in: harvester-system.helm-install-harvester-seeder-9gccc.helm.20240517.log (extracted from .gz)

But the pre-drain on dc-0 looks to be successful:

nvim harvester-system.hvst-upgrade-hjw2p-pre-drain-harvester-dc-0-xsdr2.apply.20240517.log

With the tail-end being:

Volume pvc-bf6c793f-7597-4dcb-9842-a9e862d05d72 is healthy.
Checking running engine pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a-e-ec25015f...
+ echo Checking running engine pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a-e-ec25015f...
+ '[' true ']'
+ '[' 4 -gt 2 ']'
++ kubectl get volumes.longhorn.io/pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a -n longhorn-system -o 'jsonpath={.status.robustness}'
+ robustness=healthy
+ '[' healthy = healthy ']'
+ echo 'Volume pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a is healthy.'
Volume pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a is healthy.
+ break
+ read -r lh_engine lh_volume
+ upgrade-helper vm-live-migrate-detector harvester-dc-0 --shutdown
time="2024-05-17T07:14:02Z" level=info msg="Starting VM Live Migrate Detector"
time="2024-05-17T07:14:02Z" level=info msg="Non-migratable VM(s): []"
+ kubectl taint node harvester-dc-0 --overwrite kubevirt.io/drain=draining:NoSchedule
node/harvester-dc-0 modified
+ wait_vms_out_or_shutdown
+ local vm_count
+ '[' true ']'
++ get_running_vm_count
++ local vmioutput=/tmp/vmioutput.yaml
++ rm -f /tmp/vmioutput.yaml
++ local EXIT_CODE=0
++ kubectl get vmi -A -l kubevirt.io/nodeName=harvester-dc-0 -o yaml
++ [[ 0 -eq 0 ]]
+++ yq e '.items |  map(select(.status.phase!="Succeeded")) | length' /tmp/vmioutput.yaml
++ local count=0
++ echo 0
++ rm -f /tmp/vmioutput.yaml
+ vm_count=0
+ '[' 0 = 0 ']'
+ break
+ echo 'all VMs on node harvester-dc-0 have been live-migrated or shutdown'
+ wait_evacuation_pdb_gone
all VMs on node harvester-dc-0 have been live-migrated or shutdown
+ kubectl get pdb -o name -A
+ grep kubevirt-migration-pdb-kubevirt-evacuation-
+ patch_logging_event_audit
+ detect_upgrade
++ kubectl get upgrades.harvesterhci.io hvst-upgrade-hjw2p -n harvester-system -o yaml
+ upgrade_obj='apiVersion: harvesterhci.io/v1beta1

@Vicente-Cheng
Copy link
Contributor

Hi @W0n9,
Could you also generate a support bundle for investigation?
Ref: https://docs.harvesterhci.io/v1.2/troubleshooting/harvester/#generate-a-support-bundle

@W0n9
Copy link
Author

W0n9 commented May 18, 2024

Here is my support bundle @Vicente-Cheng

Yesterday, after upgrade failed:
supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-17T09-01-49Z.001.zip
supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-17T09-01-49Z.002.zip

Today, after some fix but failed again:
supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-18T02-54-16Z.001.zip
supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-18T02-54-16Z.002.zip

@Vicente-Cheng
Copy link
Contributor

Hi @W0n9,

Could you also provide the command to unzip this?
I use 7-zip, but it looks like it is not correct.

If the SB is too large to upload, you could send the bundle to [email protected] with this issue ID.
Thanks!

@W0n9
Copy link
Author

W0n9 commented May 18, 2024 •
edited

Hi, @Vicente-Cheng

You can try to rename it from file_name.001.zip to file_name.zip.001 .

I have sent e-mail to you.

@bk201
Copy link
Member

bk201 commented May 20, 2024

@starbops Please help check this issue, thanks.

@starbops
Copy link
Member

Hi @W0n9, I couldn't find any YAML files in the support bundle files you provided, so it's hard to determine the cluster's precise state. Did you ever change the support-bundle-namespaces setting before generating the support bundle file?

Under the Advanced -> Settings page of the dashboard:
image

Please also help provide the Upgrade and Setting CRs. You could run the following commands and get the output:

kubectl -n harvester-system get upgrades.harvesterhci -o yaml
kubectl get settings.harvesterhci -o yaml

From the rancher logs (supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-17T09-01-49Z/logs/cattle-system/rancher-654c45c575-zrmkv/rancher.log) we found the following line was repeated again and again:

024-05-17T08:55:00.616288694Z 2024/05/17 08:55:00 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-fb4dda4acb64: waiting for probes: kube-controller-manager, kube-scheduler

It might be related to certificates expiration. Perhaps you could check if this works for you. Thanks.

@starbops starbops mentioned this issue May 21, 2024
Open
@W0n9
Copy link
Author

W0n9 commented May 21, 2024

Hi, @starbops
I never change the support-bundle-namespaces and keep it be default.
image
Here is the YAML:

apiVersion: v1
items:
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2023-09-09T06:41:47Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 41
    labels:
      harvesterhci.io/read-message: "true"
      harvesterhci.io/upgradeState: Succeeded
    name: hvst-upgrade-f47wk
    namespace: harvester-system
    resourceVersion: "290443078"
    uid: 4382668b-352c-4263-a827-806f60e5b4c7
  spec:
    image: ""
    logEnabled: true
    version: v1.2.0
  status:
    conditions:
    - lastUpdateTime: "2023-09-09T11:52:45Z"
      status: "True"
      type: Completed
    - lastUpdateTime: "2023-09-09T06:42:07Z"
      status: "True"
      type: LogReady
    - lastUpdateTime: "2023-09-09T06:44:00Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2023-09-09T06:46:24Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2023-09-09T07:21:42Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2023-09-09T10:25:23Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-09-09T11:52:45Z"
      status: "True"
      type: NodesUpgraded
    imageID: harvester-system/harvester-iso-7x6wp
    nodeStatuses:
      harvester-dc-0:
        state: Succeeded
      harvester-dc-1:
        state: Succeeded
      harvester-dc-2:
        state: Succeeded
      harvester-dc-3:
        state: Succeeded
    previousVersion: v1.1.2
    repoInfo: |
      release:
          harvester: v1.2.0
          harvesterChart: 1.2.0
          os: Harvester v1.2.0
          kubernetes: v1.25.9+rke2r1
          rancher: v2.7.5
          monitoringChart: 102.0.0+up40.1.2
          minUpgradableVersion: v1.1.2
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2024-05-17T06:39:35Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 22
    labels:
      harvesterhci.io/latestUpgrade: "true"
      harvesterhci.io/upgradeState: UpgradingNodes
    name: hvst-upgrade-hjw2p
    namespace: harvester-system
    resourceVersion: "599965758"
    uid: 7e32dcce-8f25-4d7d-bd86-fa16c63737f1
  spec:
    image: ""
    logEnabled: true
    version: v1.2.2
  status:
    conditions:
    - status: Unknown
      type: Completed
    - lastUpdateTime: "2024-05-17T06:40:00Z"
      status: "True"
      type: LogReady
    - lastUpdateTime: "2024-05-17T06:41:04Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2024-05-17T06:43:48Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2024-05-17T06:59:11Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2024-05-17T07:13:55Z"
      status: "True"
      type: SystemServicesUpgraded
    - status: Unknown
      type: NodesUpgraded
    imageID: harvester-system/hvst-upgrade-hjw2p
    nodeStatuses:
      harvester-dc-0:
        state: Pre-drained
      harvester-dc-1:
        state: Images preloaded
      harvester-dc-2:
        state: Images preloaded
      harvester-dc-3:
        state: Images preloaded
    previousVersion: v1.2.1
    repoInfo: |
      release:
          harvester: v1.2.2
          harvesterChart: 1.2.2
          os: Harvester v1.2.2
          kubernetes: v1.26.15+rke2r1
          rancher: v2.8.2
          monitoringChart: 103.0.3+up45.31.1
          minUpgradableVersion: v1.2.1
    upgradeLog: hvst-upgrade-hjw2p-upgradelog
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2023-04-25T13:56:45Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 31
    labels:
      harvesterhci.io/read-message: "true"
      harvesterhci.io/upgradeState: Succeeded
    name: hvst-upgrade-nszm9
    namespace: harvester-system
    resourceVersion: "217748009"
    uid: 23f16027-63e5-464d-8332-22b187f9ee14
  spec:
    image: ""
    logEnabled: true
    version: v1.1.2
  status:
    conditions:
    - lastUpdateTime: "2023-04-25T15:35:16Z"
      status: "True"
      type: Completed
    - lastUpdateTime: "2023-04-25T14:00:58Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2023-04-25T14:04:05Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2023-04-25T14:36:40Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2023-04-25T14:47:01Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-04-25T15:35:16Z"
      status: "True"
      type: NodesUpgraded
    imageID: harvester-system/harvester-iso-wjfvr
    nodeStatuses:
      harvester-dc-0:
        state: Succeeded
      harvester-dc-1:
        state: Succeeded
      harvester-dc-2:
        state: Succeeded
    previousVersion: v1.1.1
    repoInfo: |
      release:
          harvester: v1.1.2
          harvesterChart: 1.1.2
          os: Harvester v1.1.2
          kubernetes: v1.24.11+rke2r1
          rancher: v2.6.11
          monitoringChart: 100.1.0+up19.0.3
          minUpgradableVersion: v1.1.0
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2023-10-27T06:56:30Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 41
    labels:
      harvesterhci.io/read-message: "true"
      harvesterhci.io/upgradeState: Succeeded
    name: hvst-upgrade-wmxxs
    namespace: harvester-system
    resourceVersion: "599919539"
    uid: 4534471a-5492-45bc-9d30-4147ce7c1cb4
  spec:
    image: ""
    logEnabled: true
    version: v1.2.1
  status:
    conditions:
    - lastUpdateTime: "2023-10-27T09:47:15Z"
      status: "True"
      type: Completed
    - lastUpdateTime: "2023-10-27T06:56:57Z"
      status: "True"
      type: LogReady
    - lastUpdateTime: "2023-10-27T06:58:48Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2023-10-27T07:02:28Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2023-10-27T07:10:38Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2023-10-27T07:11:49Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-10-27T09:47:15Z"
      status: "True"
      type: NodesUpgraded
    imageID: harvester-system/hvst-upgrade-wmxxs
    nodeStatuses:
      harvester-dc-0:
        state: Succeeded
      harvester-dc-1:
        state: Succeeded
      harvester-dc-2:
        state: Succeeded
      harvester-dc-3:
        state: Succeeded
    previousVersion: v1.2.0
    repoInfo: |
      release:
          harvester: v1.2.1
          harvesterChart: 1.2.1
          os: Harvester v1.2.1
          kubernetes: v1.25.9+rke2r1
          rancher: v2.7.5
          monitoringChart: 102.0.0+up40.1.2
          minUpgradableVersion: v1.1.2
kind: List
metadata:
  resourceVersion: ""
apiVersion: v1
items:
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: additional-ca
    resourceVersion: "8451"
    uid: 8b9493e6-d17c-4e12-a0c4-e52bf1bea2f2
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: 1.1.9
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: api-ui-version
    resourceVersion: "8448"
    uid: 969dc386-9401-403d-9373-10efa8fa73b7
  status: {}
  value: 1.1.9
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: auto-disk-provision-paths
    resourceVersion: "8469"
    uid: e7556dd1-af28-4b8e-a3a9-b9d289d9e7d0
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 29
    name: backup-target
    resourceVersion: "339358074"
    uid: 79121a2e-fef6-422f-8510-866eaf575847
  status:
    conditions:
    - lastUpdateTime: "2023-11-27T09:58:03Z"
      status: "True"
      type: configured
  value: '{"type":"s3","endpoint":"https://s3-east-internal.buct.edu.cn:9000","accessKeyId":"","secretAccessKey":"","bucketName":"vm-backups","bucketRegion":"bj","cert":"","virtualHostedStyle":false}'
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    annotations:
      harvesterhci.io/hash: 30cff57968b3b923f62143d6987eb1eb077ca0d8061b21f0db658123
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 3
    name: cluster-registration-url
    resourceVersion: "14406"
    uid: 5b7684de-3054-4cec-beec-6d6265d89bd4
  status:
    conditions:
    - lastUpdateTime: "2023-04-25T07:23:29Z"
      status: "True"
      type: configured
  value: https://rancher.buct.edu.cn/v3/import/td7vhx6j6sg6qxp88g5flhzzsdndz96xjrl7zvd2nzq2hwkp5b788t_c-m-76mpd8w7.yaml
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    annotations:
      harvesterhci.io/upgrade-patched: v1.2.2
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: containerd-registry
    resourceVersion: "599960369"
    uid: de373ce8-27d0-4004-8947-609e0f1f0315
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"driver.longhorn.io":{"volumeSnapshotClassName":"longhorn-snapshot","backupVolumeSnapshotClassName":"longhorn"}}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: csi-driver-config
    resourceVersion: "8454"
    uid: a7fd421c-5e71-42c0-ad13-2e268f6b1a16
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: longhorn
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: default-storage-class
    resourceVersion: "8460"
    uid: a6f605fb-8789-4570-aa68-ee8870da980c
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: "120"
  kind: Setting
  metadata:
    creationTimestamp: "2023-09-09T07:25:51Z"
    generation: 1
    name: default-vm-termination-grace-period-seconds
    resourceVersion: "217801286"
    uid: e488e5d9-a0f7-41ff-922e-2b7c1dfb593f
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"harvester-cloud-provider":">=0.0.1 <0.3.0","harvester-csi-provider":">=0.0.1
    <0.3.0"}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 3
    name: harvester-csi-ccm-versions
    resourceVersion: "217801288"
    uid: f9926d02-cf54-4b64-8662-dc13109361eb
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: http-proxy
    resourceVersion: "8425"
    uid: 322b0282-a4b9-4262-8095-e3e9185bc4a1
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: info
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: log-level
    resourceVersion: "8457"
    uid: 9c4e74df-9f16-4249-9fd0-da88f8932763
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-09-09T07:25:51Z"
    generation: 1
    name: ntp-servers
    resourceVersion: "217801281"
    uid: 3cf5d828-79b5-484e-abdb-4482a1d961c9
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"cpu":1600,"memory":150,"storage":200}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: overcommit-config
    resourceVersion: "8423"
    uid: 1f0fdcf1-1e50-4fc0-abd3-646f3fe0d018
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: https://releases.rancher.com/harvester
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: release-download-url
    resourceVersion: "8459"
    uid: 4ba73310-bcd8-4c2c-84c6-af147d4058fb
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: dev
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 5
    name: server-version
    resourceVersion: "599949818"
    uid: d1cbce86-a2d6-4bb4-b73a-caf15630c82b
  status: {}
  value: v1.2.2
- apiVersion: harvesterhci.io/v1beta1
  default: '{}'
  kind: Setting
  metadata:
    annotations:
      harvesterhci.io/hash: d14a028c2a3a2bc9476102bb288234c415a2b01f828ea62ac5b3e42f
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 2
    name: ssl-certificates
    resourceVersion: "8498"
    uid: 514e3aeb-5e62-4b0c-ac33-cb8a2a399182
  status:
    conditions:
    - lastUpdateTime: "2023-04-25T07:17:36Z"
      status: "False"
      type: configured
- apiVersion: harvesterhci.io/v1beta1
  default: '{}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: ssl-parameters
    resourceVersion: "8463"
    uid: da2eeb6e-0340-4aaf-85a1-e1079b2e9d51
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: storage-network
    resourceVersion: "8455"
    uid: 9ee25fee-5bbb-4a31-9663-204d04c090db
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"repository":"rancher/support-bundle-kit","tag":"v0.0.37","imagePullPolicy":"IfNotPresent"}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 66
    name: support-bundle-image
    resourceVersion: "599953683"
    uid: 9554026e-23ce-45b8-935a-e4146a7278a9
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: support-bundle-namespaces
    resourceVersion: "8461"
    uid: 886c947d-dd8e-46a9-a934-ecd0282900c9
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: "10"
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: support-bundle-timeout
    resourceVersion: "8442"
    uid: 357d73d0-7e6d-4b60-a3bf-962a7d483b49
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: https://releases.rancher.com/harvester-ui/dashboard/release-harvester-v1.2/index.html
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 2
    name: ui-index
    resourceVersion: "217801285"
    uid: 237c20d9-09c8-4d43-a38b-12f5860cd02d
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: /usr/share/harvester/harvester
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: ui-path
    resourceVersion: "8429"
    uid: 4e838361-c370-48fd-b86d-f3b3bb799c43
  status: {}
  value: /usr/share/harvester/harvester
- apiVersion: harvesterhci.io/v1beta1
  default: https://releases.rancher.com/harvester-ui/plugin/harvester-release-harvester-v1.2/harvester-release-harvester-v1.2.umd.min.js
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 2
    name: ui-plugin-index
    resourceVersion: "217801290"
    uid: 8270d656-502f-4f73-b778-5b7ae82687ca
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: auto
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: ui-source
    resourceVersion: "8470"
    uid: 839901a8-0944-474c-ac21-7469438719ef
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: upgradable-versions
    resourceVersion: "8462"
    uid: 84d85c47-5057-415b-8ba1-7040a49ae09e
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: "true"
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: upgrade-checker-enabled
    resourceVersion: "8432"
    uid: 2d2b4c3d-8a23-40df-bab6-edc60a62368c
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: https://harvester-upgrade-responder.rancher.io/v1/checkupgrade
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: upgrade-checker-url
    resourceVersion: "8437"
    uid: fa4e35fe-d384-48df-ba77-8cb12c175b9e
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: vip-pools
    resourceVersion: "8453"
    uid: 4b1dedb7-9e36-4a07-a37f-606db62aac86
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"enable":true,"period":300}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: vm-force-reset-policy
    resourceVersion: "8452"
    uid: 5fae2e04-cd67-4214-848d-9cd21b1b1fd4
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: longhorn
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: volume-snapshot-class
    resourceVersion: "8450"
    uid: 9616bfa4-5c17-42eb-84ce-32d84238401c
  status: {}
kind: List
metadata:
  resourceVersion: ""

@Vicente-Cheng
Copy link
Contributor

Hi @W0n9,
Could you also check the suggestion from @starbops

It might be related to certificates expiration. Perhaps you could check if #3863 (comment) works for you. Thanks.

@Vicente-Cheng Vicente-Cheng mentioned this issue May 22, 2024
Open
@W0n9 W0n9 changed the title [BUG] Upgrade v1.2.1->v1.2.2 stuck at Pre-drained node due to incompatible kubeVersion [BUG] Upgrade v1.2.1->v1.2.2 stuck at Pre-drained node due to certificates expiration May 22, 2024
@W0n9
Copy link
Author

W0n9 commented May 22, 2024

Hi @Vicente-Cheng ,
I have tried #3863 (comment) in my 2 long-running harvester cluster, and it works.
Thanks

@bk201 bk201 added require/doc Improvements or additions to documentation and removed reproduce/needed Reminder to add a reproduce label and to remove this one labels May 22, 2024
@bk201
Copy link
Member

bk201 commented May 22, 2024

cert rotate is implemented in v1.3.0, @starbops can you help add this (https://docs.harvesterhci.io/v1.3/upgrade/v1-1-2-to-v1-2-0#7-upgrade-stuck-in-the-upgrading-system-service-state) and seeder one to the known issue sections of 1.2.2 upgrade page? Thanks.

@starbops starbops mentioned this issue Jun 12, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@starbops starbops

Labels
kind/bug Issues that are defects reported by users or that we know have reached a real release require/doc Improvements or additions to documentation severity/needed Reminder to add a severity label and to remove this one
Projects
Community Issue Review
New
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Vicente-Cheng @bk201 @starbops @irishgordo @W0n9