Skip to content

Commit

Permalink
Update Kubernetes self-hosted upgrade docs
Browse files Browse the repository at this point in the history
  • Loading branch information
mdlinville committed Nov 5, 2024
1 parent 1f61531 commit 70f6b55
Show file tree
Hide file tree
Showing 12 changed files with 574 additions and 614 deletions.
63 changes: 63 additions & 0 deletions src/current/_includes/common/upgrade/disable-auto-finalization.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,65 @@
{% if page.path contains 'kubernetes' %}

<section class="filter-content" markdown="1" data-scope="operator">
For clusters managed by the Operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade).
</section>

<section class="filter-content" markdown="1" data-scope="manual">

By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version).

To disable auto-finalization:

1. Connect to the cluster using the SQL shell:

~~~ shell
$ kubectl exec -it cockroachdb-client-secure \
-- ./cockroach sql \
--certs-dir=/cockroach-certs \
--host=cockroachdb-public
~~~

1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version.
Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade).
{{site.data.alerts.callout_info}}
Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade.

We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization.
{{site.data.alerts.end}}
</section>
<section class="filter-content" markdown="1" data-scope="helm">
By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version).
To disable auto-finalization:
1. Connect to the cluster using the SQL shell:
~~~ shell
$ kubectl exec -it cockroachdb-client-secure \
-- ./cockroach sql \
--certs-dir=/cockroach-certs \
--host=cockroachdb-public
~~~
1. Set the [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) `cluster.preserve_downgrade_option` to the cluster's current major version.

Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade) or [roll it back]({#roll-back-a-major-version-upgrade).

{{site.data.alerts.callout_info}}
Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade. This cluster setting does not persist after finalization is complete, but must be set before each major-version upgrade.
We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization.
{{site.data.alerts.end}}

</section>

{% else %}

By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must [restore the cluster to the previous version]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}#support-for-restoring-backups-into-a-newer-version).

To disable auto-finalization:
Expand All @@ -18,3 +80,4 @@ Previously, to disable automatic finalization and preserve the ability to roll b
We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2 and persists after finalization is complete. Either of these settings prevents automatic finalization.
{{site.data.alerts.end}}
{% endif %}
40 changes: 40 additions & 0 deletions src/current/_includes/common/upgrade/finalize-kubernetes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{% assign major_version_numeric = page.version.version | remove_first: 'v' %}

To finalize a major-version upgrade:

1. Connect to the cluster using the SQL shell:

~~~ shell
$ kubectl exec -it cockroachdb-client-secure \
-- ./cockroach sql \
--certs-dir=/cockroach-certs \
--host=cockroachdb-public
~~~

1. Run the following command:

{% include_cached copy-clipboard.html %}
~~~ sql
> RESET CLUSTER SETTING cluster.preserve_downgrade_option;
~~~

A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version`.
You can monitor the process of the migration in the DB Console [Jobs page]({% link {{ page.version.version }}/ui-jobs-page.md %}). Migration jobs have names in the format `{{ major_version_numeric }}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.
The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load.
{{site.data.alerts.callout_info}}
Finalization is not complete until all [schema change]({% link {{ page.version.version }}/online-schema-changes.md %}) jobs reach a terminal state. Finalization can take as long as the longest-running schema change.
{{site.data.alerts.end}}
When all migration jobs have completed, the upgrade is complete.
1. To confirm that finalization has completed, check the cluster version:
{% include_cached copy-clipboard.html %}
~~~ sql
> SHOW CLUSTER SETTING version;
~~~
If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of [decommissioning nodes]({% link {{ page.version.version }}/node-shutdown.md %}?filters=decommission#status-change) where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc/).
10 changes: 6 additions & 4 deletions src/current/_includes/common/upgrade/finalize-self-hosted.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
To finalize a major-version upgrade, run the following command. Replace `{VERSION}` new major version, such as `{{ page.version.version }}:
{% assign major_version_numeric = page.version.version | remove_first: 'v' %}

To finalize a major-version upgrade:

1. Connect to the cluster using the SQL shell:

Expand All @@ -7,14 +9,14 @@ To finalize a major-version upgrade, run the following command. Replace `{VERSIO
cockroach sql
~~~

1. Run the following command. Replace `{VERSION}` with the new major version, such as `{{ page.version.version }}`.
1. Run the following command. Replace `{VERSION}` with the new major version, such as `{{ major_version_numeric }}`.

{% include_cached copy-clipboard.html %}
~~~ shell
~~~ sql
SET CLUSTER SETTING version '{VERSION}';
~~~

A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return `{{ previous_version_numeric }}`.
A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version.

You can monitor the process of the migration in the DB Console [Jobs page]({% link {{ page.version.version }}/ui-jobs-page.md %}). Migration jobs have names in the format `{{ major_version_numeric }}-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.

Expand Down
Loading

0 comments on commit 70f6b55

Please sign in to comment.