diff --git a/docs/lock-options.md b/docs/lock-options.md deleted file mode 100644 index 0e40e8d5e..000000000 --- a/docs/lock-options.md +++ /dev/null @@ -1,63 +0,0 @@ -# lock-ddl-per-table option improvements - -To block DDL statements on an instance, Percona Server implemented LOCK -TABLES FOR BACKUP. Percona XtraBackup uses this lock for the -duration of the backup. This lock does not affect DML statements. - -Percona XtraBackup has also implemented `--lock-ddl-per-table`, which -blocks DDL statements by using metadata locks (MDL). - -The following procedures describe a simplified backup operation when using -`--lock-ddl-per-table`: - -1. Parse and copy all redo logs after the checkpoint mark - -2. Fork a dedicated thread to continue following new redo log entries - -3. List the tablespaces required to copy - -4. Iterate through the list. The following steps occur with each listed tablespace: - - * Query INFORMATION_SCHEMA.INNODB_TABLES to find which tables belong - to the tablespace ID and take an MDL on the underlying table or tables in - case there is a shared tablespace. - - * Copy the tablespace `.ibd` files. - -The backup process may encounter a redo log event, generated by the bulk -load -operations, which notifies backup tools that data file changes have been -omitted from the redo log. This event is an `MLOG_INDEX_LOAD`. If this -event is found by the redo follow thread, the backup continues and assumes -the backup is safe because the MDL protects tablespaces already copied and -the MLOG_INDEX_LOAD event is for a tablespace that is not copied. - -These assumptions may not be correct and may lead to inconsistent backups. - -## `--lock-ddl-per-table` redesign - -The `--lock-ddl-per-table` option has been redesigned to minimize inconsistent backups. - -The following procedure reorders the steps: - -* The MDL lock acquired at the beginning of the backup - -* Scan the redo logs. An `MLOG_INDEX_LOAD` event may be recorded if a `CREATE INDEX` statement has occurred before the backup starts. At this time, the backup process is safe and can parse and accept the event. - -* After the first scan has completed, the follow redo log thread is initiated. This thread stops the backup process if an `MLOG_INDEX_LOAD` event is found. - -* Gather the tablespace list to copy - -* Copy the `.ibd` files. - -## Other improvements - -The following improvements have been added: - -* If the `.ibd` file belongs to a temporary table, the `SELECT` query is skipped. - -* For a FullText Index, an MDL is acquired on the base table. - -* A `SELECT` query that acquires an MDL does not retrieve any data. - - diff --git a/docs/reduction-in-locks.md b/docs/reduction-in-locks.md new file mode 100644 index 000000000..6fa3ce349 --- /dev/null +++ b/docs/reduction-in-locks.md @@ -0,0 +1,107 @@ +# Reduced backup lock time + +--8<--- "pro-build-announcement.md" + +!!! important + + The `--lock-ddl=REDUCED` option is a [tech preview](./glossary.md#tech-preview). Before using this option in production, we recommend that you test restoring production from physical backups in your environment, and also use the alternative backup method for redundancy. + +[Percona XtraBackup 8.4.0-2](./release-notes/8.4.0-2.md) adds the [`--lock-ddl=REDUCED`](#how-the---lock-ddlreduced-option-works) option to reduce the time the server remains locked by `xtrabackup` during full and incremental backups. Now, you can execute `Data Definition Language` (DDL) operations while the backup is in progress. + +### Benefits + +The `--lock-ddl=REDUCED` option features are as follows: + +* Rather than holding the backup lock throughout the entire backup process, the lock is only acquired for a very short period. + +* DDL operations are allowed on the server while the backup is in progress. + +* The redo log tracks all file operations to maintain consistency. The backup process generates new metadata files to handle any DDL operations that occur during the backup. When you run `--prepare`, the `xtrabackup` first processes these metadata files to ensure a consistent database state with respect to file-level operations. Then the redo and undo logs are used to make the backup fully consistent. + +### Limitations + +* Certain DDL operations consume significant resources, and xtrabackup simultaneously requires I/O to copy and read files. This requirement can increase resource demand from DDL operations during the backup process. + +* Backups may be larger because new `DDL` operations are executed concurrently on the server, and the files generated by the server are included in the backup. Additionally, certain `DDL` operations, such as `ADD INDEX` or encryption changes to existing data files, will cause the data files to be recopied, increasing the backup size. + +* The number of open file handles in your operating system should be configured to match the number of files in the server data directory. + +## How the backup lock with the `--lock-ddl` option works + +Percona Server for MySQL implemented `LOCK TABLES FOR BACKUP` to block `DDL` operations on a server. Percona `xtrabackup` utilizes this lock during the backup process to ensure that any `DDL` operation does not corrupt the backup. With MySQL-compatible servers `xtrabackup` utilizes `Lock Instance For Backup` to lock the server during the backup. The lock does not affect the `Data Manipulation Language` (DML) operations. + +With the [`--lock-ddl=ON`](./xtrabackup-option-reference.md/#lock-ddl) option the backup lock is enabled by default from the start of the backup process. +{ .power-number } + +1. Set a backup lock on the server to prevent any DDL operations, such as creating or altering tables. + +2. Parse and copy the redo logs from the checkpoint up to the current `Log Sequence Number` (LSN), and start tracking new file operations. + +3. Identify and copy the `.ibd` files. + +4. Copy `non-InnoDB` files. + +5. Gather a sync point from all engines (InnoDB LSN, binlog, Global Transaction Identifier (GTID), etc.) by querying the log status. + +6. Stop the redo thread once it copies at least up to sync point. + +7. Release the backup lock on the server. + +If the backup lock is disabled with the [`--lock-ddl=OFF`](./xtrabackup-option-reference.md/#lock-ddl) option, a backup continues while concurrent DDL operations are executed. These backups may be invalid and may fail at either the `backup` or the `--prepare` step. + +## How the `--lock-ddl=REDUCED` option works + +Taking backup with the `--lock-ddl=REDUCED` option includes the following phases: + +### Phase 1: Operations performed without the lock + +!!! warning + + The `ALTER INSTANCE ROTATE MASTER KEY` command should not be executed during Phase 1 if there are encrypted tables, as it will cause the backup to fail. + +The following operations are performed without the lock: +{ .power-number } + +1. Parse and copy the redo logs from the checkpoint up to the current `LSN`, and start tracking new file operations. +2. Start the redo log thread to copy the redo logs. +3. Track file operations by parsing the `MLOG_FILE_*` records from the redo log. These records help track changes in the files being backed up to ensure consistency. +4. Copy the `.ibd` files. + +### Phase 2: Operations performed under the lock + +The following operations are performed under the lock: +{ .power-number } + +1. Take the backup lock on the server to prevent new DDL operations, such as creating or altering tables. +2. Copy `non-InnoDB` files. +3. Check the file operations that were tracked and recopy the tablespaces. +4. Create additional `metadata` files to perform the required actions (deletions or renames) on the already copied files. This approach ensures that the backup remains consistent and accurate without disrupting the streaming process. This is particularly important for streaming backups. +5. Gather a sync point from all engines (InnoDB LSN, binlog, GTID, etc.) by querying the `log_status`. +6. Stop the redo thread once it copies at least up to sync point at step 4. +7. Release the backup lock on the server. + +??? note "The list of metadata files" + + | File | Description | + |--------------|------------------------------------------------------------------------------------------------------------------| + | `.new` | For new files that are created while the backup in progress and for files that need to be recopied due to encryption or `ADD INDEX` operations. | + | `.del` | For deleted files. For a file that has been copied by backup but deleted while the backup is in progress, create a `space_id.del` file. | + | `.ren` | For a file that has been copied but renamed while the backup is in progress, create a space_id.ren file. This file should contain the new file name. | + | `.crpt` | For files that cannot be fully copied due to encryption changes. These files will be recopied, and a `.crpt` file will be created for the existing file. | + | `.new.meta` | Similar to `.new`, but for incremental backups. | + | `.new.delta` | Similar to `.new`, but for incremental backups. Create `t1.ibd.meta` and `t1.ibd.delta` instead of `t1.ibd`. | + +### Phase 3. Handling new metadata files in `--prepare` + +The following operations are performed to ensure data remains intact and consistent: +{ .power-number } + +1. Remove the `.crpt` files that match the name after stripping the extension. This step is crucial before the `IBD` scan because these files are incomplete (they can be zero size). +2. Perform a scan to create a mapping of `space_id` to file names. +3. Delete the file matching the `space_id` using `space_id.del`. In case of incremental backups, also delete the corresponding `.new.meta` and `.new.delta` files. +4. Rename the file matching the `space_id` to the name specified in the file using `space_id.ren`. +5. Replace the file that matches the name without the `.new` extension using the `.new` extension. +6. Replace the `meta` and `delta` files that match the name without the `.new` in the name using `.new.meta` and `.new.delta`. + +After `Phase 3` is completed, the regular crash recovery starts. + diff --git a/docs/xtrabackup-option-reference.md b/docs/xtrabackup-option-reference.md index afd45e55c..b1fef0e5e 100644 --- a/docs/xtrabackup-option-reference.md +++ b/docs/xtrabackup-option-reference.md @@ -27,7 +27,6 @@ $ xtrabackup --prepare --target-dir=/data/backup/mysql/ For all modes, the default options are read from the xtrabackup and mysqld configuration groups from the following files in the given order: - 1. `/etc/my.cnf` @@ -598,7 +597,15 @@ Usage: `--lock-ddl` Enabled by default to ensure that any DDL event does not corrupt the backup. Any DML events continue to occur. A DDL lock protects table and view definitions. -If the option is disabled, a backup continues while concurrent DDL events are executed. These backups are invalid and fail in the Prepare step. +The available values are the following: + +|Option| Description| +|-------|-----------| +|`--lock-ddl=ON`| The backup lock is enabled and is taken at the beginning of the backup.| +|`--lock-ddl=OFF`|The backup lock is not taken.| +|`--lock-ddl=REDUCED`|This option value is a [tech preview](./glossary.md#tech-preview).

The option value has been added in [Percona XtraBackup 8.4.0-2](./release-notes/8.4.0-2.md) to reduce the time the instance remains under backup lock. The backup lock is taken after copying the `.ibd` files and before copying the `non-InnoDB` files.| + +If the backup lock is disabled with the `--lock-ddl=OFF` option, a backup continues while concurrent DDL events are executed. These backups may be invalid and may fail at either the `backup` or the `--prepare` step. Use a [safe-slave-backup](#safe-slave-backup) option to stop a SQL replica thread before copying the InnoDB files. @@ -606,10 +613,9 @@ Use a [safe-slave-backup](#safe-slave-backup) option to stop a SQL replica threa Usage: `--lock-ddl-per-table` -Deprecated. Use the [`–lock-ddl`](#lock-ddl) option instead +Deprecated in Percona XtraBackup 8.0. Use the [`–lock-ddl`](#lock-ddl) option instead -Lock DDL for each table before xtrabackup starts to copy -it and until the backup is completed. +Lock DDL for each table before xtrabackup starts to copy it and until the backup is completed. ### lock-ddl-timeout diff --git a/mkdocs-base.yml b/mkdocs-base.yml index c7af762cf..a94f8d443 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -164,9 +164,10 @@ nav: - Restore features: - point-in-time-recovery.md - restore-individual-tables.md + - Pro build features: + - reduction-in-locks.md - flush-tables-with-read-lock.md - log-enhancements.md - - lock-options.md - smart-memory-estimation.md - working-with-binary-logs.md - Binaries: diff --git a/snippets/pro-build-announcement.md b/snippets/pro-build-announcement.md new file mode 100644 index 000000000..e9fb23e0d --- /dev/null +++ b/snippets/pro-build-announcement.md @@ -0,0 +1,3 @@ +Percona XtraBackup Pro includes the [capabilities](pxb-pro.md#capabilities) that are typically requested by large enterprises. Percona XtraBackup Pro contains packages created and tested by Percona. These packages are supported only for Percona Customers with a subscription. + +[Become a Percona Customer](https://www.percona.com/about/contact){.md-button} \ No newline at end of file