diff --git a/docs/lock-options.md b/docs/lock-options.md deleted file mode 100644 index 0e40e8d5e..000000000 --- a/docs/lock-options.md +++ /dev/null @@ -1,63 +0,0 @@ -# lock-ddl-per-table option improvements - -To block DDL statements on an instance, Percona Server implemented LOCK -TABLES FOR BACKUP. Percona XtraBackup uses this lock for the -duration of the backup. This lock does not affect DML statements. - -Percona XtraBackup has also implemented `--lock-ddl-per-table`, which -blocks DDL statements by using metadata locks (MDL). - -The following procedures describe a simplified backup operation when using -`--lock-ddl-per-table`: - -1. Parse and copy all redo logs after the checkpoint mark - -2. Fork a dedicated thread to continue following new redo log entries - -3. List the tablespaces required to copy - -4. Iterate through the list. The following steps occur with each listed tablespace: - - * Query INFORMATION_SCHEMA.INNODB_TABLES to find which tables belong - to the tablespace ID and take an MDL on the underlying table or tables in - case there is a shared tablespace. - - * Copy the tablespace `.ibd` files. - -The backup process may encounter a redo log event, generated by the bulk -load -operations, which notifies backup tools that data file changes have been -omitted from the redo log. This event is an `MLOG_INDEX_LOAD`. If this -event is found by the redo follow thread, the backup continues and assumes -the backup is safe because the MDL protects tablespaces already copied and -the MLOG_INDEX_LOAD event is for a tablespace that is not copied. - -These assumptions may not be correct and may lead to inconsistent backups. - -## `--lock-ddl-per-table` redesign - -The `--lock-ddl-per-table` option has been redesigned to minimize inconsistent backups. - -The following procedure reorders the steps: - -* The MDL lock acquired at the beginning of the backup - -* Scan the redo logs. An `MLOG_INDEX_LOAD` event may be recorded if a `CREATE INDEX` statement has occurred before the backup starts. At this time, the backup process is safe and can parse and accept the event. - -* After the first scan has completed, the follow redo log thread is initiated. This thread stops the backup process if an `MLOG_INDEX_LOAD` event is found. - -* Gather the tablespace list to copy - -* Copy the `.ibd` files. - -## Other improvements - -The following improvements have been added: - -* If the `.ibd` file belongs to a temporary table, the `SELECT` query is skipped. - -* For a FullText Index, an MDL is acquired on the base table. - -* A `SELECT` query that acquires an MDL does not retrieve any data. - - diff --git a/docs/reduction-in-locks.md b/docs/reduction-in-locks.md new file mode 100644 index 000000000..328a16467 --- /dev/null +++ b/docs/reduction-in-locks.md @@ -0,0 +1,65 @@ +# Reduced backup lock time + +!!! important + + The `--lock-ddl=REDUCED` option is a [tech preview](./glossary.md#tech-preview). Before using this option in production, we recommend that you test restoring production from physical backups in your environment, and also use the alternative backup method for redundancy. + +Percona Server for MySQL implemented `LOCK TABLES FOR BACKUP` to block Data Definition Language (DDL) statements on a server. Percona XtraBackup utilizes this lock during the backup process to ensure that any `DDL` event does not corrupt the backup. The lock does not affect the Data Manipulation Language (DML) statements. + +## How the backup lock works + +With the [`--lock-ddl=ON`](./xtrabackup-option-reference.md/#lock-ddl) option the backup lock is enabled by default from the start of the backup process. + +* Set a backup lock: Set a backup lock on the server to prevent any `DDL` operations, ensuring no new tables are created, renamed, or dropped during the backup. + +* Copy the redo logs: Transfer all redo logs from the checkpoint up to the current Log Sequence Number (LSN) and start tracking new entries. + +* Identify and copy all InnoDB tablespace, `.ibd`, files. This step takes most of the time and is performed under the lock. + +* Copy non-InnoDB files. + +* Gather sync point: Collect synchronization points from all engines (InnoDB LSN, binlog, Global Transaction Identifier (GTID), etc.) by querying the log status. + +* Stop the redo follow thread: Cease the redo fllow thread once it has copied up to the sync point. + +* Release the backup lock on the server. + +If the backup lock is disabled with the [`--lock-ddl=OFF`](./xtrabackup-option-reference.md/#lock-ddl) option, a backup continues while concurrent DDL events are executed. Such backups are invalid and fail at `--prepare` step. + +## The `--lock-ddl=REDUCED` option overview + +[Percona XtraBackup 8.4.0-2](./release-notes/8.4.0-2.md) adds the [`--lock-ddl=REDUCED`](./xtrabackup-option-reference.md/#lock-ddl) option to reduce the time the instance remains under the backup lock. The backup lock is now taken after copying the `.ibd` files and before copying the `non-InnoDB` files. + +Copying the InnoDB files with the `--lock-ddl=REDUCED` option is split into two stages. + +`Operations performed without the lock` + +1. Copying all redo logs from checkpoint up to the current LSN and start following new entries. +2. Start the redo log thread. +3. Tracking file operations from the redo log. +4. Copying of all .ibd without taking any lock. +5. Acquiring Low-traffic Flush Barrier (LTFB)/Low-traffic Insert Barrier (LIFB). This step ensures no new DDL operations, such as creating or altering tables, will occur. +6. Quering the log_status to discover the LSN. This step ensures that all subsequent operations are tracked and that the backup process captures a consistent state of the database. +7. Copying all InnoDB files. +8. Ensuring the redo log has catch up to LSN from step 5. + +`Operations performed under the lock` + +9. Checking the file operations that were tracked and recopying the tablespaces. +10. Creating additional `meta` files to perform the required actions (deletions or renames) on the already copied files. This approach ensures that the backup remains consistent and accurate without disrupting the streaming process. This step is required for taking streaming backups. +11. Gathering a sync point from all engines (InnoDB LSN, binlog, GTID, etc.) by querying the `log_status`. +12. Stoping the redo follow thread once it copies at least up to sync point at step 11. +13. Releasing LTFB/LIFB. +14. Processing the new files during `--prepare` phase before crash recovery starts. + +The `--lock-ddl=REDUCED` option key features are as follows: + +* Reduced locking: Instead of holding a full `LOCK TABLES FOR BACKUP` for the entire duration of the backup, it only locks the tables for a shorter period after copying the InnoDB files. + +* DDL statements: It allows some `DDL` statements to proceed, reducing the downtime and impact on the database operations. + +* Consistency: Ensures that the backup remains consistent by carefully timing the locks to avoid conflicts with ongoing DDL operations. + + + + diff --git a/docs/xtrabackup-option-reference.md b/docs/xtrabackup-option-reference.md index afd45e55c..732a019d7 100644 --- a/docs/xtrabackup-option-reference.md +++ b/docs/xtrabackup-option-reference.md @@ -27,7 +27,6 @@ $ xtrabackup --prepare --target-dir=/data/backup/mysql/ For all modes, the default options are read from the xtrabackup and mysqld configuration groups from the following files in the given order: - 1. `/etc/my.cnf` @@ -598,7 +597,14 @@ Usage: `--lock-ddl` Enabled by default to ensure that any DDL event does not corrupt the backup. Any DML events continue to occur. A DDL lock protects table and view definitions. -If the option is disabled, a backup continues while concurrent DDL events are executed. These backups are invalid and fail in the Prepare step. +The available values are the following: + +|Option| Description| +|`--lock-ddl=ON`| The backup lock is enabled and is taken at the beginning of the backup.| +|`--lock-ddl=OFF`|The backup lock is not taken.| +|`--lock-ddl=REDUCED`|This option value is a [tech preview](./glossary.md#tech-preview). The option value has been added in [Percona XtraBackup 8.4.0-2](./release-notes/8.4.0-2.md) to reduce the time the instance remains under backup lock. The backup lock is taken after copying the `.ibd` files and before copying the `non-InnoDB` files.| + +If the option is disabled, a backup continues while concurrent DDL events are executed. These backups are invalid and fail at `--prepare` step. Use a [safe-slave-backup](#safe-slave-backup) option to stop a SQL replica thread before copying the InnoDB files. @@ -606,10 +612,9 @@ Use a [safe-slave-backup](#safe-slave-backup) option to stop a SQL replica threa Usage: `--lock-ddl-per-table` -Deprecated. Use the [`–lock-ddl`](#lock-ddl) option instead +Deprecated in Percona XtraBackup 8.0. Use the [`–lock-ddl`](#lock-ddl) option instead -Lock DDL for each table before xtrabackup starts to copy -it and until the backup is completed. +Lock DDL for each table before xtrabackup starts to copy it and until the backup is completed. ### lock-ddl-timeout diff --git a/mkdocs-base.yml b/mkdocs-base.yml index c7af762cf..f730e11e0 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -159,6 +159,7 @@ nav: - lru-dump-backup.md - throttling-backups.md - store-backup-history.md + - reduction-in-locks.md - Prepare features: - dictionary-cache.md - Restore features: @@ -166,7 +167,6 @@ nav: - restore-individual-tables.md - flush-tables-with-read-lock.md - log-enhancements.md - - lock-options.md - smart-memory-estimation.md - working-with-binary-logs.md - Binaries: