PXB-3254 Document reduction in locks

	modified:   docs/lock-options.md
	new file:   docs/reduction-in-locks.md
	modified:   docs/xtrabackup-option-reference.md
	modified:   mkdocs-base.yml

docs/reduction-in-locks.md

@@ -0,0 +1,107 @@
+# Reduced backup lock time
+
+--8<--- "pro-build-announcement.md"
+
+!!! important
+
+    The `--lock-ddl=REDUCED` option is a [tech preview](./glossary.md#tech-preview). Before using this option in production, we recommend that you test restoring production from physical backups in your environment, and also use the alternative backup method for redundancy.
+
+[Percona XtraBackup 8.4.0-2](./release-notes/8.4.0-2.md) adds the [`--lock-ddl=REDUCED`](#how-the---lock-ddlreduced-option-works) option to reduce the time the server remains locked by `xtrabackup` during full and incremental backups. Now, you can execute `Data Definition Language` (DDL) operations while the backup is in progress.
+
+### Benefits
+
+The `--lock-ddl=REDUCED` option features are as follows:
+
+* Rather than holding the backup lock throughout the entire backup process, the lock is only acquired for a very short period.
+
+* DDL operations are allowed on the server while the backup is in progress.
+
+* The redo log tracks all file operations to maintain consistency. The backup process generates new metadata files to handle any DDL operations that occur during the backup. When you run `--prepare`, the `xtrabackup` first processes these metadata files to ensure a consistent database state with respect to file-level operations. Then the redo and undo logs are used to make the backup fully consistent.
+
+### Limitations
+
+* Certain DDL operations consume significant resources, and xtrabackup simultaneously requires I/O to copy and read files. This requirement can increase resource demand from DDL operations during the backup process.
+
+* Backups may be larger because new `DDL` operations are executed concurrently on the server, and the files generated by the server are included in the backup. Additionally, certain `DDL` operations, such as `ADD INDEX` or encryption changes to existing data files, will cause the data files to be recopied, increasing the backup size.
+
+* The number of open file handles in your operating system should be configured to match the number of files in the server data directory.
+
+## How the backup lock with the `--lock-ddl` option works
+
+Percona Server for MySQL implemented `LOCK TABLES FOR BACKUP` to block `DDL` operations on a server. Percona `xtrabackup` utilizes this lock during the backup process to ensure that any `DDL` operation does not corrupt the backup. With MySQL-compatible servers `xtrabackup` utilizes `Lock Instance For Backup` to lock the server during the backup. The lock does not affect the `Data Manipulation Language` (DML) operations.
+
+With the [`--lock-ddl=ON`](./xtrabackup-option-reference.md/#lock-ddl) option the backup lock is enabled by default from the start of the backup process.
+{ .power-number }
+
+1. Set a backup lock on the server to prevent any DDL operations, such as creating or altering tables.
+
+2. Parse and copy the redo logs from the checkpoint up to the current `Log Sequence Number` (LSN), and start tracking new file operations.
+
+3. Identify and copy the `.ibd` files.
+
+4. Copy `non-InnoDB` files.
+
+5. Gather a sync point from all engines (InnoDB LSN, binlog, Global Transaction Identifier (GTID), etc.) by querying the log status.
+
+6. Stop the redo thread once it copies at least up to sync point.
+
+7. Release the backup lock on the server.
+
+If the backup lock is disabled with the [`--lock-ddl=OFF`](./xtrabackup-option-reference.md/#lock-ddl) option, a backup continues while concurrent DDL operations are executed. These backups may be invalid and may fail at either the `backup` or the `--prepare` step.
+
+## How the `--lock-ddl=REDUCED` option works
+
+Taking backup with the `--lock-ddl=REDUCED` option includes the following phases:
+
+### Phase 1: Operations performed without the lock
+
+!!! warning
+
+    The `ALTER INSTANCE ROTATE MASTER KEY` command should not be executed during Phase 1 if there are encrypted tables, as it will cause the backup to fail.
+
+The following operations are performed without the lock:
+{ .power-number }
+
+1. Parse and copy the redo logs from the checkpoint up to the current `LSN`, and start tracking new file operations.
+2. Start the redo log thread to copy the redo logs. 
+3. Track file operations by parsing the `MLOG_FILE_*` records from the redo log. These records help track changes in the files being backed up to ensure consistency.
+4. Copy the `.ibd` files.
+
+### Phase 2: Operations performed under the lock
+
+The following operations are performed under the lock:
+{ .power-number }
+
+1. Take the backup lock on the server to prevent new DDL operations, such as creating or altering tables.
+2. Copy `non-InnoDB` files.
+3. Check the file operations that were tracked and recopy the tablespaces.
+4. Create additional `metadata` files to perform the required actions (deletions or renames) on the already copied files. This approach ensures that the backup remains consistent and accurate without disrupting the streaming process. This is particularly important for streaming backups.
+5. Gather a sync point from all engines (InnoDB LSN, binlog, GTID, etc.) by querying the `log_status`.
+6. Stop the redo thread once it copies at least up to sync point at step 4.
+7. Release the backup lock on the server.
+
+??? note "The list of metadata files"
+
+    | File         | Description                                                                                                      |
+    |--------------|------------------------------------------------------------------------------------------------------------------|
+    | `.new`       | For new files that are created while the backup in progress and for files that need to be recopied due to encryption or `ADD INDEX` operations.                                  |
+    | `.del`       | For deleted files. For a file that has been copied by backup but deleted while the backup is in progress, create a `space_id.del` file.                                      |
+    | `.ren`       | For a file that has been copied but renamed while the backup is in progress, create a space_id.ren file. This file should contain the new file name.                                                  |
+    | `.crpt`      | For files that cannot be fully copied due to encryption changes. These files will be recopied, and a `.crpt` file will be created for the existing file. |
+    | `.new.meta`  | Similar to `.new`, but for incremental backups.                                                                  |
+    | `.new.delta` | Similar to `.new`, but for incremental backups. Create `t1.ibd.meta` and `t1.ibd.delta` instead of `t1.ibd`.      |
+
+### Phase 3. Handling new metadata files in `--prepare`
+
+The following operations are performed to ensure data remains intact and consistent:
+{ .power-number }
+
+1. Remove the `.crpt` files that match the name after stripping the extension. This step is crucial before the `IBD` scan because these files are incomplete (they can be zero size).
+2. Perform a scan to create a mapping of `space_id` to file names.
+3. Delete the file matching the `space_id` using `space_id.del`. In case of incremental backups, also delete the corresponding `.new.meta` and `.new.delta` files.
+4. Rename the file matching the `space_id` to the name specified in the file using `space_id.ren`.
+5. Replace the file that matches the name without the `.new` extension using the `.new` extension.
+6. Replace the `meta` and `delta` files that match the name without the `.new` in the name using `.new.meta` and `.new.delta`.
+
+After `Phase 3` is completed, the regular crash recovery starts.
+

docs/xtrabackup-option-reference.md

@@ -27,7 +27,6 @@ $ xtrabackup --prepare --target-dir=/data/backup/mysql/
 For all modes, the default options are read from the xtrabackup and
 mysqld configuration groups from the following files in the given order:
 
-
 1. `/etc/my.cnf`
 
 
@@ -598,18 +597,25 @@ Usage: `--lock-ddl`
 
 Enabled by default to ensure that any DDL event does not corrupt the backup. Any DML events continue to occur. A DDL lock protects table and view definitions.
 
-If the option is disabled, a backup continues while concurrent DDL events are executed. These backups are invalid and fail in the Prepare step.
+The available values are the following:
+
+|Option| Description|
+|-------|-----------|
+|`--lock-ddl=ON`| The backup lock is enabled and is taken at the beginning of the backup.|
+|`--lock-ddl=OFF`|The backup lock is not taken.|
+|`--lock-ddl=REDUCED`|This option value is a [tech preview](./glossary.md#tech-preview). <br><br> The option value has been added in [Percona XtraBackup 8.4.0-2](./release-notes/8.4.0-2.md) to reduce the time the instance remains under backup lock. The backup lock is taken after copying the `.ibd` files and before copying the `non-InnoDB` files.|
+
+If the backup lock is disabled with the `--lock-ddl=OFF` option, a backup continues while concurrent DDL events are executed. These backups may be invalid and may fail at either the `backup` or the `--prepare` step.
 
 Use a [safe-slave-backup](#safe-slave-backup) option to stop a SQL replica thread before copying the InnoDB files.
 
 ### lock-ddl-per-table
 
 Usage: `--lock-ddl-per-table`
 
-Deprecated. Use the [`–lock-ddl`](#lock-ddl) option instead
+Deprecated in Percona XtraBackup 8.0. Use the [`–lock-ddl`](#lock-ddl) option instead
 <!--26-1-24-->
-Lock DDL for each table before xtrabackup starts to copy
-it and until the backup is completed.
+Lock DDL for each table before xtrabackup starts to copy it and until the backup is completed.
 
 ### lock-ddl-timeout
 

mkdocs-base.yml

@@ -164,9 +164,10 @@ nav:
       - Restore features:
         - point-in-time-recovery.md
         - restore-individual-tables.md
+      - Pro build features:
+        - reduction-in-locks.md
       - flush-tables-with-read-lock.md
       - log-enhancements.md
-      - lock-options.md
       - smart-memory-estimation.md
       - working-with-binary-logs.md
   - Binaries:

snippets/pro-build-announcement.md

@@ -0,0 +1,3 @@
+Percona XtraBackup Pro includes the [capabilities](pxb-pro.md#capabilities) that are typically requested by large enterprises. Percona XtraBackup Pro contains packages created and tested by Percona. These packages are supported only for Percona Customers with a subscription.
+
+[Become a Percona Customer](https://www.percona.com/about/contact){.md-button}