Skip to content
/ zkr Public

ZooKeeper Reaper - log view and restore utility

License

Notifications You must be signed in to change notification settings

Nordstrom/zkr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
Jul 1, 2022
Jul 1, 2022
Jul 8, 2022
Dec 20, 2019
Jul 1, 2022
Jul 1, 2022
Dec 20, 2019
Jul 8, 2022
Jul 1, 2022
Jul 1, 2022
Jul 1, 2022
Jul 1, 2022
Jul 1, 2022
Jul 1, 2022
Jul 1, 2022

Repository files navigation

zkr - ZooKeeper Reaper

Utility to backup/restore ZooKeeper znodes or view and restore ZooKeeper transactions from transaction log/Exhibitor backup files.

Background

This project provides a command-line utility that can be used to backup and restore ZooKeeper znodes into JSON, optionally compressed, and write to a file or S3.

The logs command can replay transactions from Exhibitor transaction log and backup archive (gzip'd) logs. The logs command ignores ephemeral znodes.

Usage

Commands

ZooKeeper Reaper v0.5 - ZooKeeper backup/restore utility
Usage: zkr [-hV] [COMMAND]
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  help     Displays help information about the specified command
  backup   Backup ZooKeeper znodes
  logs     View/write ZooKeeper/Exhibitor transaction logs and backups
  restore  Restore ZooKeeper znodes from backup

Backup command

Usage: zkr backup [-cdflv] [--pretty] [-m=<maxRetries>] [-p=<path>] [-r=<repeatMin>] [-s=<sessionTimeoutMs>]
                  [--s3-bucket=<s3bucket>] [--s3-region=<s3region>] [--superdigest-password=<superDigestPassword>]
                  [-z=<host>] [-e=<excludes>[,<excludes>...]]... [-i=<includes>[,<includes>...]]... <file> [COMMAND]
Backup ZooKeeper znodes
      <file>               Transaction log or backup file
  -c, --compress           Compress output (default: false)
  -d, --dry-run            Do not actually perform the actions (default: false)
  -e, --exclude=<excludes>[,<excludes>...]
                           Comma-delimited list of paths to exclude (regex)
  -f, --ephemeral          Backup ephemeral znodes (default: false)
  -i, --include=<includes>[,<includes>...]
                           Comma-delimited list of paths to include (regex)
  -l, --not-leader         Perform backup/restore even if ZooKeeper is not ensemble leader or standalone (i.e.,
                             follower) (default: false)
  -m, --max-retries=<maxRetries>
                           Maximum number of retries to read consistent data (default: 5)
  -p, --path=<path>        ZooKeeper root path for backup/restore (default: /)
      --pretty             Pretty print JSON output (default: false)
  -r, --repeat-every=<repeatMin>
                           Perform periodic backup every <repeatMin> minutes
  -s, --session-timeout-ms=<sessionTimeoutMs>
                           ZooKeeper session timeout in milliseconds (default: 30000)
      --s3-bucket=<s3bucket>
                           S3 bucket containing Exhibitor transaction logs/backups or zkr backup files
      --s3-region=<s3region>
                           AWS Region (default: us-west-2)
      --superdigest-password=<superDigestPassword>
                           ZooKeeper superdigest password. ZKR_SUPERDIGEST_PASSWORD environment variable if found.
  -v, --verbose            Verbose (DEBUG) logging level (default: false)
  -z, --zookeeper=<host>   Target ZooKeeper host:port (default: localhost:2181)
Commands:
  help  Displays help information about the specified command

By default, backup will only execute if --zookeeper is the leader of an ensemble or standalone. To backup a follower specify --not-leader.

Restore command

Usage: zkr restore [-cdlov] [--info] [-p=<path>] [-s=<sessionTimeoutMs>] [--s3-bucket=<s3bucket>]
                   [--s3-region=<s3region>] [--superdigest-password=<superDigestPassword>] [-z=<host>] [-e=<excludes>[,
                   <excludes>...]]... [-i=<includes>[,<includes>...]]... <file> [COMMAND]
Restore ZooKeeper znodes from backup
      <file>                 Transaction log or backup file
  -c, --compress             Compressed input (default: false)
  -d, --dry-run              Do not actually perform the actions (default: false)
  -e, --exclude=<excludes>[,<excludes>...]
                             Comma-delimited list of paths to exclude (regex)
  -i, --include=<includes>[,<includes>...]
                             Comma-delimited list of paths to include (regex)
      --info                 Print information about transaction log or backup then exit  (default: false)
  -l, --not-leader           Perform backup/restore even if ZooKeeper is not ensemble leader or standalone (i.e.,
                               follower) (default: false)
  -o, --overwrite-existing   Overwrite existing znodes (default: false)
  -p, --path=<path>          ZooKeeper root path for backup/restore (default: /)
  -s, --session-timeout-ms=<sessionTimeoutMs>
                             ZooKeeper session timeout in milliseconds (default: 30000)
      --s3-bucket=<s3bucket> S3 bucket containing Exhibitor transaction logs/backups or zkr backup files
      --s3-region=<s3region> AWS Region (default: us-west-2)
      --superdigest-password=<superDigestPassword>
                             ZooKeeper superdigest password. ZKR_SUPERDIGEST_PASSWORD environment variable if found.
  -v, --verbose              Verbose (DEBUG) logging level (default: false)
  -z, --zookeeper=<host>     Target ZooKeeper host:port (default: localhost:2181)
Commands:
  help  Displays help information about the specified command

By default, restore will only execute if --zookeeper is the leader of an ensemble or 'standalone'. To restore to a follower specify --not-leader. Use --dry-run to see what would be restored without actually writing the znodes.

Logs command

Usage: zkr logs [-lorv] [--info] [-p=<path>] [-s=<sessionTimeoutMs>] [--s3-bucket=<s3bucket>] [--s3-region=<s3region>]
                [--superdigest-password=<superDigestPassword>] [-z=<host>] [-e=<excludes>[,<excludes>...]]...
                [-i=<includes>[,<includes>...]]... <file> [COMMAND]
View/write ZooKeeper/Exhibitor transaction logs and backups
      <file>                 Transaction log or backup file
  -e, --exclude=<excludes>[,<excludes>...]
                             Comma-delimited list of paths to exclude (regex)
  -i, --include=<includes>[,<includes>...]
                             Comma-delimited list of paths to include (regex)
      --info                 Print information about transaction log or backup then exit  (default: false)
  -l, --not-leader           Perform backup/restore even if ZooKeeper is not ensemble leader or standalone (i.e.,
                               follower) (default: false)
  -o, --overwrite-existing   Overwrite existing znodes (default: false)
  -p, --path=<path>          ZooKeeper root path for backup/restore (default: /)
  -r, --restore              Execute (restore) transactions (default: false)
  -s, --session-timeout-ms=<sessionTimeoutMs>
                             ZooKeeper session timeout in milliseconds (default: 30000)
      --s3-bucket=<s3bucket> S3 bucket containing Exhibitor transaction logs/backups or zkr backup files
      --s3-region=<s3region> AWS Region (default: us-west-2)
      --superdigest-password=<superDigestPassword>
                             ZooKeeper superdigest password. ZKR_SUPERDIGEST_PASSWORD environment variable if found.
  -v, --verbose              Verbose (DEBUG) logging level (default: false)
  -z, --zookeeper=<host>     Target ZooKeeper host:port (default: localhost:2181)
Commands:
  help  Displays help information about the specified command

By default, the logs command will only display the log transactions. Use --restore to actually write or overwrite znodes.

Build

This project was developed with gradle 6.7.1, java 11 and kotlin 1.4.20

./gradlew clean assemble

This will create a shaded jar named build/libs/zkr-all.jar

Usage

To see the available options, run:

java -jar build/libs/zkr-all.jar --help

The only required options is -z/--zookeeper which is a standard ZooKeeper connection string (e.g., localhost:2181)

Backup/restore Kafka

Backup

This tool can backup topics and acls in a Kafka cluster from either the Exhibitor transaction log/backup files. It is very important to NOT backup broker ephemeral nodes:

  • /kafka/controller
  • /kafka/brokers/ids/

This is done by default as --ephemeral's default value false

Restore

Restoring can be done using Exhibitor/ZooKeeper transaction logs or backup files or a zkr backup file but must be done in a specific order:

  • start Exhibitor/ZooKeeper (ONLY) with backups disabled (so you don't accidentally overwrite your backup files)
  • Run zkr with --overwrite-existing option
  • start brokers, et al

Quick demo

Run a quick demo to recover Kafka metadata stored in ZooKeeper

Create a Kafka cluster and topic

  • Run docker-compose up
  • Create a topic by running: echo "hello" | kafkacat -b localhost:9092 -P -t hello

Take a backup, then tear it all down

  • Take a backup: ./zkr.sh backup --zookeeper localhost:2181 -l backup-file
  • Tear it all down: docker-compose down

Restart cluster using the backup

  • Run docker-compose up --no-start
  • Run docker-compose start zookeeper
  • Restore ZooKeeper by running: ./zkr.sh restore --zookeeper localhost:2181 -l backup.json
  • Start Kafka cluster: docker-compose start kafka-broker

Check the restore by listing Kafka topics: kafkacat -b localhost:9092 -L

ZooKeeper security

via jaas.conf

Create a jaas.conf file with appropriate values. For example:

Server {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  user_admin="developer"
  user_developer="developer";
};
Client {
  org.apache.zookeeper.server.auth.DigestLoginModule required
  username="admin"
  password="developer";
};

Invoke zkr thusly:

java -Djava.security.auth.login.config=./jaas.conf -cp build/libs/zkr-all.jar zkr.Zkr <command> <options>

Or use the zkr script in the root directory.

superdigest support

It may be necessary to add superdigest authorization to access restricted znodes (e.g., /kafka/config/users)

This is supported by specifying the superdigest password in an environment variable ZKR_SUPERDIGEST_PASSWORD or using the --superdigest-password parameter.

ATTRIBUTIONS

This utility borrowed heavily from:

  • ZooKeeper transaction and Exhibitor backup view/restore is based on ZooKeeper LogFormatter
  • Backup/Restore is based on zoocreeper