Skip to content

Latest commit

 

History

History
479 lines (409 loc) · 20.4 KB

RELEASING.adoc

File metadata and controls

479 lines (409 loc) · 20.4 KB
Raw

RELEASING KUDU

Overview

This document gives the steps required to perform a Kudu release, and is a resource for Kudu release managers. Make sure to read and understand the ASF policy on software releases prior to conducting the release management activity. Pay attention to the 3rd-party license policy, making sure the release doesn’t include anything from the Category X. The command-line snippets in this document are for bash (Bourne-again shell). To edit or update this document, edit RELEASING.adoc in master.

Stating Intent to Release

Note
 All email messages must be sent from your apache.org email address, otherwise with company or personal email addresses sometimes emails are not delivered. If using gmail.com service for email correspondence, follow this guide to add your apache.org address to be used as From address when sending email messages.

  1. A week before branching: send an email to [email protected] to announce that the branch will be happening, including a deadline for when new commits will need gatekeeper approval. Start compiling the release notes.

  2. A day before branching: send another email to [email protected] to warn about it.

Compile the release notes

Start a Google doc for the release notes that is editable by anyone, then use https://s.apache.org to generate a short URL like kudu-1.y-rn and share it on [email protected] asking for help. The author of a commit probably knows best if a change belongs to the release notes, and how to succintly describe the change. Of course, there will be contributors who are less active and maybe not even monitoring the dev@ list, so you’ll still have to write release notes for their changes.

Use git shortlog <prev-version>..HEAD to get a list of contributions and add it to the doc as a summary of all commits. Note the commit hash at HEAD, as you’ll need to generate another list between this commit and the tip when finally branching.

Creating the Branch

  1. Create a new branch from master:

      git checkout master
  git pull
  git checkout -b branch-1.x.y

  2. Make a note of the SHA1 for the tip of the new branch, which is the first field of the result of this command:

      git log --oneline -n1

  3. Push the branch to public remote https://gitbox.apache.org/repos/asf/kudu.git. The following example assumes it’s called apache.

      git push apache branch-1.x.y

  4. Create a new branch on Gerrit. Go to http://gerrit.cloudera.org:8080/#/admin/projects/kudu,branches and create a new branch with the same name and the previously-noted SHA1.

  5. Ask someone with permissions to fix the gerrit.cloudera.org mirroring configuration. Cloudera hosts the Gerrit server and a Cloudera employee will have to perform this step because SSH access is behind a firewall. The steps are as follows:

    1. Ensure your public SSH key is in ~gerrit/.ssh/authorized_keys on gerrit.cloudera.org

    2. From behind the firewall, ssh [email protected] to log in.

    3. Back up the existing replication configuration file by executing cp ~/etc/replication.config ~/etc/replication.config.bak.`date '+%Y%m%d.%H%M%S'`

    4. Edit ~/etc/replication.config to add a line for the new branch, such as branch-1.x.y

    5. Send email to the dev lists for Kudu and Impala ([email protected] and [email protected]) indicating that you are going to restart Gerrit (example). It is best to do the restart at some time of day when you don’t expect many people to be using the system, since Gerrit can take a few minutes to restart.

    6. Restart Gerrit: ~/bin/gerrit.sh restart

    7. Make sure Gerrit has been successfully restarted: after a few minutes, try to open the project list page in your favorite browser.

  6. As needed, patches can be cherry-picked to the new branch.

Write the release notes

  1. Copy the release notes from the shared doc to docs/release_notes.adoc.

  2. Edit and format the release notes.

  3. Get the number of contributors and the list of new contributors using the following command, and add them to the contributors section (see previous release notes for examples):

      export PREV="1.15.0" # the previous release
  git shortlog -sn $PREV.. | wc -l
  diff <(git shortlog -sn ${PREV} | cut -f 2 | sort) \
    <(git shortlog -sn ${PREV}.. | cut -f 2 | sort) \
    | grep -E "^>" | sed 's/>/*/'

  4. Commit the release notes to the newly created branch and submit it to Gerrit. Use git shortlog -sn $PREV.. to list the contributors and add them to the reviewers to give everyone a chance to comment on the release notes of their changes.

Updating Versions in Master

  1. Check out the master branch and bump the version in version.txt. Don’t update kudu-version in examples/java/java-example/pom.xml yet: it should be updated later on when release artifacts are published (see below).

  2. Bump the version numbers in docs/release_notes.adoc.

  3. Commit and push that change to Gerrit.

  4. Notify [email protected] that the new branch is available (see here for an example).

Preparing a Release Candidate

  1. Before building a release candidate, make sure you have followed the Apache committer guide for setting up your PGP keys. In addition to the MIT PGP public key server referenced in the committer guide, consider adding your keys to one of SKS OpenPGP keyservers or Ubuntu OpenPGP keyserver. The MIT keyserver sometimes is not available, and it takes some time to propagate key updates anyways: the latter two servers are used by the Apache Maven repo server to verify the signature of the uploaded maven artifacts as of October 2019.

  2. If building and signing on a remote/shared machine, consider forwarding GPG agent via SSH. That means you can keep your secret keys on a local machine even when signing the artifacts to be released (works even for a hardware token like a smartcard, etc.).

  3. Out of the Kudu git workspace, checkout the release SVN repository. Later on, the officially released and signed artifacts will be put into this repository. At this point, just add your PGP key to the KEYS file (if it’s not there yet), making it available for the signature verification:

      svn co https://dist.apache.org/repos/dist/release/kudu/ kudu-dist-release
  cd kudu-dist-release
  (gpg --list-sigs <your-email-address> && gpg --armor --export <your-email-address>) >> KEYS
  svn commit --username=<your_apache_username> -m "Adding my key to the KEYS file"

  4. When close to building a release candidate, try building a source tarball (on a supported platform):

      ./build-support/build_source_release.py

  5. Fix any issues it finds, such as RAT.

  6. Make sure kudu-binary JAR artifact can be successfully built both on Linux and macOS:

      ./build-support/mini-cluster/build_mini_cluster_binaries.sh

  7. Test the full Java build. This will sign and build everything without deploying any artifacts:

      # Run a gpg-agent if you don't normally.
  gpg-agent --daemon
  # List keys with identifiers in the traditional 8-character key ID format.
  # Take a note of the identifier of the key you want to use for signing.
  gpg --list-secret-keys --keyid-format=short
  cd java
  ./gradlew clean assemble  -PforceSigning -Psigning.gnupg.keyName=<8-character-pgp-key-id>

  8. Create a new version update commit which removes the -SNAPSHOT suffix (same process as above).

  9. In the newly created branch branch-1.x.y of the project git repo, update version-related information under the examples sub-directory (e.g., kudu-version in examples/java/java-example/pom.xml, version in examples/java/insert-loadgen/pom.xml, etc.). The idea is making all examples use the artifacts of the newly released 1.x.y version of Kudu. Commit and send out the changes for review on Gerrit.

  10. When ready, create a new lightweight tag and push it to the Apache Git repository.

      git tag 1.x.y-RC1
  git push apache 1.x.y-RC1

  11. Build a source tarball against the RC tag.

  12. Out of the Kudu git workspace, checkout the dev Subversion (SVN) repository. Create a new sub-directory named correspondingly. Copy the artifacts to this sub-directory and commit.

      svn co --depth=immediates https://dist.apache.org/repos/dist/dev/kudu/ kudu-dev-release
  cd kudu-dev-release
  mkdir 1.x.y-RC1
  cp <path_to_kudu_git_workspace>/build/apache-kudu-1.x.y.tar.* 1.x.y-RC1
  svn add 1.x.y-RC1
  svn commit --username=<your_apache_username> -m "Adding Kudu 1.x.y RC1"

  13. Create a Maven staging repository for the release candidate Java artifacts.

      # Run a gpg-agent if you don't normally
  gpg-agent --daemon
  cd java
  ./gradlew clean assemble -PforceSigning -Psigning.gnupg.keyName=<8-character-pgp-key-id>
  # Turn off bash history: this is to avoid exposing the credentials
  # via .bash_history file.
  set +o history
  ./gradlew --no-parallel publish \
      -Psigning.gnupg.keyName=<8-character-pgp-key-id> \
      -PmavenUsername='<APACHE-LDAP-USERNAME>' \
      -PmavenPassword='<APACHE-LDAP-PASSWORD>'
  # Turn on bash history.
  set -o history
    Note
    		 If publish is executed without --no-parallel, uploading a number of artifacts fails with "peer not authenticated" errors.

  14. Build and deploy new binary test JARs for the RC on macOS and Linux. Build the Linux JAR on a CentOS 7 image, and build the macOS JAR on macOS Big Sur if possible (see KUDU-2724 to remove the need to build on an old version of macOS).

      # Build a binary JAR for the local operating system. Make sure the thirdparty
  # components were built to match the source code that the RC is being built
  # with. It's a good idea to clone the Kudu git repo into a dedicated
  # workspace, rebuilding the thirdparty compoments from scratch for particular
  # release. The resulting JAR is output into the build/mini-cluster directory.
  ./build-support/mini-cluster/build_mini_cluster_binaries.sh
  # Sign and publish all matching kudu-binary artifacts from the
  # build/mini-cluster directory to the Maven staging repository that hosts
  # the Java artifacts of the Apache Kudu project (see above).
  # Turn off bash history: this is to avoid exposing the credentials persisted
  # in .bash_history file.
  set +o history
  ./build-support/mini-cluster/publish_mini_cluster_binaries.sh -a=deploy \
      -u='<APACHE-LDAP-USERNAME>' -p='<APACHE-LDAP-PASSWORD>'
  # Turn bash history back on.
  set -o history
    Note
    		 If the binary test JAR artifacts are deployed by the same person and from the same machine as the Java Maven artifacts, they should appear in the same staging repository. Otherwise, they will create a separate Maven staging repository, which is not a problem.
    Tip
    		 To publish an artifact outside of the build/mini-cluster directory, e.g. if JAR was built by someone else or for another OS, pass the -j=directory_name argument to the publishing script to specify the location of the JAR file to sign and publish.

  15. Close the Maven staging repository (or repositories).

    Go to the repository manager and log into the repository server using your Apache credentials. Make sure to enable Adobe Flash in your browser for this Web site. Now, go the staging repository and look for ‘orgapachekudu-####’ in the staging repositories list. You can check the Content tab at the bottom to make sure you have all of the expected stuff (client, various integrations, etc.). Hit the checkbox next to your new staging repo and hit Close. Enter something similar to "Apache Kudu 1.x.y-RC1" into the description box and confirm. Wait a minute or two and hit Refresh, and your staging repo should now have a URL shown in its summary tab (e.g. https://repository.apache.org/content/repositories/orgapachekudu-1005)

Initiating a Vote for an RC

  1. Send an email to [email protected] to start the RC process, using this example as a template.

  2. Reminder that voting on a release requires a Majority Approval by the PMC.

  3. Cycle through as many RCs as required.

  4. Always send an email with a different subject to indicate the result. For example.

  5. After the vote passes, send an email to [email protected] indicating the result.

Release

  1. For a release to be made official, the result release candidate must be put in the release SVN repository. Create a new sub-directory in the release SVN repository for the new release and copy the files from the dev repository:

      cd kudu-dist-release
  mkdir 1.x.y
  cp <path_to_kudu-dev-release>/1.x.y-RC1/* 1.x.y
  svn add 1.x.y
  svn commit --username=<your_apache_username> -m "Adding files for Kudu 1.x.y"

  2. In the Kudu git repo, create a signed tag from the RC’s tag, verify the signature has been applied and verifiable, and push it to the Apache Git repository:

      gpg --list-secret-keys --keyid-format=short
  git tag -u <gpg_key_id> -m 'Release Apache Kudu 1.x.y' 1.x.y 1.x.y-RC1
  git tag -v 1.x.y
  git push apache 1.x.y

  3. Release the staged Java artifacts. Select the release candidate staging repository in Nexus, and click Release. You should shortly be able to see the artifacts in Maven Central.

  4. Release the Python artifacts. You will need to setup an account on PyPi.org and ask to be added to the kudu-python PyPi project if you have not done this before.

      # Prepare and sign the python source distribution.
  cd python
  rm -rf dist/*
  python setup.py sdist
  gpg --detach-sign -a dist/kudu-python-1.x.y.tar.gz
  # Upload the distribution to PyPi using twine.
  pip install twine
  twine upload dist/*

    Note: You can upload to the test PyPi by adding --repository-url https://test.pypi.org/legacy/ to the twine command.

  5. Generate the version-specific documentation from that branch following these instructions.

    Warning
    		 The site MUST NOT be built on Mac. See the documentation building instructions for details.

  6. Update the index.md file in the releases directory. Add a line about the newly released version into the Latest release section, move line about the previous one into the Previous releases. Overall, update the Previous releases and Archived releases to reflect the new contents of the release SVN repository. See the last item of this section to get the idea on what the new contents of the release SVN repository is going to be.

  7. A new sub-directory named after the release version should be added into the releases directory: it should contain apidocs, cpp-client-api, docs sub-dirs and index.md file. Add the newly added sub-directory into the git staging:

      # Assuming current working directory is the root of the git workspace.
  git add releases/1.x.y

  8. Take a quick look at the auto-generated releases/1.x.y/index.md file to make sure the download links meet the current criteria. The criteria keep changing and the announcement will be rejected if our release page doesn’t meet the criteria.

  9. Commit the changes:

      git commit -a -m "update website for 1.x.y release"

  10. Submit these changes to the gh-pages Gerrit branch and get them reviewed.

  11. Add a release blog post about the new release and send it out for review, similar to this.

  12. Once the reviews are finished and the commits are pushed to gh-pages branch, update the website following these instructions.

  13. Build and push the Docker images (run the below command from the project root).

     ./docker/docker-build.py --action push --platforms linux/amd64 linux/arm64

    To push the images to Dockerhub, you need to be granted permissions by the ASF infra team, which you can request via an Apache JIRA ticket similar to INFRA-23166.

  14. About 24 hours after all artifacts have been published, send an email to [email protected], [email protected], and [email protected] to announce the new release. The email should be similar to this. The announcement email must be sent from your apache.org address, otherwise apache.org mailer will not deliver the message to the addressees of the announce@ group at least. If using gmail.com service for email correspondence, follow this guide to add your apache.org address to be used as From address when sending email messages. Also post the announcement to the blog and tweet it with the @ApacheKudu Twitter handle.

  15. Update the version number on the branch you released from back to a SNAPSHOT for the next patch release, such as 1.6.1-SNAPSHOT after the 1.6.0 release.

  16. In the master branch of the project git repo, update version-related information under the examples sub-directory to make all examples using the fresh artifacts of latest released version. This should be done only after publishing the officially released artifacts, so they are available while building the examples.

  17. In the master branch of the project git repo, add the previous release notes to the prior_release_notes.adoc and send out the change for review on Gerrit.

  18. About another 24 hours later, clean up the SVN. If releasing a new minor version, delete the oldest minor version branch in the release repo (e.g. if 1.7.1, 1.8.0, and 1.9.0 exist and you just released 1.10.0, delete 1.7.1). If releasing a maintenance version, delete the previous maintenance branch (e.g. if you released 1.2.1, delete 1.2.0). Also delete any release candidates from the dev SVN.

  19. Set the release date for the just released version in Apache JIRA or ask someone with permission to do so. After logging in, in the left sidebar switch to the Releases item. Once at the Releases page, apply the Unreleased filter. In the row corresponding to the new release, click at the ellipsis in the Action column and select the Release item, setting the appropriate release date.