Upgrade Red Hat Quay

Red Hat Quay 3.3

Upgrade Red Hat Quay

Red Hat OpenShift Documentation Team

Abstract

Upgrade Red Hat Quay

Preface

This guide describes how to upgrade to Red Hat Quay v3.2. To upgrade from Red Hat Quay v2.y.z to v3.3.4, you must:

  • Upgrade from Red Hat Quay v2.y.z to v3.0.z
  • Then upgrade from Red Hat Quay v3.0.z to v3.3.4 sequentially via each minor Red Hat Quay release

To upgrade from v3.0.z to v3.3.4 you must shut down your Red Hat Quay cluster, make a small change to the configuration, then upgrade your Red Hat Quay v3.0.z containers sequentially to each minor Red Hat Quay version (e.g. 3.1.0, 3.2.0, etc..) until reaching the latest v3.3.4 containers. Procedures for these upgrades are described below.

Chapter 1. Red Hat Quay v2.y.z to v3.0.z upgrade

For the v2.y.z to v3.0.z part of the upgrade, you can either do the whole upgrade with Red Hat Quay down (Synchronous) or only bring down Red Hat Quay for a few minutes and have bulk of the upgrade continue with Red Hat Quay running (Background).

In a background upgrade, it could take much longer to run the upgrade (depending on how many tags need to be processed), but it takes less total downtime. The downside of a background upgrade is that you won’t have access to the latest features until the upgrade completes (the cluster runs from the quay v3 container in v2 compatibility mode until the upgrade is done).

See the Red Hat Quay Release Notes for information on v3 features.

Chapter 2. Red Hat Quay v3.y.z to v3.3.4 upgrade

Red Hat Quay supports upgrades from a prior (N-1) minor version only. For example, upgrading from 3.0.1 to 3.2.1 is not supported. You will need to first upgrade from 3.0.1 to 3.1.0 before attempting to upgrade to 3.2.1. This is required to ensure that any necessary database migrations are done correctly and in the right order during the upgrade.

Once your Red Hat Quay cluster is running on the correct N-1 minor version and your V3_UPGRADE_MODE is completed, you can upgrade that cluster to v3.3.4 by:

  • Taking all hosts in the Red Hat Quay cluster out of service
  • Making a small change to the configuration (as described later)
  • Use the new configuration file to fully migrate to a new v3.3.4 container
  • Checking that the new v3.3.4 container is performing well
  • Add the v3.3.4 hosts back into service.

Chapter 3. Overview of Red Hat Quay v3.0 Upgrade

Follow the procedure below if you are beginning with a Red Hat Quay v2 cluster. Before upgrading to the latest Red Hat Quay 3.x version, you must first migrate that cluster to v3.0.z, as described here. Once your cluster is running v3.0.z, you can then upgrade to the latest 3.x version by sequentially upgrading to each minor version in turn (3.0 to 3.1 to 3.2, etc…​)

Before beginning your Red Hat Quay v2 to v3.0 upgrade, please note the following:

  • Synchronous upgrade: For a synchronous upgrade, expect less than one hour of total downtime for small installations. Consider a small installation to contain a few thousand container image tags or fewer. For that size installation, you could probably get by with just a couple hours of scheduled downtime. The entire Red Hat Quay service is down for the duration, so if you were to try a synchronous upgrade on a registry with millions of tags, you could potentially be down for several days.
  • Background upgrade: For a background upgrade (also called a compatibility mode upgrade), after a short shutdown your Red Hat Quay cluster upgrade runs in the background. For large Red Hat Quay registries, this could take weeks to complete, but the cluster continues to operate in v2 mode for the duration of the upgrade. As a point of reference, one Red Hat Quay v3 upgrade took four days to process approximately 30 million tags across six machines.
  • Full features on completion: Before you have access to features associated with Docker version 2, schema 2 changes (such as support for containers of different architectures), the entire migration must complete. Other v3 features are immediately available when you switch over.
  • Upgrade complete: When the upgrade is complete, you need to set V3_UPGRADE_MODE: complete in the Red Hat Quay config.yaml file for the new features to be available. All new Red Hat Quay v3 installations automatically have that set.

Chapter 4. Prerequisites

To assure the best results, we recommend the following prerequisites:

  • Back up your Red Hat Quay database before starting the upgrade (doing regular backups is a general best practice). A good time to do this is right after you have taken down the Red Hat Quay cluster to do the upgrade.
  • Back up your storage (also a general best practice).

  • Upgrade your current Red Hat Quay 2.y.z setup to the latest 2.y.z version (currently 2.9.5) before starting the v3 upgrade. To do that, you can:

    • While the Red Hat Quay cluster is still running, take one node and change the quay container on that system to a quay container that is running the latest 2.9.z version.
    • Wait for all the database migrations to run, bringing the database up to the latest 2.9.z version. This should only take a few minutes to a half an hour.
    • Once that is done, replace the quay container on all the existing nodes with the same latest 2.9.z version. With the entire Red Hat Quay cluster on the new version, you can proceed to the v3 upgrade.

Chapter 5. Running the upgrade to Red Hat Quay v3.0.z

Choose between a synchronous upgrade (complete the upgrade in downtime) and a background upgrade (complete the upgrade while Red Hat Quay is still running). Both of these major-release upgrades require that the Red Hat Quay cluster be down for at least a short period of time.

Regardless of which upgrade type you choose, during the time that the Red Hat Quay cluster is down, if you are using builder and clair images, you need to also upgrade to those new images:

  • The builder image (quay.io/redhat/quay-builder:v3.0.5)
  • The clair image (quay.io/redhat/clair-jwt:v3.0.5)

Both of those images are available from the quay.io/redhat repository.

5.1. Run a synchronous upgrade

To run a synchronous upgrade, where your whole cluster is down for the entire upgrade, do the following:

  1. Take down your entire Red Hat Quay cluster, including any quay-builder and clair containers.

  2. Add the following setting to the config.yaml file on all nodes:

    V3_UPGRADE_MODE: complete

  3. Pull and start up the v3 container on a single node and wait for however long it takes to do the upgrade (it should take just a few minutes). Use the following container or later:

    quay.io/redhat/quay:v3.0.5

    Note that the quay container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example: 

    # docker run --restart=always -p 443:8443 -p 80:8080 \
   --sysctl net.core.somaxconn=4096 \
   --privileged=true \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d quay.io/redhat/quay:v3.0.5
  4. After the upgrade completes, bring the Red Hat Quay v3 container up on all other nodes.
  5. Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
  6. Verify that Red Hat Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).

5.2. Run a background upgrade

To run a background upgrade, you need only bring down your cluster for a short period of time on two occasions. When you bring the cluster back up after the first downtime, the quay v3 container runs in v2 compatibility mode as it backfills the database. This background process can take hours or even days to complete. Background upgrades are recommended for large installations where downtime of more than a few hours would be a problem.

For this type of upgrade, you put Red Hat Quay into a compatibility mode, where you have a v3 quay container running, but it is running on the old data model while the upgrade completes. Here’s what you do:

  1. Pull the Red Hat Quay v3 container to all the nodes. Use the following container or later:

    quay.io/redhat/quay:v3.0.5

  2. Take down your entire Red Hat Quay cluster, including any quay-builder and clair containers.

  3. Edit the config.yaml file on each node and set the upgrade mode to background as follows:

    V3_UPGRADE_MODE: background

  4. Bring the Red Hat Quay v3 container up on a single node and wait for the migrations to complete (should take a few minutes maximum). Here is an example of that command:

    Note that the quay container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example: 

    # docker run --restart=always -p 443:8443 -p 80:8080 \
   --sysctl net.core.somaxconn=4096 \
   --privileged=true \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d quay.io/redhat/quay:v3.0.5
  5. Bring the Red Hat Quay v3 container up on all the other nodes.
  6. Monitor the /upgradeprogress API endpoint until it reports done enough to move to the next step (the status reaches 99%). For example, view https://myquay.example.com/upgradeprogress or use some other tool to query the API.
  7. Once the background process is far enough along you have to schedule another maintenance window.
  8. During your scheduled maintenance, take the entire Red Hat Quay cluster down.

  9. Edit the config.yaml file on each node and set the upgrade mode to complete as follows:

    V3_UPGRADE_MODE: complete

  10. Bring Red Hat Quay back up on one node to have it do a final check.
  11. Once the final check is done, bring Red Hat Quay v3 back up on all the other nodes.
  12. Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.
  13. Verify Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).

Chapter 6. Running the upgrade to Red Hat Quay v3.3.4

As noted earlier, once your cluster is running any Red Hat Quay 3.0.z or 3.1.z version, to upgrade your cluster to v3.3.4 you must bring down your entire cluster and make a small change to the configuration before bringing it back up with the 3.3.4 version. Here’s how to do that.

Warning

Once you set the value of DATABASE_SECRET_KEY in this procedure, do not ever change it. If you do, all tokens will need to be regenerated.

  1. Take all hosts in the Red Hat Quay cluster out of service.

  2. Generate some random data to use as a database secret key. For example: 

    $ openssl rand -hex 48
2d023adb9c477305348490aa0fd9c

  3. Add a new DATABASE_SECRET_KEY field to your config.yaml file. For example: 

    DATABASE_SECRET_KEY: "2d023adb9c477305348490aa0fd9c"
    Note

    For an OpenShift installation, the config.yaml file is stored as a secret.

  4. Bring up one quay container to complete the migration to v3.3.4.
  5. Once the migration is done, make sure the same config.yaml is available on all nodes and bring up the new quay v3.3.4 service on those nodes.
  6. Start v3.0.z versions of quay-builder and clair to replace any instances of those containers you want to return to your cluster.

Legal Notice

Copyright © 2021 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.