Upgrade Red Hat Quay

Red Hat Quay 3

Upgrade Red Hat Quay

Red Hat OpenShift Documentation Team

Abstract

Upgrade Red Hat Quay

Preface

This guide describes how to upgrade to Red Hat Quay v3.1. To upgrade from Red Hat Quay v2.y.z to v3.1.0, you must:

  • Upgrade from Red Hat Quay v2.y.z to v3.0.z
  • Then upgrade from Red Hat Quay v3.0.z to v3.1.0

To upgrade from v3.0.z to v3.1.0 you simply need to do a rolling upgrade of your Red Hat Quay v3.0.z containers to the latest v3.1.0 containers. Procedures for these upgrades are described below.

Chapter 1. Red Hat Quay v2.y.z to v3.0.z upgrade

For the v2.y.z to v3.0.z portion, you can either do the whole upgrade with Red Hat Quay down (Synchronous) or only bring down Red Hat Quay for a few minutes and have bulk of the upgrade continue with Red Hat Quay running (Background).

In a background upgrade, it could take much longer to run the upgrade (depending on how many tags need to be processed), but it takes less total downtime. The downside of a background upgrade is that you won’t have access to the latest features until the upgrade completes (the cluster runs from the quay v3 container in v2 compatibility mode until the upgrade is done).

See the Red Hat Quay Release Notes for information on v3 features.

Chapter 2. Red Hat Quay v3.0.z to v3.1.0 upgrade

Once your Red Hat Quay cluster is running on any v3.0.z version and your V3_UPGRADE_MODE is completed, you can upgrade that cluster from v3.0.z to v3.1.0 by simply:

  • Adding new v3.1.0 quay containers into your Red Hat Quay cluster one at a time
  • Checking that the new containers are performing well
  • Continuing to add more containers into the cluster
  • Removing the older versions of the quay container (a rolling upgrade) until all the old containers are replaced

Chapter 3. Overview of Red Hat Quay v3 Upgrade

Before beginning your Red Hat Quay v2 to v3 upgrade, please note the following:

  • Synchronous upgrade: For a synchronous upgrade, expect less than one hour of total downtime for small installations. Consider a small installation to contain a few thousand container image tags or fewer. For that size installation, you could probably get by with just a couple hours of scheduled downtime. The entire Red Hat Quay service is down for the duration, so if you were to try a synchronous upgrade on a registry with millions of tags, you could potentially be down for several days.
  • Background upgrade: For a background upgrade (also called a compatibility mode upgrade), after a short shutdown your Red Hat Quay cluster upgrade runs in the background. For large Red Hat Quay registries, this could take weeks to complete, but the cluster continues to operate in v2 mode for the duration of the upgrade. As a point of reference, one Red Hat Quay v3 upgrade took four days to process approximately 30 million tags across six machines.
  • Full features on completion: Before you have access to features associated with Docker version 2, schema 2 changes (such as support for containers of different architectures), the entire migration must complete. Other v3 features are immediately available when you switch over.
  • Upgrade complete: When the upgrade is complete, you need to set V3_UPGRADE_MODE: complete in the Red Hat Quay config.yaml file for the new features to be available. All new Red Hat Quay v3 installations automatically have that set.

Chapter 4. Prerequisites

To assure the best results, we recommend the following prerequisites:

  • Back up your Red Hat Quay database before starting the upgrade (doing regular backups is a general best practice). A good time to do this is right after you have taken down the Red Hat Quay cluster to do the upgrade.
  • Back up your storage (also a general best practice).
  • Upgrade your current Red Hat Quay 2.y.z setup to the latest 2.y.z version (currently 2.9.5) before starting the v3 upgrade. To do that, you can:

    • While the Red Hat Quay cluster is still running, take one node and change the quay container on that system to a quay container that is running the latest 2.9.z version.
    • Wait for all the database migrations to run, bringing the database up to the latest 2.9.z version. This should only take a few minutes to a half an hour.
    • Once that is done, replace the quay container on all the existing nodes with the same latest 2.9.z version. With the entire Red Hat Quay cluster on the new version, you can proceed to the v3 upgrade.

Chapter 5. Running the upgrade to Red Hat Quay v3.0.z

Choose between a synchronous upgrade (complete the upgrade in downtime) and a background upgrade (complete the upgrade while Red Hat Quay is still running). Both of these major-release upgrades require that the Red Hat Quay cluster be down for at least a short period of time.

Regardless of which upgrade type you choose, during the time that the Red Hat Quay cluster is down, you are using builder and clair images, you need to also upgrade to those new images:

  • The builder image (quay.io/redhat/quay-builder:v3.0.5)
  • The clair image (quay.io/redhat/clair-jwt:v3.0.5)

Both of those images are available from the new quay.io/redhat repository.

5.1. Run a synchronous upgrade

To run a synchronous upgrade, where your whole cluster is down for the entire upgrade, do the following:

  1. Take down your entire Red Hat Quay cluster.
  2. Add the following setting to the config.yaml file on all nodes:

    V3_UPGRADE_MODE: complete

  3. Pull and start up the v3 container on a single node and wait for however long it takes to do the upgrade (it should take just a few minutes). Use the following container or later:

    quay.io/redhat/quay:v3.0.5

    Note that the quay container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example:

    # docker run --restart=always -p 443:8443 -p 80:8080 \
       --sysctl net.core.somaxconn=4096 \
       -v /mnt/quay/config:/conf/stack:Z \
       -v /mnt/quay/storage:/datastorage:Z \
       -d quay.io/redhat/quay:v3.0.5
  4. After the upgrade completes, bring the Red Hat Quay v3 container up on all other nodes.
  5. Verify that Red Hat Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).

5.2. Run a background upgrade

To run a background upgrade, you need only bring down your cluster for a short period of time on two occasions. When you bring the cluster back up after the first downtime, the quay v3 container runs in v2 compatibility mode as it backfills the database. This background process can take hours or even days to complete. Background upgrades are recommended for large installations where downtime of more than a few hours would be a problem.

For this type of upgrade, you put Red Hat Quay into a compatibility mode, where you have a v3 quay container running, but it is running on the old data model while the upgrade completes. Here’s what you do:

  1. Pull the Red Hat Quay v3 container to all the nodes. Use the following container or later:

    quay.io/redhat/quay:v3.0.5

  2. Take down Red Hat Quay on all nodes.
  3. Edit the config.yaml file on each node and set the upgrade mode to background as follows:

    V3_UPGRADE_MODE: background

  4. Bring the Red Hat Quay v3 container up on a single node and wait for the migrations to complete (should take a few minutes maximum). Here is an example of that command:

    Note that the quay container comes up on ports 8080 and 8443 for v3, instead of 80 and 443, as they did for v2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example:

    # docker run --restart=always -p 443:8443 -p 80:8080 \
       --sysctl net.core.somaxconn=4096 \
       -v /mnt/quay/config:/conf/stack:Z \
       -v /mnt/quay/storage:/datastorage:Z \
       -d quay.io/redhat/quay:v3.0.5
  5. Bring the Red Hat Quay v3 container up on all the other nodes.
  6. Monitor the /upgradeprogress API endpoint until it reports done (the status reaches 100%). For example, view https://myquay.example.com/upgradeprogress or use some other tool to query the API.
  7. Once the background process is done, you have to schedule another maintenance window.
  8. During your scheduled maintenance, take the entire Red Hat Quay cluster down.
  9. Edit the config.yaml file on each node and set the upgrade mode to complete as follows:

    V3_UPGRADE_MODE: complete

  10. Bring Red Hat Quay back up on one node to have it do a final check.
  11. Once the final check is done, bring Red Hat Quay v3 back up on all the other nodes.
  12. Verify Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).

Chapter 6. Running the upgrade to Red Hat Quay v3.1.0

As noted earlier, once your cluster is running any Red Hat Quay 3.0.z version, you can upgrade your cluster to v3.1.0 by simply doing a rolling upgrade. To do that, just add new v3.1.0 containers to the cluster one at a time, check that they are working, then decommission the older v3.0.z containers.

Legal Notice

Copyright © 2019 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.