Chapter 3. Standalone upgrade

In general, Red Hat Quay supports upgrades from a prior (N-1) minor version only. For example, upgrading directly from Red Hat Quay 3.0.5 to the latest version of 3.5 is not supported. Instead, users would have to upgrade as follows:

  1. 3.0.5 → 3.1.3
  2. 3.1.3 → 3.2.2
  3. 3.2.2 → 3.3.4
  4. 3.3.4 → 3.4.z
  5. 3.4.z → 3.5.z

This is required to ensure that any necessary database migrations are done correctly and in the right order during the upgrade.

In some cases, Red Hat Quay supports direct, single-step upgrades from prior (N-2, N-3) minor versions. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases. The following upgrade paths are supported:

  1. 3.3.z → 3.6.z
  2. 3.4.z → 3.6.z
  3. 3.4.z → 3.7.z
  4. 3.5.z → 3.7.z

For users wanting to upgrade via the Quay Operator, see Upgrading Quay by upgrading the Quay Operator.

This document describes the steps needed to perform each individual upgrade. Determine your current version and then follow the steps in sequential order, starting with your current version and working up to your desired target version.

See the Red Hat Quay Release Notes for information on features for individual releases.

The general procedure for a manual upgrade consists of the following steps:

  1. Stop the Quay and Clair containers.
  2. Backup the database and image storage (optional but recommended).
  3. Start Clair using the new version of the image.
  4. Wait until Clair is ready to accept connections before starting the new version of Quay.

3.1. Accessing images

Images for Quay 3.4.0 and later are available from registry.redhat.io and registry.access.redhat.com, with authentication set up as described in Red Hat Container Registry Authentication.

Images for Quay 3.3.4 and earlier are available from quay.io, with authentication set up as described in Accessing Red Hat Quay without a CoreOS login.

3.2. Upgrade to 3.7.z from 3.6.z

3.2.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.7.13
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.7.13
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-6

3.3. Upgrade to 3.7.z from 3.5.z

3.3.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.7.13
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.7.13
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-6

3.4. Upgrade to 3.7.z from 3.4.z

3.4.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.7.13
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.7.13
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-6

3.5. Upgrade to 3.7.z from 3.3.z

Upgrading to Red Hat Quay 3.7 from 3.3. is unsupported. Users must first upgrade to 3.6 from 3.3, and then upgrade to 3.7. For more information, see Upgrade to 3.6.z from 3.3.z.

3.6. Upgrade to 3.6.z from 3.5.z

3.6.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.6.0
  • Clair: registry.redhat.io/quay/clair-rhel8:v.3.60
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-6

3.7. Upgrade to 3.6.z from 3.4.z

+

Note

Red Hat Quay 3.6 supports direct, single-step upgrade from 3.4.z. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases.

Upgrading to Red Hat Quay 3.6 from 3.4.z requires a database migration which does not support downgrading back to a prior version of Red Hat Quay. Please back up your database before performing this migration.

Users will also need to configure a completely new Clair v4 instance to replace the old Clair v2 when upgrading from 3.4.z. For instructions on configuring Clair v4, see Setting up Clair on a non-OpenShift Red Hat Quay deployment.

3.7.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.6.0
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.6.0
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-6

3.8. Upgrade to 3.6.z from 3.3.z

+

Note

Red Hat Quay 3.6 supports direct, single-step upgrade from 3.3.z. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases.

Upgrading to Red Hat Quay 3.6.z from 3.3.z requires a database migration which does not support downgrading back to a prior version of Red Hat Quay. Please back up your database before performing this migration.

Users will also need to configure a completely new Clair v4 instance to replace the old Clair v2 when upgrading from 3.3.z. For instructions on configuring Clair v4, see Setting up Clair on a non-OpenShift Red Hat Quay deployment.

3.8.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.6.0
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.6.0
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-6

3.8.2. Swift configuration when upgrading from 3.3.z to 3.6

When upgrading from Red Hat Quay 3.3.z to 3.6.z, some users might receive the following error: Switch auth v3 requires tenant_id (string) in os_options. As a workaround, you can manually update your DISTRIBUTED_STORAGE_CONFIG to add the os_options and tenant_id parameters: 

  DISTRIBUTED_STORAGE_CONFIG:
    brscale:
    - SwiftStorage
    - auth_url: http://****/v3
      auth_version: "3"
      os_options:
        tenant_id: ****
        project_name: ocp-base
        user_domain_name: Default
      storage_path: /datastorage/registry
      swift_container: ocp-svc-quay-ha
      swift_password: *****
      swift_user: *****

3.9. Upgrade to 3.5.7 from 3.4.z

3.9.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.5.7
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.5.7
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10:1
  • Redis: registry.redhat.io/rhel8/redis-5

3.10. Upgrade to 3.4.6 from 3.3.z

Upgrading to Quay 3.4 requires a database migration which does not support downgrading back to a prior version of Quay. Please back up your database before performing this migration.

3.10.1. Target images

  • Quay: registry.redhat.io/quay/quay-rhel8:v3.4.6
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.4.6
  • PostgreSQL: registry.redhat.io/rhel8/postgresql-10
  • Redis: registry.redhat.io/rhel8/redis-5

3.11. Upgrade to 3.3.4 from 3.2.z

3.11.1. Target images

  • Quay: quay.io/redhat/quay:v3.3.4
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.3.4
  • PostgreSQL: rhscl/postgresql-96-rhel7
  • Redis: registry.access.redhat.com/rhscl/redis-32-rhel7

3.12. Upgrade to 3.2.2 from 3.1.z

Once your cluster is running any Red Hat Quay 3.1.z version, to upgrade your cluster to 3.2.2 you must bring down your entire cluster and make a small change to the configuration before bringing it back up with the 3.2.2 version.

Warning

Once you set the value of DATABASE_SECRET_KEY in this procedure, do not ever change it. If you do so, then existing robot accounts, API tokens, etc. cannot be used anymore. You would have to create a new robot account and API tokens to use with Quay.

  1. Take all hosts in the Red Hat Quay cluster out of service.

  2. Generate some random data to use as a database secret key. For example: 

    $ openssl rand -hex 48
2d023adb9c477305348490aa0fd9c

  3. Add a new DATABASE_SECRET_KEY field to your config.yaml file. For example: 

    DATABASE_SECRET_KEY: "2d023adb9c477305348490aa0fd9c"
    Note

    For an OpenShift installation, the config.yaml file is stored as a secret.

  4. Bring up one Quay container to complete the migration to 3.2.2.
  5. Once the migration is done, make sure the same config.yaml is available on all nodes and bring up the new quay 3.2.2 service on those nodes.
  6. Start 3.0.z versions of quay-builder and Clair to replace any instances of those containers you want to return to your cluster.

3.12.1. Target images

  • Quay: quay.io/redhat/quay:v3.2.2
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.7.13
  • PostgreSQL: rhscl/postgresql-96-rhel7
  • Redis: registry.access.redhat.com/rhscl/redis-32-rhel7

3.13. Upgrade to 3.1.3 from 3.0.z

3.13.1. Target images

  • Quay: quay.io/redhat/quay:v3.1.3
  • Clair: registry.redhat.io/quay/clair-rhel8:v3.7.13
  • PostgreSQL: rhscl/postgresql-96-rhel7
  • Redis: registry.access.redhat.com/rhscl/redis-32-rhel7

3.14. Upgrade to 3.0.5 from 2.9.5

For the 2.9.5 to 3.0.5 upgrade, you can either do the whole upgrade with Red Hat Quay down (synchronous upgrade) or only bring down Red Hat Quay for a few minutes and have the bulk of the upgrade continue with Red Hat Quay running (background upgrade).

A background upgrade could take longer to run the upgrade depending on how many tags need to be processed. However, there is less total downtime. The downside of a background upgrade is that you will not have access to the latest features until the upgrade completes. The cluster runs from the Quay v3 container in v2 compatibility mode until the upgrade is complete.

3.14.1. Overview of upgrade

Follow the procedure below if you are starting with a Red Hat Quay 2.y.z cluster. Before upgrading to the latest Red Hat Quay 3.x version, you must first migrate that cluster to 3.0.5, as described here. Once your cluster is running 3.0.5, you can then upgrade to the latest 3.x version by sequentially upgrading to each minor version in turn. For example:

  1. 3.0.5 → 3.1.3
  2. 3.1.3 → 3.2.2
  3. 3.2.2 → 3.3.4
  4. 3.3.4 → 3.4.z

Before beginning your Red Hat Quay 2.y.z to 3.0 upgrade, please note the following:

  • Synchronous upgrade: For a synchronous upgrade, expect less than one hour of total downtime for small installations. Consider a small installation to contain a few thousand container image tags or fewer. For that size installation, you could probably get by with just a couple hours of scheduled downtime. The entire Red Hat Quay service is down for the duration, so if you were to try a synchronous upgrade on a registry with millions of tags, you could potentially be down for several days.
  • Background upgrade: For a background upgrade (also called a compatibility mode upgrade), after a short shutdown your Red Hat Quay cluster upgrade runs in the background. For large Red Hat Quay registries, this could take weeks to complete, but the cluster continues to operate in v2 mode for the duration of the upgrade. As a point of reference, one Red Hat Quay v3 upgrade took four days to process approximately 30 million tags across six machines.
  • Full features on completion: Before you have access to features associated with Docker version 2, schema 2 changes (such as support for containers of different architectures), the entire migration must complete. Other v3 features are immediately available when you switch over.
  • Upgrade complete: When the upgrade is complete, you need to set V3_UPGRADE_MODE: complete in the Red Hat Quay config.yaml file for the new features to be available. All new Red Hat Quay v3 installations automatically have that set.

3.14.2. Prerequisites

To assure the best results, we recommend the following prerequisites:

  • Back up your Red Hat Quay database before starting the upgrade (doing regular backups is a general best practice). A good time to do this is right after you have taken down the Red Hat Quay cluster to do the upgrade.
  • Back up your storage (also a general best practice).

  • Upgrade your current Red Hat Quay 2.y.z setup to the latest 2.9.z version (currently 2.9.5) before starting the v3 upgrade. To do that:

    • While the Red Hat Quay cluster is still running, take one node and change the Quay container on that system to a Quay container that is running the latest 2.9.z version.
    • Wait for all the database migrations to run, bringing the database up to the latest 2.9.z version. This should only take a few minutes to a half an hour.
    • Once that is done, replace the Quay container on all the existing nodes with the same latest 2.9.z version. With the entire Red Hat Quay cluster on the new version, you can proceed to the v3 upgrade.

3.14.3. Choosing upgrade type

Choose between a synchronous upgrade (complete the upgrade in downtime) and a background upgrade (complete the upgrade while Red Hat Quay is still running). Both of these major-release upgrades require that the Red Hat Quay cluster be down for at least a short period of time.

Regardless of which upgrade type you choose, during the time that the Red Hat Quay cluster is down, if you are using builder and Clair images, you need to also upgrade to those new images:

  • Builder: quay.io/redhat/quay-builder:v3.0.5
  • Clair: quay.io/redhat/clair-jwt:v3.0.5

Both of those images are available from the registry.redhat.io/quay repository.

3.14.4. Running a synchronous upgrade

To run a synchronous upgrade, where your whole cluster is down for the entire upgrade, do the following:

  1. Take down your entire Red Hat Quay cluster, including any quay-builder and Clair containers.

  2. Add the following setting to the config.yaml file on all nodes:

    V3_UPGRADE_MODE: complete

  3. Pull and start up the v3 container on a single node and wait for however long it takes to do the upgrade (it will take a few minutes). Use the following container or later:

    • Quay: quay.io/redhat/quay:v3.0.5

      Note that the Quay container comes up on ports 8080 and 8443 for Red Hat Quay 3, instead of 80 and 443, as they did for Red Hat Quay 2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example: 

    # docker run --restart=always -p 80:8080 -p 443:8443 \
   --sysctl net.core.somaxconn=4096 \
   --privileged=true \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d quay.io/redhat/quay:v3.0.5
  4. After the upgrade completes, bring the Red Hat Quay 3 container up on all other nodes.
  5. Start 3.0.z versions of quay-builder and Clair to replace any instances of those containers you want to return to your cluster.
  6. Verify that Red Hat Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).

3.14.5. Running a background upgrade

To run a background upgrade, you need only bring down your cluster for a short period of time on two occasions. When you bring the cluster back up after the first downtime, the quay v3 container runs in v2 compatibility mode as it backfills the database. This background process can take hours or even days to complete. Background upgrades are recommended for large installations where downtime of more than a few hours would be a problem.

For this type of upgrade, you put Red Hat Quay into a compatibility mode, where you have a Quay 3 container running, but it is running on the old data model while the upgrade completes. Here’s what you do:

  1. Pull the Red Hat Quay 3 container to all the nodes. Use the following container or later:

    quay.io/redhat/quay:v3.0.5

  2. Take down your entire Red Hat Quay cluster, including any quay-builder and Clair containers.

  3. Edit the config.yaml file on each node and set the upgrade mode to background as follows:

    V3_UPGRADE_MODE: background

  4. Bring the Red Hat Quay 3 container up on a single node and wait for the migrations to complete (should take a few minutes maximum). Here is an example of that command:

    Note that the Quay container comes up on ports 8080 and 8443 for Red Hat Quay 3, instead of 80 and 443, as they did for Red Hat Quay 2. Therefore, we recommend remapping 8080 and 8443 into 80 and 443, respectively, as shown in this example: 

    # docker run --restart=always -p 80:8080 -p 443:8443 \
   --sysctl net.core.somaxconn=4096 \
   --privileged=true \
   -v /mnt/quay/config:/conf/stack:Z \
   -v /mnt/quay/storage:/datastorage:Z \
   -d quay.io/redhat/quay:v3.0.5
  5. Bring the Red Hat Quay 3 container up on all the other nodes.
  6. Monitor the /upgradeprogress API endpoint until it reports done enough to move to the next step (the status reaches 99%). For example, view https://myquay.example.com/upgradeprogress or use some other tool to query the API.
  7. Once the background process is far enough along you have to schedule another maintenance window.
  8. During your scheduled maintenance, take the entire Red Hat Quay cluster down.

  9. Edit the config.yaml file on each node and set the upgrade mode to complete as follows: 

    V3_UPGRADE_MODE: complete
  10. Bring Red Hat Quay back up on one node to have it do a final check.
  11. Once the final check is done, bring Red Hat Quay v3 back up on all the other nodes.
  12. Start 3.0.z versions of quay-builder and Clair to replace any instances of those containers you want to return to your cluster.
  13. Verify Quay is working, including pushes and pulls of containers compatible with Docker version 2, schema 2. This can include windows container images and images of different computer architectures (arm, ppc, etc.).

3.14.6. Target images

  • Quay: quay.io/redhat/quay:v3.0.5
  • Clair: quay.io/redhat/clair-jwt:v3.0.5
  • Redis: registry.access.redhat.com/rhscl/redis-32-rhel7
  • PostgreSQL: rhscl/postgresql-96-rhel7
  • Builder: quay.io/redhat/quay-builder:v3.0.5