Chapter 13. Upgrading the Quay Operator Overview
The Quay Operator follows a synchronized versioning scheme, which means that each version of the Operator is tied to the version of Quay and the components that it manages. There is no field on the
QuayRegistry custom resource which sets the version of Quay to deploy; the Operator only knows how to deploy a single version of all components. This scheme was chosen to ensure that all components work well together and to reduce the complexity of the Operator needing to know how to manage the lifecycles of many different versions of Quay on Kubernetes.
13.1. Operator Lifecycle Manager
The Quay Operator should be installed and upgraded using the Operator Lifecycle Manager (OLM). When creating a
Subscription with the default
approvalStrategy: Automatic, OLM will automatically upgrade the Quay Operator whenever a new version becomes available.
When the Quay Operator is installed via Operator Lifecycle Manager, it may be configured to support automatic or manual upgrades. This option is shown on the Operator Hub page for the Quay Operator during installation. It can also be found in the Quay Operator
Subscription object via the
approvalStrategy field. Choosing
Automatic means that your Quay Operator will automatically be upgraded whenever a new Operator version is released. If this is not desirable, then the
Manual approval strategy should be selected.
13.2. Upgrading the Quay Operator
The standard approach for upgrading installed Operators on OpenShift is documented at Upgrading installed Operators.
In general, Red Hat Quay supports upgrades from a prior (N-1) minor version only. For example, upgrading directly from Red Hat Quay 3.0.5 to the latest version of 3.5 is not supported. Instead, users would have to upgrade as follows:
- 3.0.5 → 3.1.3
- 3.1.3 → 3.2.2
- 3.2.2 → 3.3.4
- 3.3.4 → 3.4.z
- 3.4.z → 3.5.z
This is required to ensure that any necessary database migrations are done correctly and in the right order during the upgrade.
In some cases, Red Hat Quay supports direct, single-step upgrades from prior (N-2, N-3) minor versions. This exception to the normal, prior minor version-only, upgrade simplifies the upgrade procedure for customers on older releases. The following upgrade paths are supported:
- 3.3.z → 3.6.z
- 3.4.z → 3.6.z
- 3.4.z → 3.7.z
- 3.5.z → 3.7.z
- 3.7.z → 3.8.z
For users on standalone deployments of Quay wanting to upgrade to 3.8, see the Standalone upgrade guide.
13.2.1. Upgrading Quay
To update Quay from one minor version to the next, for example, 3.4 → 3.5, you need to change the update channel for the Quay Operator.
z stream upgrades, for example, 3.4.2 → 3.4.3, updates are released in the major-minor channel that the user initially selected during install. The procedure to perform a
z stream upgrade depends on the
approvalStrategy as outlined above. If the approval strategy is set to
Automatic, the Quay Operator will upgrade automatically to the newest
z stream. This results in automatic, rolling Quay updates to newer
z streams with little to no downtime. Otherwise, the update must be manually approved before installation can begin.
13.2.2. Notes on upgrading directly from 3.3.z or 3.4.z to 3.6
188.8.131.52. Upgrading with edge routing enabled
- Previously, when running a 3.3.z version of Red Hat Quay with edge routing enabled, users were unable to upgrade to 3.4.z versions of Red Hat Quay. This has been resolved with the release of Red Hat Quay 3.6.
When upgrading from 3.3.z to 3.6, if
tls.terminationis set to
nonein your Red Hat Quay 3.3.z deployment, it will change to HTTPS with TLS edge termination and use the default cluster wildcard certificate. For example:
apiVersion: redhatcop.redhat.io/v1alpha1 kind: QuayEcosystem metadata: name: quay33 spec: quay: imagePullSecretName: redhat-pull-secret enableRepoMirroring: true image: quay.io/quay/quay:v3.3.4-2 ... externalAccess: hostname: quayv33.apps.devcluster.openshift.com tls: termination: none database: ...
184.108.40.206. Upgrading with custom TLS certificate/key pairs without Subject Alternative Names
There is an issue for customers using their own TLS certificate/key pairs without Subject Alternative Names (SANs) when upgrading from Red Hat Quay 3.3.4 to Red Hat Quay 3.6 directly. During the upgrade to Red Hat Quay 3.6, the deployment is blocked, with the error message from the Quay Operator pod logs indicating that the Quay TLS certificate must have SANs.
If possible, you should regenerate your TLS certificates with the correct hostname in the SANs. A possible workaround involves defining an environment variable in the
quay-config-editor pods after upgrade to enable CommonName matching:
GODEBUG=x509ignoreCN=0 flag enables the legacy behavior of treating the CommonName field on X.509 certificates as a host name when no SANs are present. However, this workaround is not recommended, as it will not persist across a redeployment.
220.127.116.11. Configuring Clair v4 when upgrading from 3.3.z or 3.4.z to 3.6 using the Quay Operator
To set up Clair v4 on a new Red Hat Quay deployment on OpenShift, it is highly recommended to use the Quay Operator. By default, the Quay Operator will install or upgrade a Clair deployment along with your Red Hat Quay deployment and configure Clair automatically.
For instructions on setting up Clair v4 on OpenShift, see Setting Up Clair on a Red Hat Quay OpenShift deployment.
13.2.3. Swift configuration when upgrading from 3.3.z to 3.6
When upgrading from Red Hat Quay 3.3.z to 3.6.z, some users might receive the following error:
Switch auth v3 requires tenant_id (string) in os_options. As a workaround, you can manually update your
DISTRIBUTED_STORAGE_CONFIG to add the
DISTRIBUTED_STORAGE_CONFIG: brscale: - SwiftStorage - auth_url: http://****/v3 auth_version: "3" os_options: tenant_id: **** project_name: ocp-base user_domain_name: Default storage_path: /datastorage/registry swift_container: ocp-svc-quay-ha swift_password: ***** swift_user: *****
13.2.4. Changing the update channel for an Operator
The subscription of an installed Operator specifies an update channel, which is used to track and receive updates for the Operator. To upgrade the Quay Operator to start tracking and receiving updates from a newer channel, change the update channel in the Subscription tab for the installed Quay Operator. For subscriptions with an
Automatic approval strategy, the upgrade begins automatically and can be monitored on the page that lists the Installed Operators.
13.2.5. Manually approving a pending Operator upgrade
If an installed Operator has the approval strategy in its subscription set to
Manual, when new updates are released in its current update channel, the update must be manually approved before installation can begin. If the Quay Operator has a pending upgrade, this status will be displayed in the list of Installed Operators. In the
Subscription tab for the Quay Operator, you can preview the install plan and review the resources that are listed as available for upgrade. If satisfied, click
Approve and return to the page that lists Installed Operators to monitor the progress of the upgrade.
The following image shows the Subscription tab in the UI, including the update
Approval strategy, the
Upgrade status and the
The list of Installed Operators provides a high-level summary of the current Quay installation:
13.3. Upgrading a QuayRegistry
When the Quay Operator starts, it immediately looks for any
QuayRegistries it can find in the namespace(s) it is configured to watch. When it finds one, the following logic is used:
status.currentVersionis unset, reconcile as normal.
status.currentVersionequals the Operator version, reconcile as normal.
status.currentVersiondoes not equal the Operator version, check if it can be upgraded. If it can, perform upgrade tasks and set the
status.currentVersionto the Operator’s version once complete. If it cannot be upgraded, return an error and leave the
QuayRegistryand its deployed Kubernetes objects alone.
13.4. Upgrading a QuayEcosystem
Upgrades are supported from previous versions of the Operator which used the
QuayEcosystem API for a limited set of configurations. To ensure that migrations do not happen unexpectedly, a special label needs to be applied to the
QuayEcosystem for it to be migrated. A new
QuayRegistry will be created for the Operator to manage, but the old
QuayEcosystem will remain until manually deleted to ensure that you can roll back and still access Quay in case anything goes wrong. To migrate an existing
QuayEcosystem to a new
QuayRegistry, follow these steps:
"quay-operator/migrate": "true"to the
$ oc edit quayecosystem <quayecosystemname>
metadata: labels: quay-operator/migrate: "true"
Wait for a
QuayRegistryto be created with the same
QuayEcosystemwill be marked with the label
status.registryEndpointof the new
QuayRegistryis set, access Quay and confirm all data and settings were migrated successfully.
When you are confident everything worked correctly, you may delete the
QuayEcosystemand Kubernetes garbage collection will clean up all old resources.
13.4.1. Reverting QuayEcosystem Upgrade
If something goes wrong during the automatic upgrade from
QuayRegistry, follow these steps to revert back to using the
QuayRegistryusing either the UI or
$ kubectl delete -n <namespace> quayregistry <quayecosystem-name>
If external access was provided using a
Route, change the
Routeto point back to the original
Serviceusing the UI or
QuayEcosystem was managing the Postgres database, the upgrade process will migrate your data to a new Postgres database managed by the upgraded Operator. Your old database will not be changed or removed but Quay will no longer use it once the migration is complete. If there are issues during the data migration, the upgrade process will exit and it is recommended that you continue with your database as an unmanaged component.
13.4.2. Supported QuayEcosystem Configurations for Upgrades
The Quay Operator will report errors in its logs and in
status.conditions if migrating a
QuayEcosystem component fails or is unsupported. All unmanaged components should migrate successfully because no Kubernetes resources need to be adopted and all the necessary values are already provided in Quay’s
Ephemeral database not supported (
volumeSize field must be set).
Nothing special needed.
Route access is supported for automatic migration. Manual migration required for other methods.
LoadBalancerwithout custom hostname: After the
QuayEcosystemis marked with label
"quay-operator/migration-complete": "true", delete the
metadata.ownerReferencesfield from existing
Servicebefore deleting the
QuayEcosystemto prevent Kubernetes from garbage collecting the
Serviceand removing the load balancer. A new
Servicewill be created with
<QuayEcosystem-name>-quay-app. Edit the
spec.selectorof the existing
Serviceto match the
spec.selectorof the new
Serviceso traffic to the old load balancer endpoint will now be directed to the new pods. You are now responsible for the old
Service; the Quay Operator will not manage it.
Ingresswith custom hostname: A new
LoadBalancerwill be created with
<QuayEcosystem-name>-quay-app. Change your DNS settings to point to the
status.loadBalancerendpoint provided by the new
Nothing special needed.
QuayEcosystem did not have a managed object storage component, so object storage will always be marked as unmanaged. Local storage is not supported.
Nothing special needed.
- For more details on the Red Hat Quay Operator, see the upstream quay-operator project.