Chapter 16. Backing up and restoring Red Hat Quay managed by the Red Hat Quay Operator
Use the content within this section to back up and restore Red Hat Quay when managed by the Red Hat Quay Operator on OpenShift Container Platform
16.1. Backing up Red Hat Quay
This procedure describes how to create a backup of Red Hat Quay deployed on OpenShift Container Platform using the Red Hat Quay Operator
Prerequisites
-
A healthy Red Hat Quay deployment on OpenShift Container Platform using the Red Hat Quay Operator. The status condition
Availableis set totrue. -
The components
quay,postgresandobjectstorageare set tomanaged: true -
If the component
clairis set tomanaged: truethe componentclairpostgresis also set tomanaged: true(starting with Red Hat Quay Operator v3.7 or later)
If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to create a backup of the data. You can refer to the tools described in this guide as a starting point on how to backup your external PostgreSQL database or object storage.
16.1.1. Red Hat Quay configuration backup
Use the following procedure to back up your Red Hat Quay configuration.
Procedure
To back the
QuayRegistrycustom resource by exporting it, enter the following command:$ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml
Edit the resulting
quayregistry.yamland remove the status section and the following metadata fields:metadata.creationTimestamp metadata.finalizers metadata.generation metadata.resourceVersion metadata.uid
Backup the managed keys secret by entering the following command:
NoteIf you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Red Hat Quay for the first time. These are stored in a secret called
<quay_registry_name>-quay_registry_managed_secret_keysin the namespace of theQuayRegistryresource.$ oc get secret -n <quay_namespace> <quay_registry_name>_quay_registry_managed_secret_keys -o yaml > managed_secret_keys.yaml
Edit the resulting
managed_secret_keys.yamlfile and remove the entrymetadata.ownerReferences. Yourmanaged_secret_keys.yamlfile should look similar to the following:apiVersion: v1 kind: Secret type: Opaque metadata: name: <quayname>_quay_registry_managed_secret_keys namespace: <quay_namespace> data: CONFIG_EDITOR_PW: <redacted> DATABASE_SECRET_KEY: <redacted> DB_ROOT_PW: <redacted> DB_URI: <redacted> SECRET_KEY: <redacted> SECURITY_SCANNER_V4_PSK: <redacted>
All information under the
dataproperty should remain the same.Redirect the current
Quayconfiguration file by entering the following command:$ oc get secret -n <quay-namespace> $(oc get quayregistry <quay_registry_name> -n <quay_namespace> -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yamlBackup the
/conf/stack/config.yamlfile mounted inside of theQuaypods:$ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml
16.1.2. Scaling down your Red Hat Quay deployment
Use the following procedure to scale down your Red Hat Quay deployment.
This step is needed to create a consistent backup of the state of your Red Hat Quay deployment. Do not omit this step, including in setups where PostgreSQL databases and/or S3-compatible object storage are provided by external services (unmanaged by the Red Hat Quay Operator).
Procedure
Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.
For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). Your
QuayRegistryresource should look similar to the following:apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: … - kind: horizontalpodautoscaler managed: false 1 - kind: quay managed: true overrides: 2 replicas: 0 - kind: clair managed: true overrides: replicas: 0 - kind: mirror managed: true overrides: replicas: 0 …For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay Operator first and then the managed Red Hat Quay resources:
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>
Wait for the
registry-quay-app,registry-quay-mirrorandregistry-clair-apppods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by running the following command:$ oc get pods -n <quay_namespace>
Example output:
$ oc get pod
Example output
quay-operator.v3.7.1-6f9d859bd-p5ftc 1/1 Running 0 12m quayregistry-clair-postgres-7487f5bd86-xnxpr 1/1 Running 1 (12m ago) 12m quayregistry-quay-app-upgrade-xq2v6 0/1 Completed 0 12m quayregistry-quay-config-editor-6dfdcfc44f-hlvwm 1/1 Running 0 73s quayregistry-quay-database-859d5445ff-cqthr 1/1 Running 0 12m quayregistry-quay-redis-84f888776f-hhgms 1/1 Running 0 12m
16.1.3. Backing up the Red Hat Quay managed database
Use the following procedure to back up the Red Hat Quay managed database.
If your Red Hat Quay deployment is configured with external, or unmanged, PostgreSQL database(s), refer to your vendor’s documentation on how to create a consistent backup of these databases.
Procedure
Identify the Quay PostgreSQL pod name:
$ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'Example output:
quayregistry-quay-database-59f54bb7-58xs7
Obtain the Quay database name:
$ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}' quayregistry-quay-databaseDownload a backup database:
$ oc exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database > backup.sql
16.1.3.1. Backing up the Red Hat Quay managed object storage
Use the following procedure to back up the Red Hat Quay managed object storage. The instructions in this section apply to the following configurations:
- Standalone, multi-cloud object gateway configurations
- OpenShift Data Foundations storage requires that the Red Hat Quay Operator provisioned an S3 object storage bucket from, through the ObjectStorageBucketClaim API
If your Red Hat Quay deployment is configured with external (unmanged) object storage, refer to your vendor’s documentation on how to create a copy of the content of Quay’s storage bucket.
Procedure
Decode and export the
AWS_ACCESS_KEY_IDby entering the following command:$ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)Decode and export the
AWS_SECRET_ACCESS_KEY_IDby entering the following command:$ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)Create a new directory:
$ mkdir blobs
Copy all blobs to the directory by entering the following command:
$ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs
16.1.4. Scale the Red Hat Quay deployment back up
Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.
For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your
QuayRegistryresource should look similar to the following:apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: … - kind: horizontalpodautoscaler managed: true 1 - kind: quay 2 managed: true - kind: clair managed: true - kind: mirror managed: true …For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay Operator again:
$ oc scale --replicas=1 deployment $(oc get deployment -n <quay_operator_namespace> | awk '/^quay-operator/ {print $1}') -n <quay_operator_namespace>
Check the status of the Red Hat Quay deployment by entering the following command:
$ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>
Example output:
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: registry namespace: <quay-namespace> ... spec: ... status: - lastTransitionTime: '2022-06-20T05:31:17Z' lastUpdateTime: '2022-06-20T17:31:13Z' message: All components reporting as healthy reason: HealthChecksPassing status: 'True' type: Available
16.2. Restoring Red Hat Quay
Use the following procedures to restore Red Hat Quay when the Red Hat Quay Operator manages the database. It should be performed after a backup of your Red Hat Quay registry has been performed. See Backing up Red Hat Quay for more information.
Prerequisites
- Red Hat Quay is deployed on OpenShift Container Platform using the Red Hat Quay Operator.
- A backup of the Red Hat Quay configuration managed by the Red Hat Quay Operator has been created following the instructions in the Backing up Red Hat Quay section
- Your Red Hat Quay database has been backed up.
- The object storage bucket used by Red Hat Quay has been backed up.
-
The components
quay,postgresandobjectstorageare set tomanaged: true -
If the component
clairis set tomanaged: true, the componentclairpostgresis also set tomanaged: true(starting with Red Hat Quay Operator v3.7 or later) - There is no running Red Hat Quay deployment managed by the Red Hat Quay Operator in the target namespace on your OpenShift Container Platform cluster
If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to restore their data from a backup prior to restore Red Hat Quay
16.2.1. Restoring Red Hat Quay and its configuration from a backup
Use the following procedure to restore Red Hat Quay and its configuration files from a backup.
These instructions assume you have followed the process in the Backing up Red Hat Quay guide and create the backup files with the same names.
Procedure
Restore the backed up Red Hat Quay configuration by entering the following command:
$ oc create -f ./config-bundle.yaml
ImportantIf you receive the error
Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with$ oc delete Secret config-bundle-secret -n <quay-namespace>and recreate it with$ oc create -f ./config-bundle.yaml.Restore the generated keys from the backup by entering the following command:
$ oc create -f ./managed-secret-keys.yaml
Restore the
QuayRegistrycustom resource:$ oc create -f ./quay-registry.yaml
Check the status of the Red Hat Quay deployment and wait for it to be available:
$ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
16.2.2. Scaling down your Red Hat Quay deployment
Use the following procedure to scale down your Red Hat Quay deployment.
Procedure
Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.
For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). Your
QuayRegistryresource should look similar to the following:apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: … - kind: horizontalpodautoscaler managed: false 1 - kind: quay managed: true overrides: 2 replicas: 0 - kind: clair managed: true overrides: replicas: 0 - kind: mirror managed: true overrides: replicas: 0 …For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay Operator first and then the managed Red Hat Quay resources:
$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>$ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>
Wait for the
registry-quay-app,registry-quay-mirrorandregistry-clair-apppods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command:$ oc get pods -n <quay-namespace>
Example output:
registry-quay-config-editor-77847fc4f5-nsbbv 1/1 Running 0 9m1s registry-quay-database-66969cd859-n2ssm 1/1 Running 0 6d1h registry-quay-redis-7cc5f6c977-956g8 1/1 Running 0 5d21h
16.2.3. Restoring your Red Hat Quay database
Use the following procedure to restore your Red Hat Quay database.
Procedure
Identify your
Quaydatabase pod by entering the following command:$ oc get pod -l quay-component=postgres -n <quay-namespace> -o jsonpath='{.items[0].metadata.name}'Example output:
quayregistry-quay-database-59f54bb7-58xs7
Upload the backup by copying it from the local environment and into the pod:
$ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql
Open a remote terminal to the database by entering the following command:
$ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm
Enter psql by running the following command:
bash-4.4$ psql
You can list the database by running the following command:
postgres=# \l
Example output
List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges ----------------------------+----------------------------+----------+------------+------------+----------------------- postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 | quayregistry-quay-database | quayregistry-quay-database | UTF8 | en_US.utf8 | en_US.utf8 |Drop the database by entering the following command:
postgres=# DROP DATABASE "quayregistry-quay-database";
Example output
DROP DATABASE
Exit the postgres CLI to re-enter bash-4.4:
\q
Redirect your PostgreSQL database to your backup database:
sh-4.4$ psql < /tmp/backup.sql
Exit bash by entering the following command:
sh-4.4$ exit
16.2.4. Restore your Red Hat Quay object storage data
Use the following procedure to restore your Red Hat Quay object storage data.
Procedure
Export the
AWS_ACCESS_KEY_IDby entering the following command:$ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)Export the
AWS_SECRET_ACCESS_KEYby entering the following command:$ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)Upload all blobs to the bucket by running the following command:
$ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage -o jsonpath='{.spec.host}') ./blobs s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')
16.2.5. Scaling up your Red Hat Quay deployment
Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.
For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your
QuayRegistryresource should look similar to the following:apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: name: registry namespace: ns spec: components: … - kind: horizontalpodautoscaler managed: true 1 - kind: quay 2 managed: true - kind: clair managed: true - kind: mirror managed: true …For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay Operator again:
$ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
Check the status of the Red Hat Quay deployment:
$ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>
Example output:
apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: ... name: registry namespace: <quay-namespace> ... spec: ... status: - lastTransitionTime: '2022-06-20T05:31:17Z' lastUpdateTime: '2022-06-20T17:31:13Z' message: All components reporting as healthy reason: HealthChecksPassing status: 'True' type: Available