Chapter 8. Backing up and restoring Red Hat Quay managed by the Red Hat Quay Operator

Use the content within this section to back up and restore Red Hat Quay when managed by the Red Hat Quay Operator on OpenShift Container Platform

8.1. Backing up Red Hat Quay

This procedure describes how to create a backup of Red Hat Quay deployed on OpenShift Container Platform using the Red Hat Quay Operator

Prerequisites

  • A healthy Red Hat Quay deployment on OpenShift Container Platform using the Red Hat Quay Operator. The status condition Available is set to true.
  • The components quay, postgres and objectstorage are set to managed: true
  • If the component clair is set to managed: true the component clairpostgres is also set to managed: true (starting with Red Hat Quay Operator v3.7 or later)
Note

If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to create a backup of the data. You can refer to the tools described in this guide as a starting point on how to backup your external PostgreSQL database or object storage.

8.1.1. Red Hat Quay configuration backup

Use the following procedure to back up your Red Hat Quay configuration.

Procedure

  1. To back the QuayRegistry custom resource by exporting it, enter the following command:

    $ oc get quayregistry <quay_registry_name> -n <quay_namespace> -o yaml > quay-registry.yaml
  2. Edit the resulting quayregistry.yaml and remove the status section and the following metadata fields:

      metadata.creationTimestamp
      metadata.finalizers
      metadata.generation
      metadata.resourceVersion
      metadata.uid
  3. Backup the managed keys secret by entering the following command:

    Note

    If you are running a version older than Red Hat Quay 3.7.0, this step can be skipped. Some secrets are automatically generated while deploying Red Hat Quay for the first time. These are stored in a secret called <quay_registry_name>-quay_registry_managed_secret_keys in the namespace of the QuayRegistry resource.

    $ oc get secret -n <quay_namespace> <quay_registry_name>_quay_registry_managed_secret_keys -o yaml > managed_secret_keys.yaml
  4. Edit the resulting managed_secret_keys.yaml file and remove the entry metadata.ownerReferences. Your managed_secret_keys.yaml file should look similar to the following:

    apiVersion: v1
    kind: Secret
    type: Opaque
    metadata:
      name: <quayname>_quay_registry_managed_secret_keys
      namespace: <quay_namespace>
    data:
      CONFIG_EDITOR_PW: <redacted>
      DATABASE_SECRET_KEY: <redacted>
      DB_ROOT_PW: <redacted>
      DB_URI: <redacted>
      SECRET_KEY: <redacted>
      SECURITY_SCANNER_V4_PSK: <redacted>

    All information under the data property should remain the same.

  5. Redirect the current Quay configuration file by entering the following command:

    $ oc get secret -n <quay-namespace>  $(oc get quayregistry <quay_registry_name> -n <quay_namespace>  -o jsonpath='{.spec.configBundleSecret}') -o yaml > config-bundle.yaml
  6. Backup the /conf/stack/config.yaml file mounted inside of the Quay pods:

    $ oc exec -it quay_pod_name -- cat /conf/stack/config.yaml > quay_config.yaml

8.1.2. Scaling down your Red Hat Quay deployment

Use the following procedure to scale down your Red Hat Quay deployment.

Important

This step is needed to create a consistent backup of the state of your Red Hat Quay deployment. Do not omit this step, including in setups where PostgreSQL databases and/or S3-compatible object storage are provided by external services (unmanaged by the Red Hat Quay Operator).

Procedure

  1. Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.

    1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Red Hat Quay, mirror workers, and Clair (if managed). Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: false 1
          - kind: quay
            managed: true
            overrides: 2
              replicas: 0
          - kind: clair
            managed: true
            overrides:
              replicas: 0
          - kind: mirror
            managed: true
            overrides:
              replicas: 0
          …
      1
      Disable auto scaling of Quay, Clair and Mirroring workers
      2
      Set the replica count to 0 for components accessing the database and objectstorage
    2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay Operator first and then the managed Red Hat Quay resources:

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>
      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>
      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>
  2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by the Red Hat Quay Operator) to disappear. You can check their status by running the following command:

    $ oc get pods -n <quay_namespace>

    Example output:

    $ oc get pod

    Example output

    quay-operator.v3.7.1-6f9d859bd-p5ftc               1/1     Running     0             12m
    quayregistry-clair-postgres-7487f5bd86-xnxpr       1/1     Running     1 (12m ago)   12m
    quayregistry-quay-app-upgrade-xq2v6                0/1     Completed   0             12m
    quayregistry-quay-config-editor-6dfdcfc44f-hlvwm   1/1     Running     0             73s
    quayregistry-quay-database-859d5445ff-cqthr        1/1     Running     0             12m
    quayregistry-quay-redis-84f888776f-hhgms           1/1     Running     0             12m

8.1.3. Backing up the Red Hat Quay managed database

Use the following procedure to back up the Red Hat Quay managed database.

Note

If your Red Hat Quay deployment is configured with external, or unmanged, PostgreSQL database(s), refer to your vendor’s documentation on how to create a consistent backup of these databases.

Procedure

  1. Identify the Quay PostgreSQL pod name:

    $ oc get pod -l quay-component=postgres -n <quay_namespace> -o jsonpath='{.items[0].metadata.name}'

    Example output:

    quayregistry-quay-database-59f54bb7-58xs7
  2. Obtain the Quay database name:

    $ oc -n <quay_namespace> rsh $(oc get pod -l app=quay -o NAME -n <quay_namespace> |head -n 1) cat /conf/stack/config.yaml|awk -F"/" '/^DB_URI/ {print $4}'
    quayregistry-quay-database
  3. Download a backup database:

    $ oc exec quayregistry-quay-database-59f54bb7-58xs7 -- /usr/bin/pg_dump -C quayregistry-quay-database  > backup.sql

8.1.3.1. Backing up the Red Hat Quay managed object storage

Use the following procedure to back up the Red Hat Quay managed object storage. The instructions in this section apply to the following configurations:

  • Standalone, multi-cloud object gateway configurations
  • OpenShift Data Foundations storage requires that the Red Hat Quay Operator provisioned an S3 object storage bucket from, through the ObjectStorageBucketClaim API
Note

If your Red Hat Quay deployment is configured with external (unmanged) object storage, refer to your vendor’s documentation on how to create a copy of the content of Quay’s storage bucket.

Procedure

  1. Decode and export the AWS_ACCESS_KEY_ID by entering the following command:

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
  2. Decode and export the AWS_SECRET_ACCESS_KEY_ID by entering the following command:

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
  3. Create a new directory:

    $ mkdir blobs
Note

You can also use rclone or sc3md instead of the AWS command line utility.

  1. Copy all blobs to the directory by entering the following command:

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}')  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}') ./blobs

8.1.4. Scale the Red Hat Quay deployment back up

  1. Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.

    1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: true 1
          - kind: quay 2
            managed: true
          - kind: clair
            managed: true
          - kind: mirror
            managed: true
          …
      1
      Re-enables auto scaling of Quay, Clair and Mirroring workers again (if desired)
      2
      Replica overrides are removed again to scale the Quay components back up
    2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay Operator again:

      $ oc scale --replicas=1 deployment $(oc get deployment -n <quay_operator_namespace> | awk '/^quay-operator/ {print $1}') -n <quay_operator_namespace>
  2. Check the status of the Red Hat Quay deployment by entering the following command:

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay_namespace>

    Example output:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      ...
      name: registry
      namespace: <quay-namespace>
      ...
    spec:
      ...
    status:
      - lastTransitionTime: '2022-06-20T05:31:17Z'
        lastUpdateTime: '2022-06-20T17:31:13Z'
        message: All components reporting as healthy
        reason: HealthChecksPassing
        status: 'True'
        type: Available

8.2. Restoring Red Hat Quay

Use the following procedures to restore Red Hat Quay when the Red Hat Quay Operator manages the database. It should be performed after a backup of your Red Hat Quay registry has been performed. See Backing up Red Hat Quay for more information.

Prerequisites

  • Red Hat Quay is deployed on OpenShift Container Platform using the Red Hat Quay Operator.
  • A backup of the Red Hat Quay configuration managed by the Red Hat Quay Operator has been created following the instructions in the Backing up Red Hat Quay section
  • Your Red Hat Quay database has been backed up.
  • The object storage bucket used by Red Hat Quay has been backed up.
  • The components quay, postgres and objectstorage are set to managed: true
  • If the component clair is set to managed: true, the component clairpostgres is also set to managed: true (starting with Red Hat Quay Operator v3.7 or later)
  • There is no running Red Hat Quay deployment managed by the Red Hat Quay Operator in the target namespace on your OpenShift Container Platform cluster
Note

If your deployment contains partially unmanaged database or storage components and you are using external services for PostgreSQL or S3-compatible object storage to run your Red Hat Quay deployment, you must refer to the service provider or vendor documentation to restore their data from a backup prior to restore Red Hat Quay

8.2.1. Restoring Red Hat Quay and its configuration from a backup

Use the following procedure to restore Red Hat Quay and its configuration files from a backup.

Note

These instructions assume you have followed the process in the Backing up Red Hat Quay guide and create the backup files with the same names.

Procedure

  1. Restore the backed up Red Hat Quay configuration by entering the following command:

    $ oc create -f ./config-bundle.yaml
    Important

    If you receive the error Error from server (AlreadyExists): error when creating "./config-bundle.yaml": secrets "config-bundle-secret" already exists, you must delete your existing resource with $ oc delete Secret config-bundle-secret -n <quay-namespace> and recreate it with $ oc create -f ./config-bundle.yaml.

  2. Restore the generated keys from the backup by entering the following command:

    $ oc create -f ./managed-secret-keys.yaml
  3. Restore the QuayRegistry custom resource:

    $ oc create -f ./quay-registry.yaml
  4. Check the status of the Red Hat Quay deployment and wait for it to be available:

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

8.2.2. Scaling down your Red Hat Quay deployment

Use the following procedure to scale down your Red Hat Quay deployment.

Procedure

  1. Depending on the version of your Red Hat Quay deployment, scale down your deployment using one of the following options.

    1. For Operator version 3.7 and newer: Scale down the Red Hat Quay deployment by disabling auto scaling and overriding the replica count for Quay, mirror workers and Clair (if managed). Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: false 1
          - kind: quay
            managed: true
            overrides: 2
              replicas: 0
          - kind: clair
            managed: true
            overrides:
              replicas: 0
          - kind: mirror
            managed: true
            overrides:
              replicas: 0
          …
      1
      Disable auto scaling of Quay, Clair and Mirroring workers
      2
      Set the replica count to 0 for components accessing the database and objectstorage
    2. For Operator version 3.6 and earlier: Scale down the Red Hat Quay deployment by scaling down the Red Hat Quay Operator first and then the managed Red Hat Quay resources:

      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-operator-namespace>|awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-app/ {print $1}') -n <quay-namespace>
      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/quay-mirror/ {print $1}') -n <quay-namespace>
      $ oc scale --replicas=0 deployment $(oc get deployment -n <quay-namespace>|awk '/clair-app/ {print $1}') -n <quay-namespace>
  2. Wait for the registry-quay-app, registry-quay-mirror and registry-clair-app pods (depending on which components you set to be managed by Red Hat Quay Operator) to disappear. You can check their status by running the following command:

    $ oc get pods -n <quay-namespace>

    Example output:

    registry-quay-config-editor-77847fc4f5-nsbbv   1/1     Running            0          9m1s
    registry-quay-database-66969cd859-n2ssm        1/1     Running            0          6d1h
    registry-quay-redis-7cc5f6c977-956g8           1/1     Running            0          5d21h

8.2.3. Restoring your Red Hat Quay database

Use the following procedure to restore your Red Hat Quay database.

Procedure

  1. Identify your Quay database pod by entering the following command:

    $ oc get pod -l quay-component=postgres -n  <quay-namespace> -o jsonpath='{.items[0].metadata.name}'

    Example output:

    quayregistry-quay-database-59f54bb7-58xs7
  2. Upload the backup by copying it from the local environment and into the pod:

    $ oc cp ./backup.sql -n <quay-namespace> registry-quay-database-66969cd859-n2ssm:/tmp/backup.sql
  3. Open a remote terminal to the database by entering the following command:

    $ oc rsh -n <quay-namespace> registry-quay-database-66969cd859-n2ssm
  4. Enter psql by running the following command:

    bash-4.4$ psql
  5. You can list the database by running the following command:

    postgres=# \l

    Example output

                                                      List of databases
               Name            |           Owner            | Encoding |  Collate   |   Ctype    |   Access privileges
    ----------------------------+----------------------------+----------+------------+------------+-----------------------
    postgres                   | postgres                   | UTF8     | en_US.utf8 | en_US.utf8 |
    quayregistry-quay-database | quayregistry-quay-database | UTF8     | en_US.utf8 | en_US.utf8 |

  6. Drop the database by entering the following command:

    postgres=# DROP DATABASE "quayregistry-quay-database";

    Example output

    DROP DATABASE

  7. Exit the postgres CLI to re-enter bash-4.4:

    \q
  8. Redirect your PostgreSQL database to your backup database:

    sh-4.4$ psql < /tmp/backup.sql
  9. Exit bash by entering the following command:

    sh-4.4$ exit

8.2.4. Restore your Red Hat Quay object storage data

Use the following procedure to restore your Red Hat Quay object storage data.

Procedure

  1. Export the AWS_ACCESS_KEY_ID by entering the following command:

    $ export AWS_ACCESS_KEY_ID=$(oc get secret -l app=noobaa -n <quay-namespace>  -o jsonpath='{.items[0].data.AWS_ACCESS_KEY_ID}' |base64 -d)
  2. Export the AWS_SECRET_ACCESS_KEY by entering the following command:

    $ export AWS_SECRET_ACCESS_KEY=$(oc get secret -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.AWS_SECRET_ACCESS_KEY}' |base64 -d)
  3. Upload all blobs to the bucket by running the following command:

    $ aws s3 sync --no-verify-ssl --endpoint https://$(oc get route s3 -n openshift-storage  -o jsonpath='{.spec.host}') ./blobs  s3://$(oc get cm -l app=noobaa -n <quay-namespace> -o jsonpath='{.items[0].data.BUCKET_NAME}')
Note

You can also use rclone or sc3md instead of the AWS command line utility.

8.2.5. Scaling up your Red Hat Quay deployment

  1. Depending on the version of your Red Hat Quay deployment, scale up your deployment using one of the following options.

    1. For Operator version 3.7 and newer: Scale up the Red Hat Quay deployment by re-enabling auto scaling, if desired, and removing the replica overrides for Quay, mirror workers and Clair as applicable. Your QuayRegistry resource should look similar to the following:

      apiVersion: quay.redhat.com/v1
      kind: QuayRegistry
      metadata:
        name: registry
        namespace: ns
      spec:
        components:
          …
          - kind: horizontalpodautoscaler
            managed: true 1
          - kind: quay 2
            managed: true
          - kind: clair
            managed: true
          - kind: mirror
            managed: true
          …
      1
      Re-enables auto scaling of Red Hat Quay, Clair and mirroring workers again (if desired)
      2
      Replica overrides are removed again to scale the Red Hat Quay components back up
    2. For Operator version 3.6 and earlier: Scale up the Red Hat Quay deployment by scaling up the Red Hat Quay Operator again:

      $ oc scale --replicas=1 deployment $(oc get deployment -n <quay-operator-namespace> | awk '/^quay-operator/ {print $1}') -n <quay-operator-namespace>
  2. Check the status of the Red Hat Quay deployment:

    $ oc wait quayregistry registry --for=condition=Available=true -n <quay-namespace>

    Example output:

    apiVersion: quay.redhat.com/v1
    kind: QuayRegistry
    metadata:
      ...
      name: registry
      namespace: <quay-namespace>
      ...
    spec:
      ...
    status:
      - lastTransitionTime: '2022-06-20T05:31:17Z'
        lastUpdateTime: '2022-06-20T17:31:13Z'
        message: All components reporting as healthy
        reason: HealthChecksPassing
        status: 'True'
        type: Available