Chapter 17. Migrating a standalone Quay deployment to a Red Hat Quay Operator managed deployment

Procedure

1. Back up the Quay config.yaml of your standalone deployment:

   $ mkdir /tmp/quay-backup
   $ cp /path/to/Quay/config/directory/config.yaml /tmp/quay-backup

2. Create a backup of the database that your standalone Quay deployment is using:

   $ pg_dump -h DB_HOST -p 5432 -d QUAY_DATABASE_NAME -U QUAY_DATABASE_USER -W -O > /tmp/quay-backup/quay-database-backup.sql

3. Install the AWS CLI if you do not have it already.

4. Create an ~/.aws/ directory:

   $ mkdir ~/.aws/

5. Obtain the access_key and secret_key from the Quay config.yaml of your standalone deployment:

   $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/config.yaml

   Example output:

   DISTRIBUTED_STORAGE_CONFIG:
       minio-1:
           - RadosGWStorage
           - access_key: ##########
             bucket_name: quay
             hostname: 172.24.10.50
             is_secure: false
             port: "9000"
             secret_key: ##########
             storage_path: /datastorage/registry

6. Store the access_key and secret_key from the Quay config.yaml file in your ~/.aws directory:

   $ touch ~/.aws/credentials

7. Optional: Check that your access_key and secret_key are stored:

   $ cat > ~/.aws/credentials << EOF
   [default]
   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
   EOF

   Example output:

   aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
   aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG

   Note

   If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can, you can configure these by running aws configure and manually inputting the credentials.

8. In your quay-backup directory, create a bucket_backup directory:

   $ mkdir /tmp/quay-backup/bucket-backup

9. Backup all blobs from the S3 storage:

   $ aws s3 sync --no-verify-ssl --endpoint-url https://PUBLIC_S3_ENDPOINT:PORT s3://QUAY_BUCKET/ /tmp/quay-backup/bucket-backup/

   Note

   The PUBLIC_S3_ENDPOINT can be read from the Quay config.yaml file under hostname in the DISTRIBUTED_STORAGE_CONFIG. If the endpoint is insecure, use http instead of https in the endpoint URL.

1. Scale down the Red Hat Quay Operator:

   $ oc scale --replicas=0 deployment quay-operator.v3.6.2 -n openshift-operators

2. Scale down the application and mirror deployments:

   $ oc scale --replicas=0 deployment QUAY_MAIN_APP_DEPLOYMENT QUAY_MIRROR_DEPLOYMENT

3. Copy the database SQL backup to the Quay PostgreSQL database instance:

   $ oc cp /tmp/user/quay-backup/quay-database-backup.sql quay-enterprise/quayregistry-quay-database-54956cdd54-p7b2w:/var/lib/pgsql/data/userdata

4. Obtain the database password from the Operator-created config.yaml file:

   $ oc get deployment quay-quay-app -o json | jq '.spec.template.spec.volumes[].projected.sources' | grep -i config-secret

   Example output:

         "name": "QUAY_CONFIG_SECRET_NAME"

   $ oc get secret quay-quay-config-secret-9t77hb84tb -o json | jq '.data."config.yaml"' | cut -d '"' -f2 | base64 -d -w0 > /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

   cat /tmp/quay-backup/operator-quay-config-yaml-backup.yaml | grep -i DB_URI

   Example output:

   postgresql://QUAY_DATABASE_OWNER:PASSWORD@DATABASE_HOST/QUAY_DATABASE_NAME

5. Execute a shell inside of the database pod:

   # oc exec -it quay-postgresql-database-pod -- /bin/bash

6. Enter psql:

   bash-4.4$ psql

7. Drop the database:

   postgres=# DROP DATABASE "example-restore-registry-quay-database";

   Example output:

   DROP DATABASE

8. Create a new database and set the owner as the same name:

   postgres=# CREATE DATABASE "example-restore-registry-quay-database" OWNER "example-restore-registry-quay-database";

   Example output:

   CREATE DATABASE

9. Connect to the database:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

10. Create a pg_trmg extension of your Quay database:

    example-restore-registry-quay-database=# create extension pg_trgm ;

    Example output:

    CREATE EXTENSION

11. Exit the postgres CLI to re-enter bash-4.4:

    \q

12. Set the password for your PostgreSQL deployment:

    bash-4.4$ psql -h localhost -d "QUAY_DATABASE_NAME" -U QUAY_DATABASE_OWNER -W < /var/lib/pgsql/data/userdata/quay-database-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

13. Exit bash mode:

    bash-4.4$ exit

14. Create a new configuration bundle for the Red Hat Quay Operator.

    $ touch config-bundle.yaml

15. In your new config-bundle.yaml, include all of the information that the registry requires, such as LDAP configuration, keys, and other modifications that your old registry had. Run the following command to move the secret_key to your config-bundle.yaml:

    $ cat /tmp/quay-backup/config.yaml | grep SECRET_KEY > /tmp/quay-backup/config-bundle.yaml

    Note

    You must manually copy all the LDAP, OIDC and other information and add it to the /tmp/quay-backup/config-bundle.yaml file.

16. Create a configuration bundle secret inside of your OpenShift cluster:

    $ oc create secret generic new-custom-config-bundle --from-file=config.yaml=/tmp/quay-backup/config-bundle.yaml

17. Scale up the Quay pods:

    $ oc scale --replicas=1 deployment quayregistry-quay-app
    deployment.apps/quayregistry-quay-app scaled

18. Scale up the mirror pods:

    $ oc scale --replicas=1  deployment quayregistry-quay-mirror
    deployment.apps/quayregistry-quay-mirror scaled

19. Patch the QuayRegistry CRD so that it contains the reference to the new custom configuration bundle:

    $ oc patch quayregistry QUAY_REGISTRY_NAME --type=merge -p '{"spec":{"configBundleSecret":"new-custom-config-bundle"}}'

    Note

    If Quay returns a 500 internal server error, you might have to update the location of your DISTRIBUTED_STORAGE_CONFIG to default.

20. Create a new AWS credentials.yaml in your /.aws/ directory and include the access_key and secret_key from the Operator-created config.yaml file:

    $ touch credentials.yaml

    $ grep -i DISTRIBUTED_STORAGE_CONFIG -A10 /tmp/quay-backup/operator-quay-config-yaml-backup.yaml

    $ cat > ~/.aws/credentials << EOF
    [default]
    aws_access_key_id = ACCESS_KEY_FROM_QUAY_CONFIG
    aws_secret_access_key = SECRET_KEY_FROM_QUAY_CONFIG
    EOF

    Note

    If the aws cli does not automatically collect the access_key and secret_key from the `~/.aws/credentials file, you can configure these by running aws configure and manually inputting the credentials.

21. Record the NooBaa’s publicly available endpoint:

    $ oc get route s3 -n openshift-storage -o yaml -o jsonpath="{.spec.host}{'\n'}"

22. Sync the backup data to the NooBaa backend storage:

    $ aws s3 sync --no-verify-ssl --endpoint-url https://NOOBAA_PUBLIC_S3_ROUTE /tmp/quay-backup/bucket-backup/* s3://QUAY_DATASTORE_BUCKET_NAME

23. Scale the Operator back up to 1 pod:

    $ oc scale –replicas=1 deployment quay-operator.v3.6.4 -n openshift-operators