Chapter 17. Backing up and restoring Red Hat Quay on a standalone deployment

Procedure

1. Create a temporary backup directory, for example, quay-backup:

   $ mkdir /tmp/quay-backup

2. The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install:

   $ podman run --name quay-app \
      -v /opt/quay-install/config:/conf/stack:Z \
      -v /opt/quay-install/storage:/datastorage:Z \
      registry.redhat.io/quay/quay-rhel8:v3.11.0

   Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install, by running the following command:

   $ cd /opt/quay-install

3. Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command:

   $ tar cvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

4. Back up the Quay container service by entering the following command:

   $ podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' -
   
     /usr/bin/podman run --name quay-app \
     -v /opt/quay-install/config:/conf/stack:Z \
     -v /opt/quay-install/storage:/datastorage:Z \
     registry.redhat.io/quay/quay-rhel8:v3.11.0

5. Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command:

   $ podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml

6. Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command:

   $ grep DB_URI /tmp/quay-backup/quay-config.yaml

   Example output:

   $ postgresql://<username>:test123@172.24.10.50/quay

7. Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command:

   $ pg_dump -h 172.24.10.50  -p 5432 -d quay  -U  <username>   -W -O > /tmp/quay-backup/quay-backup.sql

8. Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command:

   DISTRIBUTED_STORAGE_CONFIG:
      default:
       - S3Storage
       - s3_bucket: <bucket_name>
         storage_path: /registry
         s3_access_key: <s3_access_key>
         s3_secret_key: <s3_secret_key>
         host: <host_name>

9. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7:

   $ export AWS_ACCESS_KEY_ID=<access_key>

10. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

11. Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG:

    $ aws s3 sync s3://<bucket_name>  /tmp/quay-backup/blob-backup/ --source-region us-east-2

    Example output:

    download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a
    download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d

Procedure

1. Create a new directory that will bind-mount to /conf/stack inside of the Red Hat Quay container:

   $ mkdir /opt/new-quay-install

2. Copy the contents of your temporary backup directory created in Backing up Red Hat Quay on standalone deployments to the new-quay-install1 directory created in Step 1:

   $ cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/

3. Change into the new-quay-install directory by entering the following command:

   $ cd /opt/new-quay-install/

4. Extract the contents of your Red Hat Quay directory:

   $ tar xvf /tmp/quay-backup/quay-backup.tar.gz *

   Example output:

   config.yaml
   config.yaml.bak
   extra_ca_certs/
   extra_ca_certs/ca.crt
   ssl.cert
   ssl.key

5. Recall the DB_URI from your backed-up config.yaml file by entering the following command:

   $ grep DB_URI config.yaml

   Example output:

   postgresql://<username>:test123@172.24.10.50/quay

6. Run the following command to enter the PostgreSQL database server:

   $ sudo postgres

7. Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database, by entering the following command:

   $ psql "host=172.24.10.50  port=5432 dbname=postgres user=<username>  password=test123"
   postgres=> CREATE DATABASE example_restore_registry_quay_database;

   Example output:

   CREATE DATABASE

8. Connect to the database by running the following command:

   postgres=# \c "example-restore-registry-quay-database";

   Example output:

   You are now connected to database "example-restore-registry-quay-database" as user "postgres".

9. Create a pg_trmg extension of your Quay database by running the following command:

   example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;

   Example output:

   CREATE EXTENSION

10. Exit the postgres CLI by entering the following command:

    \q

11. Import the database backup to your new database by running the following command:

    $ psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123"  -W <  /tmp/quay-backup/quay-backup.sql

    Example output:

    SET
    SET
    SET
    SET
    SET

    Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment.

    Note

    The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database>. If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name>, <login_user_password> and <postgresql_host> at the same time.

12. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle:

    $ cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10

    Example output:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>

    Note

    Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment.

13. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13:

    $ export AWS_ACCESS_KEY_ID=<access_key>

14. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13:

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

15. Create a new s3 bucket by entering the following command:

    $ aws s3 mb s3://<new_bucket_name>  --region us-east-2

    Example output:

    $ make_bucket: quay

16. Upload all blobs to the new s3 bucket by entering the following command:

    $ aws s3 sync --no-verify-ssl \
    --endpoint-url <example_endpoint_url> 1
    /tmp/quay-backup/blob-backup/. s3://quay/

    1

    The Red Hat Quay registry endpoint must be the same before backup and after restore.

    Example output:

    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0
    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
    ...

17. Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml:

    DISTRIBUTED_STORAGE_CONFIG:
       default:
    DISTRIBUTED_STORAGE_CONFIG:
       default:
        - S3Storage
        - s3_bucket: <new_bucket_name>
          storage_path: /registry
          s3_access_key: <s3_access_key>
          s3_secret_key: <s3_secret_key>
          host: <host_name>