Chapter 6. 3scale backup and restore using templates

This section provides you, as the administrator of a Red Hat 3scale API Management installation, the information needed to:

  • Set up the backup procedures for persistent data.
  • Perform a restore from backup of the persistent data.

In case of issues with one or more of the MySQL databases, you will be able to restore 3scale correctly to its previous operational state.

6.1. Prerequisites

  • A 3scale 2.10 instance. For more information about how to install 3scale, see Installing 3scale on OpenShift.
  • jq: For extraction or transformation of JSON data.
  • An OpenShift Container Platform 4.x user account with one of the following roles in the OpenShift cluster:

    • cluster-admin
    • admin
    • edit
Note

A user with an edit cluster role locally binded in the namespace of a 3scale installation can perform backup and restore procedures.

The following sections contain information about persistent volumes, using data sets, setting up the backup procedures for persistent data, as well as restoring system databases and OpenShift secrets:

6.2. Persistent volumes and considerations

Persistent volumes

In a 3scale deployment on OpenShift:

  • A persistent volume (PV) provided to the cluster by the underlying infrastructure.
  • Storage service external to the cluster. This can be in the same data center or elsewhere.

Considerations

The backup and restore procedures for persistent data vary depending on the storage type in use. To ensure the backups and restores preserve data consistency, it is not sufficient to backup the underlying PVs for a database. For example, do not capture only partial writes and partial transactions. Use the database’s backup mechanisms instead.

Some parts of the data are synchronized between different components. One copy is considered the source of truth for the data set. The other is a copy that is not modified locally, but synchronized from the source of truth. In these cases, upon completion, the source of truth should be restored, and copies in other components synchronized from it.

6.3. Using data sets

This section explains in more detail about different data sets in the different persistent stores, their purpose, the storage type used, and whether or not it is the source of truth.

The full state of a 3scale deployment is stored across the following DeploymentConfig objects and their PVs:

NameDescription

system-mysql

MySQL database (mysql-storage)

system-storage

Volume for Files

zync-database

  • Postgres database for zync component
  • This uses HostPath as storage
  • If the pod is moved into another node the data is lost
  • The data are sync jobs and do not need to be 100% persistent.

backend-redis

Redis database (backend-redis-storage)

system-redis

Redis database (system-redis-storage)

6.3.1. Defining system-mysql

system-mysql is a relational database which stores information about users, accounts, APIs, plans, and more, in the 3scale Admin Console.

A subset of this information related to services is synchronized to the Backend component and stored in backend-redis. system-mysql is the source of truth for this information.

6.3.2. Defining system-storage

system-storage stores files to be read and written by the System component.

They fall into two categories:

  • Configuration files read by the System component at run-time
  • Static files, for example, HTML, CSS, JS, uploaded to system by its CMS feature, for the purpose of creating a Developer Portal
Note

System can be scaled horizontally with multiple pods uploading and reading said static files, hence the need for a ReadWriteMany (RWX) PersistentVolume.

6.3.3. Defining zync-database

A zync-database is a relational database which stores information related to the synchronization of identities between 3scale and an Identity provider (IdP). This information is not duplicated in other components and is the sole source of truth.

6.3.4. Defining backend-redis

backend-redis contains multiple data sets used by the Backend component:

  • Usages: This is API usage information aggregated by Backend. It is used by Backend for rate-limiting decisions and by System to display analytics information in the UI or via API.
  • Config: This is configuration information about services, rate-limits, and more, that is synchronized from System via an internal API. This is not the source of truth of this information, however System and system-mysql is.
  • AuthKeys: Storage of OAuth keys created directly in Backend. This is the source of truth for this information.
  • Queues: This is queues of background jobs to be executed by worker processes. These are ephemeral and are deleted once processed.

6.3.5. Defining system-redis

system-redis contains queues for jobs to be processed in background. These are ephemeral and are deleted once processed.

6.4. Backing up system databases

The following commands are in no specific order and can be used as you need them to back up and archive system databases.

6.4.1. Backing up system-mysql

Execute MySQL Backup Command:

oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysqldump --single-transaction -hsystem-mysql -uroot system' | gzip > system-mysql-backup.gz

6.4.2. Backing up system-storage

Archive the system-storage files to another storage:

oc rsync $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system ./local/dir

6.4.3. Backing up zync-database

Instructions to create a backup of zync-database depend on the type of deployment you have configured for 3scale.

Template-based deployments

  1. Scale down the zync DeploymentConfig to 0 pods:

    oc scale dc zync --replicas=0
    oc scale dc zync-que --replicas=0
  2. Execute Postgres backup command:

     oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r) bash -c 'pg_dumpall -c --if-exists' | gzip > zync-database-backup.gz
  3. Restore to the original count of replicas, by replacing ${ZYNC_REPLICAS} with the number of replicas, in the command below :

    oc scale dc zync --replicas=${ZYNC_REPLICAS}
    oc scale dc zync-que --replicas=${ZYNC_REPLICAS}

Operator-based deployments

  1. Get the number of replicas, by replacing in the command below ${DEPLOYMENT_NAME} with the name you defined when you created your 3scale deployment:

    ZYNC_SPEC=`oc get APIManager/${DEPLOYMENT_NAME} -o json | jq -r '.spec.zync'`
  2. Scale down the zync DeploymentConfig to 0 pods:

    oc patch APIManager/${DEPLOYMENT_NAME} --type merge -p '{"spec": {"zync": {"appSpec": {"replicas": 0}, "queSpec": {"replicas": 0}}}}'
  3. Execute Postgres backup command:

     oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r) bash -c 'pg_dumpall -c --if-exists' | gzip > zync-database-backup.gz
  4. Restore to the original count of replicas:

    oc patch APIManager ${DEPLOYMENT_NAME} --type merge -p '{"spec": {"zync":'"${ZYNC_SPEC}"'}}'

6.4.4. Backing up backend-redis

Backup the dump.rdb file from redis:

oc cp $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./backend-redis-dump.rdb

6.4.5. Backing up system-redis

Backup the dump.rdb file from redis:

oc cp $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb ./system-redis-dump.rdb

6.4.6. Backing up OpenShift secrets and ConfigMaps

The following is the list of commands for OpenShift secrets and ConfigMaps:

6.4.6.1. OpenShift secrets

oc get secrets system-smtp -o json --export > system-smtp.json
oc get secrets system-seed -o json --export > system-seed.json
oc get secrets system-database -o json --export > system-database.json
oc get secrets backend-internal-api -o json --export > backend-internal-api.json
oc get secrets system-events-hook -o json --export > system-events-hook.json
oc get secrets system-app -o json --export > system-app.json
oc get secrets system-recaptcha -o json --export > system-recaptcha.json
oc get secrets system-redis -o json --export > system-redis.json
oc get secrets zync -o json --export > zync.json
oc get secrets system-master-apicast -o json --export > system-master-apicast.json

6.4.6.2. ConfigMaps

oc get configmaps system-environment -o json --export > system-environment.json
oc get configmaps apicast-environment -o json --export > apicast-environment.json

6.5. Restoring system databases

Use the following procedures to restore OpenShift secrets and system databases:

6.5.1. Restoring a template-based deployment

Use the following steps to restore a template-based deployment.

Procedure

  1. Restore secrets before creating deploying template.:
oc apply -f system-smtp.json
  1. Template parameters will be read from copied secrets and configmaps:

    oc new-app --file /opt/amp/templates/amp.yml \
        --param APP_LABEL=$(cat system-environment.json | jq -r '.metadata.labels.app') \
        --param TENANT_NAME=$(cat system-seed.json | jq -r '.data.TENANT_NAME' | base64 -d) \
        --param SYSTEM_DATABASE_USER=$(cat system-database.json | jq -r '.data.DB_USER' | base64 -d) \
        --param SYSTEM_DATABASE_PASSWORD=$(cat system-database.json | jq -r '.data.DB_PASSWORD' | base64 -d) \
        --param SYSTEM_DATABASE=$(cat system-database.json | jq -r '.data.URL' | base64 -d | cut -d '/' -f4) \
        --param SYSTEM_DATABASE_ROOT_PASSWORD=$(cat system-database.json | jq -r '.data.URL' | base64 -d | awk -F '[:@]' '{print $3}') \
        --param WILDCARD_DOMAIN=$(cat system-environment.json | jq -r '.data.THREESCALE_SUPERDOMAIN') \
        --param SYSTEM_BACKEND_USERNAME=$(cat backend-internal-api.json | jq '.data.username' -r | base64 -d) \
        --param SYSTEM_BACKEND_PASSWORD=$(cat backend-internal-api.json | jq '.data.password' -r | base64 -d) \
        --param SYSTEM_BACKEND_SHARED_SECRET=$(cat system-events-hook.json | jq -r '.data.PASSWORD' | base64 -d) \
        --param SYSTEM_APP_SECRET_KEY_BASE=$(cat system-app.json | jq -r '.data.SECRET_KEY_BASE' | base64 -d) \
        --param ADMIN_PASSWORD=$(cat system-seed.json | jq -r '.data.ADMIN_PASSWORD' | base64 -d) \
        --param ADMIN_USERNAME=$(cat system-seed.json | jq -r '.data.ADMIN_USER' | base64 -d) \
        --param ADMIN_EMAIL=$(cat system-seed.json | jq -r '.data.ADMIN_EMAIL' | base64 -d) \
        --param ADMIN_ACCESS_TOKEN=$(cat system-seed.json | jq -r '.data.ADMIN_ACCESS_TOKEN' | base64 -d) \
        --param MASTER_NAME=$(cat system-seed.json | jq -r '.data.MASTER_DOMAIN' | base64 -d) \
        --param MASTER_USER=$(cat system-seed.json | jq -r '.data.MASTER_USER' | base64 -d) \
        --param MASTER_PASSWORD=$(cat system-seed.json | jq -r '.data.MASTER_PASSWORD' | base64 -d) \
        --param MASTER_ACCESS_TOKEN=$(cat system-seed.json | jq -r '.data.MASTER_ACCESS_TOKEN' | base64 -d) \
        --param RECAPTCHA_PUBLIC_KEY="$(cat system-recaptcha.json | jq -r '.data.PUBLIC_KEY' | base64 -d)" \
        --param RECAPTCHA_PRIVATE_KEY="$(cat system-recaptcha.json | jq -r '.data.PRIVATE_KEY' | base64 -d)" \
        --param SYSTEM_REDIS_URL=$(cat system-redis.json | jq -r '.data.URL' | base64 -d) \
        --param SYSTEM_MESSAGE_BUS_REDIS_URL="$(cat system-redis.json | jq -r '.data.MESSAGE_BUS_URL' | base64 -d)" \
        --param SYSTEM_REDIS_NAMESPACE="$(cat system-redis.json | jq -r '.data.NAMESPACE' | base64 -d)" \
        --param SYSTEM_MESSAGE_BUS_REDIS_NAMESPACE="$(cat system-redis.json | jq -r '.data.MESSAGE_BUS_NAMESPACE' | base64 -d)" \
        --param ZYNC_DATABASE_PASSWORD=$(cat zync.json | jq -r '.data.ZYNC_DATABASE_PASSWORD' | base64 -d) \
        --param ZYNC_SECRET_KEY_BASE=$(cat zync.json | jq -r '.data.SECRET_KEY_BASE' | base64 -d) \
        --param ZYNC_AUTHENTICATION_TOKEN=$(cat zync.json | jq -r '.data.ZYNC_AUTHENTICATION_TOKEN' | base64 -d) \
        --param APICAST_ACCESS_TOKEN=$(cat system-master-apicast.json | jq -r '.data.ACCESS_TOKEN' | base64 -d) \
        --param APICAST_MANAGEMENT_API=$(cat apicast-environment.json | jq -r '.data.APICAST_MANAGEMENT_API') \
        --param APICAST_OPENSSL_VERIFY=$(cat apicast-environment.json | jq -r '.data.OPENSSL_VERIFY') \
        --param APICAST_RESPONSE_CODES=$(cat apicast-environment.json | jq -r '.data.APICAST_RESPONSE_CODES') \
        --param APICAST_REGISTRY_URL=$(cat system-environment.json | jq -r '.data.APICAST_REGISTRY_URL')

6.5.2. Restoring an operator-based deployment

Use the following steps to restore operator-based deployments.

Procedure

  1. Restore secrets before creating an APIManager resource:

    oc apply -f system-smtp.json
    oc apply -f system-seed.json
    oc apply -f system-database.json
    oc apply -f backend-internal-api.json
    oc apply -f system-events-hook.json
    oc apply -f system-app.json
    oc apply -f system-recaptcha.json
    oc apply -f system-redis.json
    oc apply -f zync.json
    oc apply -f system-master-apicast.json
  2. Restore ConfigMaps before creating an APIManager resource:

    oc apply -f system-environment.json
    oc apply -f apicast-environment.json

6.5.3. Restoring system-mysql

Procedure

  1. Copy the MySQL dump to the system-mysql pod:

    oc cp ./system-mysql-backup.gz $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq '.items[0].metadata.name' -r):/var/lib/mysql
  2. Decompress the backup file:

    oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/system-mysql-backup.gz'
  3. Restore the MySQL DB Backup file:

    oc rsh $(oc get pods -l 'deploymentConfig=system-mysql' -o json | jq -r '.items[0].metadata.name') bash -c 'export MYSQL_PWD=${MYSQL_ROOT_PASSWORD}; mysql -hsystem-mysql -uroot system < ${HOME}/system-mysql-backup'

6.5.4. Restoring system-storage

Restore the Backup file to system-storage:

oc rsync ./local/dir/system/ $(oc get pods -l 'deploymentConfig=system-app' -o json | jq '.items[0].metadata.name' -r):/opt/system/public/system

6.5.5. Restoring zync-database

Instructions to restore zync-database depend on the deployment type applied for 3scale.

6.5.5.1. Template-based deployments

Procedure

  1. Scale down the zync DeploymentConfig to 0 pods:

    oc scale dc zync --replicas=0
    oc scale dc zync-que --replicas=0
  2. Copy the Zync database dump to the zync-database pod:

    oc cp ./zync-database-backup.gz $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r):/var/lib/pgsql/
  3. Decompress the backup file:

    oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/zync-database-backup.gz'
  4. Restore the PostgreSQL DB backup file:

    oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'psql -f ${HOME}/zync-database-backup'
  5. Restore to the original count of replicas, by replacing ${ZYNC_REPLICAS} with the number of replicas, in the commands below:

    oc scale dc zync --replicas=${ZYNC_REPLICAS}
    oc scale dc zync-que --replicas=${ZYNC_REPLICAS}

6.5.5.2. Operator-based deployments

Procedure

  1. Store the number of replicas, by replacing ${DEPLOYMENT_NAME} with the name you defined when you created your 3scale deployment:

    ZYNC_SPEC=`oc get APIManager/${DEPLOYMENT_NAME} -o json | jq -r '.spec.zync'`
  2. Scale down the zync DeploymentConfig to 0 pods:

    oc patch APIManager/${DEPLOYMENT_NAME} --type merge -p '{"spec": {"zync": {"appSpec": {"replicas": 0}, "queSpec": {"replicas": 0}}}}'
  3. Copy the Zync database dump to the zync-database pod:

    oc cp ./zync-database-backup.gz $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq '.items[0].metadata.name' -r):/var/lib/pgsql/
  4. Decompress the backup file:

    oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'gzip -d ${HOME}/zync-database-backup.gz'
  5. Restore the PostgreSQL DB backup file:

    oc rsh $(oc get pods -l 'deploymentConfig=zync-database' -o json | jq -r '.items[0].metadata.name') bash -c 'psql -f ${HOME}/zync-database-backup'
  6. Restore to the original count of replicas:

    oc patch APIManager ${DEPLOYMENT_NAME} --type merge -p '{"spec": {"zync":'"${ZYNC_SPEC}"'}}'

6.5.6. Ensuring information consistency between Backend and System

After restoring backend-redis a sync of the Config information from System should be forced to ensure the information in Backend is consistent with that in System, which is the source of truth.

6.5.6.1. Managing the deployment configuration for backend-redis

These steps are intended for running instances of backend-redis.

Procedure

  1. Edit the redis-config configmap:

    oc edit configmap redis-config
  2. Comment SAVE commands in the redis-config configmap:

     #save 900 1
     #save 300 10
     #save 60 10000
  3. Set appendonly to no in the redis-config configmap:

    appendonly no
  4. Redeploy backend-redis to load the new configurations:

    oc rollout latest dc/backend-redis
  5. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/backend-redis
  6. Rename the dump.rdb file:

    oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/dump.rdb ${HOME}/data/dump.rdb-old'
  7. Rename the appendonly.aof file:

    oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/appendonly.aof ${HOME}/data/appendonly.aof-old'
  8. Move the backup file to the POD:

    oc cp ./backend-redis-dump.rdb $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb
  9. Redeploy backend-redis to load the backup:

    oc rollout latest dc/backend-redis
  10. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/backend-redis
  11. Create the appendonly file:

    oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'
  12. After a while, ensure that the AOF rewrite is complete:

    oc rsh $(oc get pods -l 'deploymentConfig=backend-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress
    • While aof_rewrite_in_progress = 1, the execution is in progress.
    • Check periodically until aof_rewrite_in_progress = 0. Zero indicates that the execution is complete.
  13. Edit the redis-config configmap:

    oc edit configmap redis-config
  14. Uncomment SAVE commands in the redis-config configmap:

     save 900 1
     save 300 10
     save 60 10000
  15. Set appendonly to yes in the redis-config configmap:

    appendonly yes
  16. Redeploy backend-redis to reload the default configurations:

    oc rollout latest dc/backend-redis
  17. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/backend-redis

6.5.6.2. Managing the deployment configuration for system-redis

These steps are intended for running instances of system-redis.

Procedure

  1. Edit the redis-config configmap:

    oc edit configmap redis-config
  2. Comment SAVE commands in the redis-config configmap:

     #save 900 1
     #save 300 10
     #save 60 10000
  3. Set appendonly to no in the redis-config configmap:

    appendonly no
  4. Redeploy system-redis to load the new configurations:

    oc rollout latest dc/system-redis
  5. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/system-redis
  6. Rename the dump.rdb file:

    oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/dump.rdb ${HOME}/data/dump.rdb-old'
  7. Rename the appendonly.aof file:

    oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'mv ${HOME}/data/appendonly.aof ${HOME}/data/appendonly.aof-old'
  8. Move the Backup file to the POD:

    oc cp ./system-redis-dump.rdb $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r):/var/lib/redis/data/dump.rdb
  9. Redeploy system-redis to load the backup:

    oc rollout latest dc/system-redis
  10. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/system-redis
  11. Create the appendonly file:

    oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli BGREWRITEAOF'
  12. After a while, ensure that the AOF rewrite is complete:

    oc rsh $(oc get pods -l 'deploymentConfig=system-redis' -o json | jq '.items[0].metadata.name' -r) bash -c 'redis-cli info' | grep aof_rewrite_in_progress
    • While aof_rewrite_in_progress = 1, the execution is in progress.
    • Check periodically until aof_rewrite_in_progress = 0. Zero indicates that the execution is complete.
  13. Edit the redis-config configmap:

    oc edit configmap redis-config
  14. Uncomment SAVE commands in the redis-config configmap:

     save 900 1
     save 300 10
     save 60 10000
  15. Set appendonly to yes in the redis-config configmap:

    appendonly yes
  16. Redeploy system-redis to reload the default configurations:

    oc rollout latest dc/system-redis
  17. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/system-redis

6.5.7. Restoring backend-worker

Restore to the latest version of backend-worker:

oc rollout latest dc/backend-worker
  1. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/backend-worker

6.5.8. Restoring system-app

Restore to the latest version of system-app:

oc rollout latest dc/system-app
  1. Check the status of the rollout to ensure it has finished:

    oc rollout status dc/system-app

6.5.9. Creating equivalent Zync routes

To perform the creation of equivalent routes by Zync, force the resynchronization of all 3scale APIs and tenants that OpenShift routes with Zync:

oc exec -t $(oc get pods -l 'deploymentConfig=system-sidekiq' -o json | jq '.items[0].metadata.name' -r) -- bash -c "bundle exec rake zync:resync:domains"

You will see the following output containing information about notifications to system:

No valid API key has been set, notifications will not be sent
ActiveMerchant MODE set to 'production'
[Core] Using http://backend-listener:3000/internal/ as URL
OpenIdAuthentication.store is nil. Using in-memory store.
[EventBroker] notifying subscribers of Domains::ProviderDomainsChangedEvent 59a554f6-7b3f-4246-9c36-24da988ca800
[EventBroker] notifying subscribers of ZyncEvent caa8e941-b734-4192-acb0-0b12cbaab9ca
Enqueued ZyncWorker#d92db46bdba7a299f3e88f14 with args: ["caa8e941-b734-4192-acb0-0b12cbaab9ca", {:type=>"Provider", :id=>1, :parent_event_id=>"59a554f6-7b3f-4246-9c36-24da988ca800", :parent_event_type=>"Domains::ProviderDomainsChangedEvent", :tenant_id=>1}]
[EventBroker] notifying subscribers of Domains::ProviderDomainsChangedEvent 9010a199-2af1-4023-9b8d-297bd618096f
…

New routes are created for all the existing tenants and services, after forcing Zync to reevaluate them. Route creation might take some minutes depending on the number of services and tenants.