Chapter 10. Backup and Restore
- You must restore with the same operational image version as the backup.
- To backup and restore your Ansible Automation Platform deployment, it is vital to have your existing Ansible Automation Platform administration secret name and value recorded somewhere safe.
- It is also important to take regular manual backups of the Cloud SQL database instance and filestore backups, to ensure a deployment can be restored as close as possible to its previous working state.
The playbooks backup and restore provides backup and restore support for the Ansible Automation Platform from GCP Marketplace foundation deployment.
The restore process deploys a new Ansible Automation Platform with the filestore and SQL database instance being restored to the specified backup.
10.1. The Backup Process
A backup enables you to backup your environment by saving the database and the shared file system. A new environment is created during the restore using the saved shared file system. When the new environment is in place, the process restores the database.
The backup and restore process must use the same version. If you performed the backup with an earlier version, you must use the restore process of that version. Then, if required, you can run an upgrade.
You must also make a backup before an upgrade. For further information, read Upgrading your deployment
The backup process involves taking a backup of the Cloud SQL database and filestore instances at a given point in time. The backup playbook requires an active Ansible Automation Platform from GCP Marketplace foundation deployment to be running.
A bucket needs to be created in your project as restore information will be stored in that bucket.
The bucket can contain multiple backups from the same deployment or different ones. A backup will generate a directory named <prefix>-<deployment_name>-<timestamp> and a file named <prefix>-<deployment_name>-<timestamp>.json. The directory contains the awx and pulp databases backup, the deployment configuration and the secrets. The json file contains information for the restore procedure.
Playbooks are provided through the CLI to list and delete backups.
The following procedures describe how to backup the Ansible Automation Platform from GCP Marketplace deployment.
10.1.1. Pulling the ansible-on-clouds-ops container image
Procedure
Pull the docker image for the
ansible-on-clouds-opscontainer with the same tag as the foundation deployment.NoteBefore pulling the docker image, ensure you are logged in to registry.redhat.io using docker. Use the following command to login to registry.redhat.io.
$ docker login registry.redhat.io
For more information about registry login, see Registry Authentication
$ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel9:2.4.20230630 $ docker pull $IMAGE --platform=linux/amd64
10.1.2. Required permissions
You must have the following GCP IAM permissions to backup the stack:
required-roles: Service Account User Compute Instance Admin (v1) required-permissions: compute.instances.list deploymentmanager.deployments.get deploymentmanager.manifests.get deploymentmanager.manifests.list deploymentmanager.resources.list file.backups.create file.operations.get iap.tunnelInstances.accessViaIAP storage.objects.create storage.objects.list
10.1.3. Setting up the environment
Procedure
Create a folder to hold the configuration files.
$ mkdir command_generator_data
10.1.4. Backup requirements
You must create a bucket to store the Ansible on Clouds deployment backup. The bucket contains the backup information, the secrets and the databases backups.
Procedure
- In the Google Cloud console, navigate to Cloud Storage → Buckets
- Choose the project.
- Click Create.
- Enter a name.
- Enter the data location that fits the most your requirements. Multi and Dual regions enable you to run a restore to another region.
- Enter the storage class that fits the most your requirements.
- Enter the control access that fits the most your requirements.
- Enter the data protection that fits the most your requirements.
- Click Create
Troubleshooting
An error is raised if the bucket name is already used in another project.
10.1.5. Creating the backup data file
Procedure
Run the command generator
command_generator_varsto generate thebackup.yml.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read Command generator - Linux files owned by root.docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars gcp_backup_deployment --output-data-file /data/backup.yml
After running the command, a
$(pwd)/command_generator_data/backup.ymltemplate file is created. This template file resembles the following:gcp_backup_deployment: cloud_credentials_path: deployment_name: extra_vars: backup_prefix: aoc-backup gcp_bucket_backup_name: gcp_compute_region: gcp_compute_zone:
10.1.6. Parameters in the backup.yml file
You must populate the data file before triggering the backup. The following variables are parameters listed in the data file.
-
cloud_credentials_pathis the path for your Google Cloud service account credentials file. This must be an absolute path. -
deployment_nameis the name of the AAP deployment manager deployment you want to back up. -
backup_prefixis a prefix you would like to add to the backup name (default: aoc-backup) -
gcp_bucket_backup_nameis the bucket that was previously created to use for the backup. -
gcp_compute_regionis GCP region where the foundation deployment is deployed. This can be retrieved by checking the Deployments config in Deployment Manager. -
gcp_compute_zoneis the GCP zone where the foundation deployment is deployed. This can be retrieved by checking the Deployments config in Deployment Manager.
10.1.7. Running the backup playbook
Procedure
To run the backup, run the command generator to generate the backup command.
docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_backup_deployment --data-file /data/backup.yml
Resulting in the following ouput:
----------------------------------------------- Command to run playbook: docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \ --env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name --env GENERATE_INVENTORY=true \ $IMAGE redhat.ansible_on_clouds.gcp_backup_deployment \ -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials \ gcp_deployment_name=<deployment_name> gcp_compute_region=<region> gcp_compute_zone=<zone> \ gcp_bucket_backup_name=<bucket> backup_prefix=aoc-backup'
Run the supplied backup command to trigger the backup.
$ docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \ --env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name --env GENERATE_INVENTORY=true \ $IMAGE redhat.ansible_on_clouds.gcp_backup_deployment \ -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials \ gcp_deployment_name=<deployment_name> gcp_compute_region=<region> gcp_compute_zone=<zone> \ gcp_bucket_backup_name=<bucket> backup_prefix=aoc-backup'
When the playbook has finished running, the output resembles the following:
TASK [redhat.ansible_on_clouds.standalone_gcp_backup : [backup_deployment] Print the variable required to restore deployment my-deployment] *** ok: [localhost] => { "msg": [ "AAP on GCP Backup successful. Please note below the bucket name and backup name which are required for restore process.", "gcp_bucket_backup_name: my-bucket", "backup_name: aoc-backup-my-deployment-20230616T134002" ] } PLAY RECAP ********************************************************************* localhost : ok=38 changed=6 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
10.1.8. List backups
This playbook enables you to list the existing backups in a specific bucket.
Procedure
Populate the
command_generator_datadirectory with the configuration file template.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read technical notes.docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars gcp_backup_list --output-data-file /data/backups_list.yml
After running the command, a
$(pwd)/command_generator_data/backups_list.ymltemplate file is created. This template file resembles the following:gcp_backup_list: cloud_credentials_path: extra_vars: gcp_bucket_backup_name:To run the backup, run the command generator to generate the backup command.
docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_backup_list --data-file /data/backups_list.yml
Resulting in the following ouput:
----------------------------------------------- Command to run playbook: docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \ --env ANSIBLE_CONFIG=../gcp-ansible.cfg $IMAGE redhat.ansible_on_clouds.gcp_backup_list \ -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials gcp_bucket_backup_name=<bucket>'
- Run the supplied backup command to trigger the list of backups.
When the playbook has finished running, the output resembles the following:
TASK [redhat.ansible_on_clouds.standalone_gcp_backup_list : [list_backup] Display list of backups] *** ok: [localhost] => { "msg": [ "aoc-backup-deployment1-20230614T203926", "aoc-backup-deployment1-20230616T114134", "aoc-backup-deployment1-20230616T134002", "aoc-backup-deployment2-20230613T124127" ] } PLAY RECAP ********************************************************************* localhost : ok=11 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
10.1.9. Delete backups
There are two playbooks to delete backups:
-
Use the
gcp_backup_deleteplaybook which deletes a single backup. -
Use the
gcp_backups_deleteplaybook which deletes multiple backups at once.
gcp_backups_delete takes a array of strings ["backup1","backup2",…], while gcp_backup_delete takes only one string, that being the name of a specific backup, "backup1".
The use of gcp_backups_delete is described in this section.
Procedure
Populate the
command_generator_datadirectory with the configuration file template.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read Command generator - Linux files owned by root.docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars gcp_backups_delete --output-data-file /data/backups_delete.yml
After running the command, a
$(pwd)/command_generator_data/backups_delete.ymltemplate file is created. This template file resembles the following:gcp_backups_delete: cloud_credentials_path: extra_vars: backup_names: delete: gcp_bucket_backup_name:
The backup_names parameter must specify an array of strings, for example, ["backup1","backup2"]. The delete parameter must be set to true to successfully delete.
To delete the backups, run the command generator to generate the
gcp_backups_delete`command.docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_backups_delete --data-file /data/backups_delete.yml
Resulting in the following ouput:
----------------------------------------------- Command to run playbook: docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \ --env ANSIBLE_CONFIG=../gcp-ansible.cfg $IMAGE redhat.ansible_on_clouds.gcp_backups_delete \ -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials gcp_bucket_backup_name=<bucket> \ backup_names=<backup_names> delete=True'
- Run the supplied backup command to delete the backups.
When the playbook has finished running, the output resembles the following:
TASK [redhat.ansible_on_clouds.standalone_gcp_backup_delete : [delete_backup] Dry-run message] *** skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=23 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
10.1.10. Fixing a failed backup deletion
If the deletion of a backup failed, take the following actions:
Procedure
- Navigate to the bucket containing the backup.
- Locate the directory that has the name of the backup.
- Open the backup directory.
- Delete the directory with the name of the backup.
-
Delete the file with the backup name with
.jsonas extension. - Navigate to Filestore → Backup.
- Delete the Filestore backup with the same name as the backup.
10.2. Restore Process
The restore process creates a new deployment, and restores the filestore and SQL database instance to the specified backup.
- You must restore with the same operational image version which was used for the backup.
The following procedures describe how to restore the Ansible Automation Platform from GCP Marketplace deployment.
10.2.1. Pulling the ansible-on-clouds-ops container image
Procedure
Pull the docker image for the
ansible-on-clouds-opscontainer with the same tag as the foundation deployment.NoteBefore pulling the docker image, ensure you are logged in to registry.redhat.io using docker. Use the following command to login to registry.redhat.io.
$ docker login registry.redhat.io
For more information about registry login, see Registry Authentication
$ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel9:2.4.20230630 $ docker pull $IMAGE --platform=linux/amd64
10.2.2. Setting up the environment
Procedure
Create a folder to hold the configuration files.
$ mkdir command_generator_data
10.2.3. Required permissions
You must have the following GCP IAM permissions to restore the stack:
required-roles: Cloud SQL Client Cloud SQL Instance User Compute Instance Admin (v1) Compute Network Admin Editor Logs Writer Secret Manager Secret Accessor Service Account User required-permissions: compute.instances.list compute.networks.create deploymentmanager.deployments.create deploymentmanager.deployments.get deploymentmanager.operations.get file.instances.create file.operations.get iap.tunnelInstances.accessViaIAP secretmanager.secrets.create secretmanager.secrets.delete secretmanager.secrets.get secretmanager.secrets.update secretmanager.versions.add secretmanager.versions.list storage.objects.get storage.objects.list
10.2.4. Generating the restore.yml file
Procedure
Run the command generator
command_generator_varsto generaterestore.yml.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read technical notes.docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator_vars gcp_restore_deployment --output-data-file /data/restore.yml
After running the command, a
$(pwd)/command_generator_data/restore.ymltemplate file is created. This template file resembles the following:=============================================== Playbook: gcp_restore_deployment Description: This playbook is used to restore the Ansible Automation Platform from GCP Marketplace environment from a backup. ----------------------------------------------- This playbook is used to restore the Ansible Automation Platform from GCP Marketplace environment from a backup. For more information regarding backup and restore, visit our official documentation - https://access.redhat.com/documentation/en-us/ansible_on_clouds/2.x/html/red_hat_ansible_automation_platform_from_gcp_marketplace_guide/index ----------------------------------------------- Command generator template: docker run --rm -v <local_data_file_directory>:/data $IMAGE command_generator gcp_restore_deployment --data-file /data/restore.yml
The template resembles the following:
gcp_restore_deployment: cloud_credentials_path: deployment_name: extra_vars: backup_name: gcp_bucket_backup_name: gcp_cloud_sql_peering_network: gcp_compute_region: gcp_compute_zone: gcp_controller_internal_ip_address: gcp_existing_vpc: gcp_filestore_ip_range: gcp_hub_internal_ip_address:
10.2.5. Parameters of the restore.yml file
You can only restore into a new VPC network.
For a new VPC
If you want to restore using a new VPC set the following parameters:
-
gcp_existing_vpcmust be set tofalse.
The following parameters must be removed:
-
gcp_filestore_ip_range -
gcp_cloud_sql_peering_network -
gcp_controller_internal_ip_address -
gcp_hub_internal_ip_address
Provide values for the following parameters:
-
gcp_existing_vpcmust be set tofalse. -
cloud_credentials_pathis the path for your Google Cloud service account credentials file. -
deployment_nameis the name under which the deployment must be restored. A new deployment will be created with this name. A deployment must not already exist with this name. -
backup_nameis the name of the backup in the bucket. This name is displayed while using thegcp_backup_deploymentorgcp_backup_listcommands. -
gcp_bucket_backup_nameis the bucket name you used for the backup. -
gcp_compute_regionis the region where the backup was taken. This can be retrieved by checking the Deployments config in Deployment Manager. -
gcp_compute_zoneis the zone where the backup was taken. This can be retrieved by checking the Deployments config in Deployment Manager.
For an existing VPC
If you want to restore using an existing VPC, you must set the parameters shown above.
You must also set the following additional parameters:
-
gcp_existing_vpcis set totrue. -
gcp_filestore_ip_rangemust be set to a free ip/29 range of your VPC. For example: 192.168.245.0/29. You must not use 192.168.243.0/29 as this the default used when deploying Ansible Automation Platform from GCP Marketplace. -
gcp_cloud_sql_peering_networkmust be set to a free/24subnet. You must not use 192.168.241.0/24 as this is used during the original deployment. -
gcp_controller_internal_ip_addressmust be set to a free IP in your VPC network. -
gcp_hub_internal_ip_addressmust set to a free IP in your VPC network.
10.2.6. Running the restore command
When $(pwd)/command_generator_data/restore.yml is populated, you can use the command generator to create the restore command.
Procedure
Run the command generator.
$ docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_restore_deployment --data-file /data/restore.yml
This generates a new command containing all needed volumes, environment variables and parameters.
The generated command resembles the following:
docker run --rm --env PLATFORM=GCP -v <local_credential_file>:/home/runner/.gcp/credentials:ro \ --env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name> --env GENERATE_INVENTORY=true -\ -env CHECK_GENERATED_INVENTORY=false $IMAGE redhat.ansible_on_clouds.gcp_restore_deployment \ -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials \ gcp_deployment_name=<deployment_name> gcp_compute_region=<region> gcp_compute_zone=<zone> \ gcp_bucket_backup_name=<bucket> backup_name=<backup_name> gcp_existing_vpc=<existing_vpc>'
Run the generated command.
$ docker run --rm --env PLATFORM=GCP -v <local_credential_file>:/home/runner/.gcp/credentials:ro \ --env ANSIBLE_CONFIG=../gcp-ansible.cfg $IMAGE redhat.ansible_on_clouds.gcp_restore_deployment \ -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials \ gcp_deployment_name=<former_deployment_name> gcp_restored_deployment_name=<new_deployment_name> \ gcp_compute_region=<region> gcp_compute_zone=<zone> gcp_bucket_backup_name=<bucket> gcp_existing_vpc=False'
When the playbook has completed, the output resembles the following:
TASK [redhat.ansible_on_clouds.standalone_gcp_restore : Display internal IP addresses] *** ok: [localhost] => msg: - 'Hub internal IP: 192.168.240.21' - 'Controller internal IP: 192.168.240.20' PLAY RECAP ********************************************************************* localhost : ok=33 changed=8 unreachable=0 failed=0 skipped=6 rescued=0 ignored=2
10.2.7. Failure to restore
If a message similar to the following appears during the restore, you must contact support as the restore must be done manually.
TASK [redhat.ansible_on_clouds.standalone_gcp_restore : [restore_deployment] Restore awx db] *
fatal: [localhost -> dvernier-restore1-aap-cntrlr-x2c6]: FAILED!