Chapter 10. Backup and Restore

Important
  • You must restore with the same operational image version as the backup.
  • To backup and restore your Ansible Automation Platform deployment, it is vital to have your existing Ansible Automation Platform administration secret name and value recorded somewhere safe.
  • It is also important to take regular manual backups of the Cloud SQL database instance and filestore backups, to ensure a deployment can be restored as close as possible to its previous working state.

The playbooks backup and restore provides backup and restore support for the Ansible Automation Platform from GCP Marketplace foundation deployment.

Note

The restore process deploys a new Ansible Automation Platform with the filestore and SQL database instance being restored to the specified backup.

10.1. The Backup Process

A backup enables you to backup your environment by saving the database and the shared file system. A new environment is created during the restore using the saved shared file system. When the new environment is in place, the process restores the database.

The backup and restore process must use the same version. If you performed the backup with an earlier version, you must use the restore process of that version. Then, if required, you can run an upgrade.

You must also make a backup before an upgrade. For further information, read Upgrading your deployment

The backup process involves taking a backup of the Cloud SQL database and filestore instances at a given point in time. The backup playbook requires an active Ansible Automation Platform from GCP Marketplace foundation deployment to be running.

A bucket needs to be created in your project as restore information will be stored in that bucket.

The bucket can contain multiple backups from the same deployment or different ones. A backup will generate a directory named <prefix>-<deployment_name>-<timestamp> and a file named <prefix>-<deployment_name>-<timestamp>.json. The directory contains the awx and pulp databases backup, the deployment configuration and the secrets. The json file contains information for the restore procedure.

Playbooks are provided through the CLI to list and delete backups.

The following procedures describe how to backup the Ansible Automation Platform from GCP Marketplace deployment.

10.1.1. Pulling the ansible-on-clouds-ops container image

Procedure

  • Pull the docker image for the ansible-on-clouds-ops container with the same tag as the foundation deployment.

    Note

    Before pulling the docker image, ensure you are logged in to registry.redhat.io using docker. Use the following command to login to registry.redhat.io.

    $ docker login registry.redhat.io

    For more information about registry login, see Registry Authentication

    $ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel9:2.4.20230630
    $ docker pull $IMAGE --platform=linux/amd64

10.1.2. Required permissions

You must have the following GCP IAM permissions to backup the stack:

required-roles:

Service Account User
Compute Instance Admin (v1)
required-permissions:

compute.instances.list
deploymentmanager.deployments.get
deploymentmanager.manifests.get
deploymentmanager.manifests.list
deploymentmanager.resources.list
file.backups.create
file.operations.get
iap.tunnelInstances.accessViaIAP
storage.objects.create
storage.objects.list

10.1.3. Setting up the environment

Procedure

  • Create a folder to hold the configuration files.

    $ mkdir command_generator_data

10.1.4. Backup requirements

You must create a bucket to store the Ansible on Clouds deployment backup. The bucket contains the backup information, the secrets and the databases backups.

Procedure

  1. In the Google Cloud console, navigate to Cloud StorageBuckets
  2. Choose the project.
  3. Click Create.
  4. Enter a name.
  5. Enter the data location that fits the most your requirements. Multi and Dual regions enable you to run a restore to another region.
  6. Enter the storage class that fits the most your requirements.
  7. Enter the control access that fits the most your requirements.
  8. Enter the data protection that fits the most your requirements.
  9. Click Create

Troubleshooting

An error is raised if the bucket name is already used in another project.

10.1.5. Creating the backup data file

Procedure

  1. Run the command generator command_generator_vars to generate the backup.yml.

    Note

    On Linux, any file or directory created by the command generator is owned by root:root by default. To change the ownership of the files and directories, you can run the sudo chmod command after the files are created. For more information, read Command generator - Linux files owned by root.

    docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars gcp_backup_deployment --output-data-file /data/backup.yml
  2. After running the command, a $(pwd)/command_generator_data/backup.yml template file is created. This template file resembles the following:

    gcp_backup_deployment:
      cloud_credentials_path:
      deployment_name:
      extra_vars:
        backup_prefix: aoc-backup
        gcp_bucket_backup_name:
        gcp_compute_region:
        gcp_compute_zone:

10.1.6. Parameters in the backup.yml file

You must populate the data file before triggering the backup. The following variables are parameters listed in the data file.

  • cloud_credentials_path is the path for your Google Cloud service account credentials file. This must be an absolute path.
  • deployment_name is the name of the AAP deployment manager deployment you want to back up.
  • backup_prefix is a prefix you would like to add to the backup name (default: aoc-backup)
  • gcp_bucket_backup_name is the bucket that was previously created to use for the backup.
  • gcp_compute_region is GCP region where the foundation deployment is deployed. This can be retrieved by checking the Deployments config in Deployment Manager.
  • gcp_compute_zone is the GCP zone where the foundation deployment is deployed. This can be retrieved by checking the Deployments config in Deployment Manager.

10.1.7. Running the backup playbook

Procedure

  1. To run the backup, run the command generator to generate the backup command.

    docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_backup_deployment --data-file /data/backup.yml

    Resulting in the following ouput:

    -----------------------------------------------
    Command to run playbook:
    
    docker run --rm --env PLATFORM=GCP -v  </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \
    --env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name --env GENERATE_INVENTORY=true  \
    $IMAGE redhat.ansible_on_clouds.gcp_backup_deployment \
    -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  \
    gcp_deployment_name=<deployment_name> gcp_compute_region=<region> gcp_compute_zone=<zone> \
    gcp_bucket_backup_name=<bucket> backup_prefix=aoc-backup'
  2. Run the supplied backup command to trigger the backup.

    $ docker run --rm --env PLATFORM=GCP -v  </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \
    --env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name --env GENERATE_INVENTORY=true  \
    $IMAGE redhat.ansible_on_clouds.gcp_backup_deployment \
    -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  \
    gcp_deployment_name=<deployment_name> gcp_compute_region=<region> gcp_compute_zone=<zone> \
    gcp_bucket_backup_name=<bucket> backup_prefix=aoc-backup'
  3. When the playbook has finished running, the output resembles the following:

    TASK [redhat.ansible_on_clouds.standalone_gcp_backup : [backup_deployment] Print the variable required to restore deployment my-deployment] ***
    ok: [localhost] => {
        "msg": [
            "AAP on GCP Backup successful. Please note below the bucket name and backup name which are required for restore process.",
            "gcp_bucket_backup_name: my-bucket",
            "backup_name: aoc-backup-my-deployment-20230616T134002"
        ]
    }
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=38   changed=6    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

10.1.8. List backups

This playbook enables you to list the existing backups in a specific bucket.

Procedure

  1. Populate the command_generator_data directory with the configuration file template.

    Note

    On Linux, any file or directory created by the command generator is owned by root:root by default. To change the ownership of the files and directories, you can run the sudo chmod command after the files are created. For more information, read technical notes.

    docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars gcp_backup_list --output-data-file /data/backups_list.yml
  2. After running the command, a $(pwd)/command_generator_data/backups_list.yml template file is created. This template file resembles the following:

    gcp_backup_list:
      cloud_credentials_path:
      extra_vars:
        gcp_bucket_backup_name:
  3. To run the backup, run the command generator to generate the backup command.

    docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_backup_list --data-file /data/backups_list.yml

    Resulting in the following ouput:

    -----------------------------------------------
    Command to run playbook:
    
    docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \
    --env ANSIBLE_CONFIG=../gcp-ansible.cfg  $IMAGE redhat.ansible_on_clouds.gcp_backup_list \
    -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  gcp_bucket_backup_name=<bucket>'
  4. Run the supplied backup command to trigger the list of backups.
  5. When the playbook has finished running, the output resembles the following:

    TASK [redhat.ansible_on_clouds.standalone_gcp_backup_list : [list_backup] Display list of backups] ***
    ok: [localhost] => {
        "msg": [
            "aoc-backup-deployment1-20230614T203926",
            "aoc-backup-deployment1-20230616T114134",
            "aoc-backup-deployment1-20230616T134002",
            "aoc-backup-deployment2-20230613T124127"
        ]
    }
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=11   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

10.1.9. Delete backups

There are two playbooks to delete backups:

  • Use the gcp_backup_delete playbook which deletes a single backup.
  • Use the gcp_backups_delete playbook which deletes multiple backups at once.

gcp_backups_delete takes a array of strings ["backup1","backup2",…​], while gcp_backup_delete takes only one string, that being the name of a specific backup, "backup1".

The use of gcp_backups_delete is described in this section.

Procedure

  1. Populate the command_generator_data directory with the configuration file template.

    Note

    On Linux, any file or directory created by the command generator is owned by root:root by default. To change the ownership of the files and directories, you can run the sudo chmod command after the files are created. For more information, read Command generator - Linux files owned by root.

    docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars gcp_backups_delete --output-data-file /data/backups_delete.yml
  2. After running the command, a $(pwd)/command_generator_data/backups_delete.yml template file is created. This template file resembles the following:

    gcp_backups_delete:
      cloud_credentials_path:
      extra_vars:
        backup_names:
        delete:
        gcp_bucket_backup_name:

The backup_names parameter must specify an array of strings, for example, ["backup1","backup2"]. The delete parameter must be set to true to successfully delete.

  1. To delete the backups, run the command generator to generate the gcp_backups_delete` command.

    docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_backups_delete --data-file /data/backups_delete.yml

    Resulting in the following ouput:

    -----------------------------------------------
    Command to run playbook:
    
    docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \
    --env ANSIBLE_CONFIG=../gcp-ansible.cfg  $IMAGE redhat.ansible_on_clouds.gcp_backups_delete \
    -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  gcp_bucket_backup_name=<bucket> \
    backup_names=<backup_names> delete=True'
  2. Run the supplied backup command to delete the backups.
  3. When the playbook has finished running, the output resembles the following:

    TASK [redhat.ansible_on_clouds.standalone_gcp_backup_delete : [delete_backup] Dry-run message] ***
    skipping: [localhost]
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=23   changed=2    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

10.1.10. Fixing a failed backup deletion

If the deletion of a backup failed, take the following actions:

Procedure

  1. Navigate to the bucket containing the backup.
  2. Locate the directory that has the name of the backup.
  3. Open the backup directory.
  4. Delete the directory with the name of the backup.
  5. Delete the file with the backup name with .json as extension.
  6. Navigate to FilestoreBackup.
  7. Delete the Filestore backup with the same name as the backup.

10.2. Restore Process

The restore process creates a new deployment, and restores the filestore and SQL database instance to the specified backup.

Important
  • You must restore with the same operational image version which was used for the backup.

The following procedures describe how to restore the Ansible Automation Platform from GCP Marketplace deployment.

10.2.1. Pulling the ansible-on-clouds-ops container image

Procedure

  • Pull the docker image for the ansible-on-clouds-ops container with the same tag as the foundation deployment.

    Note

    Before pulling the docker image, ensure you are logged in to registry.redhat.io using docker. Use the following command to login to registry.redhat.io.

    $ docker login registry.redhat.io

    For more information about registry login, see Registry Authentication

    $ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel9:2.4.20230630
    $ docker pull $IMAGE --platform=linux/amd64

10.2.2. Setting up the environment

Procedure

  • Create a folder to hold the configuration files.

    $ mkdir command_generator_data

10.2.3. Required permissions

You must have the following GCP IAM permissions to restore the stack:

required-roles:

Cloud SQL Client
Cloud SQL Instance User
Compute Instance Admin (v1)
Compute Network Admin
Editor
Logs Writer
Secret Manager Secret Accessor
Service Account User
required-permissions:

compute.instances.list
compute.networks.create
deploymentmanager.deployments.create
deploymentmanager.deployments.get
deploymentmanager.operations.get
file.instances.create
file.operations.get
iap.tunnelInstances.accessViaIAP
secretmanager.secrets.create
secretmanager.secrets.delete
secretmanager.secrets.get
secretmanager.secrets.update
secretmanager.versions.add
secretmanager.versions.list
storage.objects.get
storage.objects.list

10.2.4. Generating the restore.yml file

Procedure

  1. Run the command generator command_generator_vars to generate restore.yml.

    Note

    On Linux, any file or directory created by the command generator is owned by root:root by default. To change the ownership of the files and directories, you can run the sudo chmod command after the files are created. For more information, read technical notes.

    docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator_vars gcp_restore_deployment --output-data-file /data/restore.yml
  2. After running the command, a $(pwd)/command_generator_data/restore.yml template file is created. This template file resembles the following:

    ===============================================
    Playbook: gcp_restore_deployment
    Description: This playbook is used to restore the Ansible Automation Platform from GCP Marketplace environment from a backup.
    -----------------------------------------------
    This playbook is used to restore the Ansible Automation Platform from GCP Marketplace environment from a backup.
    For more information regarding backup and restore, visit our official documentation -
    https://access.redhat.com/documentation/en-us/ansible_on_clouds/2.x/html/red_hat_ansible_automation_platform_from_gcp_marketplace_guide/index
    
    -----------------------------------------------
    Command generator template:
    
    docker run --rm -v <local_data_file_directory>:/data $IMAGE command_generator gcp_restore_deployment --data-file /data/restore.yml

    The template resembles the following:

    gcp_restore_deployment:
      cloud_credentials_path:
      deployment_name:
      extra_vars:
        backup_name:
        gcp_bucket_backup_name:
        gcp_cloud_sql_peering_network:
        gcp_compute_region:
        gcp_compute_zone:
        gcp_controller_internal_ip_address:
        gcp_existing_vpc:
        gcp_filestore_ip_range:
        gcp_hub_internal_ip_address:

10.2.5. Parameters of the restore.yml file

You can only restore into a new VPC network.

For a new VPC

If you want to restore using a new VPC set the following parameters:

  • gcp_existing_vpc must be set to false.

The following parameters must be removed:

  • gcp_filestore_ip_range
  • gcp_cloud_sql_peering_network
  • gcp_controller_internal_ip_address
  • gcp_hub_internal_ip_address

Provide values for the following parameters:

  • gcp_existing_vpc must be set to false.
  • cloud_credentials_path is the path for your Google Cloud service account credentials file.
  • deployment_name is the name under which the deployment must be restored. A new deployment will be created with this name. A deployment must not already exist with this name.
  • backup_name is the name of the backup in the bucket. This name is displayed while using the gcp_backup_deployment or gcp_backup_list commands.
  • gcp_bucket_backup_name is the bucket name you used for the backup.
  • gcp_compute_region is the region where the backup was taken. This can be retrieved by checking the Deployments config in Deployment Manager.
  • gcp_compute_zone is the zone where the backup was taken. This can be retrieved by checking the Deployments config in Deployment Manager.

For an existing VPC

If you want to restore using an existing VPC, you must set the parameters shown above.

You must also set the following additional parameters:

  • gcp_existing_vpc is set to true.
  • gcp_filestore_ip_range must be set to a free ip/29 range of your VPC. For example: 192.168.245.0/29. You must not use 192.168.243.0/29 as this the default used when deploying Ansible Automation Platform from GCP Marketplace.
  • gcp_cloud_sql_peering_network must be set to a free /24 subnet. You must not use 192.168.241.0/24 as this is used during the original deployment.
  • gcp_controller_internal_ip_address must be set to a free IP in your VPC network.
  • gcp_hub_internal_ip_address must set to a free IP in your VPC network.

10.2.6. Running the restore command

When $(pwd)/command_generator_data/restore.yml is populated, you can use the command generator to create the restore command.

Procedure

  1. Run the command generator.

    $ docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_restore_deployment --data-file /data/restore.yml

    This generates a new command containing all needed volumes, environment variables and parameters.

    The generated command resembles the following:

    docker run --rm --env PLATFORM=GCP -v <local_credential_file>:/home/runner/.gcp/credentials:ro \
    --env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name> --env GENERATE_INVENTORY=true -\
    -env CHECK_GENERATED_INVENTORY=false  $IMAGE redhat.ansible_on_clouds.gcp_restore_deployment \
    -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  \
    gcp_deployment_name=<deployment_name> gcp_compute_region=<region> gcp_compute_zone=<zone> \
    gcp_bucket_backup_name=<bucket> backup_name=<backup_name> gcp_existing_vpc=<existing_vpc>'
  2. Run the generated command.

    $ docker run --rm --env PLATFORM=GCP -v <local_credential_file>:/home/runner/.gcp/credentials:ro \
    --env ANSIBLE_CONFIG=../gcp-ansible.cfg  $IMAGE redhat.ansible_on_clouds.gcp_restore_deployment \
    -e 'gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  \
    gcp_deployment_name=<former_deployment_name> gcp_restored_deployment_name=<new_deployment_name> \
    gcp_compute_region=<region> gcp_compute_zone=<zone> gcp_bucket_backup_name=<bucket> gcp_existing_vpc=False'
  3. When the playbook has completed, the output resembles the following:

    TASK [redhat.ansible_on_clouds.standalone_gcp_restore : Display internal IP addresses] ***
    ok: [localhost] =>
      msg:
      - 'Hub        internal IP: 192.168.240.21'
      - 'Controller internal IP: 192.168.240.20'
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=33   changed=8    unreachable=0    failed=0    skipped=6    rescued=0    ignored=2

10.2.7. Failure to restore

If a message similar to the following appears during the restore, you must contact support as the restore must be done manually.

TASK [redhat.ansible_on_clouds.standalone_gcp_restore : [restore_deployment] Restore awx db] *
fatal: [localhost -> dvernier-restore1-aap-cntrlr-x2c6]: FAILED!