Chapter 9. Backup and Restore
To backup and restore your Ansible Automation Platform deployment, it is vital to ensure automatic backups have been set for EFS and that these backups are accessible for restoration.
When you create an Ansible Automation Platform deployment, automatic Amazon Elastic File System (EFS) backups are set by default. However, it is prudent to check that those backups can be restored before a disaster scenario.
In addition, it is also extremely important that regular manual snapshots of the Amazon Relational Database Service (RDS) are taken to ensure a deployment can be restored as close as possible to its previous working state.
9.1. Backing up the Ansible Automation Platform deployment
Use the following procedures to ensure a smooth backup process.
9.1.1. AWS permissions
Before starting the backup process, the AWS account used to execute a backup must have the following permissions granted.
| Service/Permission | Resources |
|---|---|
| backup
| |
| cloudformation
| |
| elasticfilesystem
| |
| iam
| arn:aws:iam::*:`role/service-role/AWSBackupDefaultServiceRole` |
| rds
| |
| secretsmanager
| |
| s3
|
9.1.2. Setting the ansible-on-clouds-ops container image
The ansible-on-clouds-ops image tag should match the version of your foundation deployment. For example, if your foundation deployment version is 2.4.20230630-00, pull the ansible-on-clouds-ops image with tag 2.4.20230630.
Procedure
Pull the
ansible-on-clouds-opscontainer image with the same tag version as the foundation deployment.NoteBefore pulling the docker image, ensure you are logged in to registry.redhat.io using docker. Use the following command to login to registry.redhat.io.
$ docker login registry.redhat.io
For more information about registry login, see Registry Authentication
$ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel9:2.4.20230630 $ docker pull $IMAGE --platform=linux/amd64
9.1.3. Generating the backup data file
The following commands create a directory, and populate it with an empty data template that, when completed, will be used during the backup.
Procedure
Create a folder to hold the configuration
$ mkdir command_generator_data
Populate the
command_generator_datafolder with the configuration file template. This creates thebackup.ymlfile within thecommand-generator_datadirectory.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read Command generator - Linux files owned by root.$ docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE \ command_generator_vars aws_backup_stack \ --output-data-file /data/backup.yml
After running these commands, a
$(pwd)/command_generator_data/backup.ymltemplate file is created. This template file resembles the following:aws_backup_stack: ansible_config_path: cloud_credentials_path: deployment_name: extra_vars: aws_backup_iam_role_arn: aws_backup_vault_name: Default aws_region: aws_s3_bucket: aws_ssm_bucket_name: backup_prefix: aoc-backup
9.1.4. Updating the backup data file
You must populate the data file before triggering the backup. The following variables are parameters listed in the data file.
-
ansible_config_path(Optional) Only use if overriding with a customeransible_config. -
cloud_credentials_pathis the path to your AWS credentials file. -
deployment_nameis the name of the foundation deployment. This is the same name you used when you deployed the foundation. aws_backup_iam_role_arnis the Amazon Resource Name (ARN) of the AWS IAM Role that has permissions to perform backup operations.NoteYou can use the AWS Backup Default Service Role for this which has the format
arn:aws:iam::<Your AWS Account Number>:role/service-role/AWSBackupDefaultServiceRole.aws_backup_vault_nameis the name of the backup vault that will hold the EFS recovery points.NotePlease ensure the referenced backup vault exists and the AWS Backup IAM role provided above has the required permissions to create an EFS recovery point inside the vault.
-
aws_regionis the region where the foundation deployment is deployed. -
aws_s3_bucketis the name of the S3 bucket where the backup files are stored. This creates a new bucket if one does not already exist. Every backup is stored in the bucket.
For guidance on bucket naming, see Bucket naming rules.
For guidance on creating S3 buckets, see AWS Creating a bucket. aws_ssm_bucket_nameis the name of the S3 bucket where temporary configuration files for the AWS SSM are stored. You can use an existing bucket or create a new one.NoteThe
aws_ssm_bucket_nameparameter is ONLY used to store temporary config files. It does NOT need to be saved for use in other playbooks. Any valid existing bucket can be used. For more information on creating S3 buckets, read AWS Creating A Bucket in the AWS documentation.-
backup_prefixis a prefix you would like to add to the backup name (default: aoc-backup)
When you have populated the data file, it should resemble the following. The values in this file are provided as examples:
The optional value ansible_config_path has been removed. If you do not wish to use this optional value and want to use the default value for this variables, you 'must' also remove it for your data file like it was done in the example below. If you wish to use this optional variable, it 'must' be included in the data file and be assigned a value.
aws_backup_stack:
cloud_credentials_path: ~/.aws/credentials
deployment_name: AnsibleAutomationPlatform
extra_vars:
aws_backup_iam_role_arn: arn:aws:iam::<Your AWS Account Number>:role/service-role/AWSBackupDefaultServiceRole
aws_backup_vault_name: Default
aws_region: us-east-1
aws_s3_bucket: ansible-automation-platform-bucket
aws_ssm_bucket_name: aap-ssm-bucket
backup_prefix: aoc-backup9.1.5. Running the backup playbook
The following procedure runs the backup playbook as a container.
Procedure
To run the backup, run the command generator.
$ docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator aws_backup_stack --data-file /data/backup.yml
This generates the backup CLI command
---------------------------------------------- docker run --rm --env PLATFORM=AWS -v ~/.aws/credentials:/home/runner/.aws/credentials:ro \ --env ANSIBLE_CONFIG=../aws-ansible.cfg --env DEPLOYMENT_NAME=AnsibleAutomationPlatform --env GENERATE_INVENTORY=true \ $IMAGE redhat.ansible_on_clouds.aws_backup_stack \ -e 'aws_foundation_stack_name=AnsibleAutomationPlatform aws_region=us-east-1 aws_backup_vault_name=Default \ aws_backup_iam_role_arn=arn:aws:iam::<Your AWS Account Number>:role/service-role/AWSBackupDefaultServiceRole \ aws_s3_bucket=ansible-automation-platform-bucket aws_ssm_bucket_name=aap-ssm-bucket backup_prefix=aoc-backup' ===============================================
Run the generated command to trigger the backup.
$ docker run --rm --env PLATFORM=AWS -v ~/.aws/credentials:/home/runner/.aws/credentials:ro \ --env ANSIBLE_CONFIG=../aws-ansible.cfg --env DEPLOYMENT_NAME=AnsibleAutomationPlatform --env GENERATE_INVENTORY=true \ $IMAGE redhat.ansible_on_clouds.aws_backup_stack \ -e 'aws_foundation_stack_name=AnsibleAutomationPlatform aws_region=us-east-1 aws_backup_vault_name=Default \ aws_backup_iam_role_arn=arn:aws:iam::<Your AWS Account Number>:role/service-role/AWSBackupDefaultServiceRole \ aws_s3_bucket=ansible-automation-platform-bucket aws_ssm_bucket_name=aap-ssm-bucket backup_prefix=aoc-backup'
The backup can take several minutes to complete, depending on the database size. A successful backup returns a log similar to the following:
{ "msg": [ "Successfully backed up AnsibleAutomationPlatform!", "Please note below the bucket name, region and backup name which are required for restore process.", "aws_s3_bucket: ansible-automation-platform-bucket ", "aws_region: us-east-1", "aws_backup_name: ansible-automation-platform-bucket-20230706T163309", "Your backup files can be found at:", "https://s3.console.aws.amazon.com/s3/buckets/ansible-automation-platform-bucket?region=us-east-1&prefix=aoc-backup-AnsibleAutomationPlatform-20230706T163309/&showversions=false" ] }- Your Ansible Automation Platform from AWS Marketplace deployment is now successfully backed up. As the log shows, the playbook successfully creates a backup folder in the S3 bucket specified above.
9.1.6. Deleting backups
There are two playbooks to delete backups:
-
Use the
aws_backup_deleteplaybook which deletes a single backup. -
Use the
aws_backups_deleteplaybook which deletes multiple backups at once.
aws_backups_delete takes a array of strings ["backup1","backup2",…], while aws_backup_delete takes only one string, that being the name of a specific backup, "backup1".
The use of aws_backups_delete is desscribed in this section.
Procedure
Populate the
command_generator_datadirectory with the configuration file template.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read Command generator - Linux files owned by root.docker run --rm -v $(pwd)/command_generator_data/:/data $IMAGE command_generator_vars aws_backups_delete --output-data-file /data/backups_delete.yml
Produces the following output:
=============================================== Playbook: aws_backups_delete Description: This playbook delete a specified backup. ----------------------------------------------- This playbook delete a specified backup ----------------------------------------------- Command generator template: docker run --rm -v <local_data_file_directory>:/data $IMAGE command_generator aws_backups_delete --data-file /data/backups_delete.yml
After running the command, a
$(pwd)/command_generator_data/backups_delete.ymltemplate file is created. This template file resembles the following:aws_backups_delete: cloud_credentials_path: extra_vars: aws_backup_names: aws_region: aws_s3_bucket: delete:
The aws_backup_names parameter must specify an array of strings, for example, ["backup1","backup2"]. The delete parameter must be set to true to successfully delete.
To delete the backups, run the command generator to generate the
aws_backups_deletecommand.docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator aws_backups_delete --data-file /data/backups_delete.yml
Resulting in the following ouput:
Command to run playbook: docker run --rm --env PLATFORM=AWS -v ~/.aws/credentials:/home/runner/.aws/credentials:ro \ --env ANSIBLE_CONFIG=../aws-ansible.cfg $IMAGE redhat.ansible_on_clouds.aws_backups_delete \ -e 'aws_region=<region> aws_s3_bucket=<bucket> aws_backup_names=["backup1","backup2"] delete=True' ===============================================
- Run the supplied backup command to delete the backups.
When the playbook has finished running, the output resembles the following:
TASK [redhat.ansible_on_clouds.standalone_aws_backup_delete : [delete_backup] Dry-run message] *** skipping: [localhost] PLAY RECAP ********************************************************************* localhost : ok=21 changed=2 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
9.1.6.1. Failing to delete a backup
If the deletion of backup fails, take the following steps.
Procedure
- Navigate to the bucket containing the backup.
- Locate the directory that has the name of the backup.
- Open the backup directory.
-
Open the file names
restore-vars.jsonand note last part of theRecovery_Point_ARNand theBackup_Vault_Name. - Navigate to AWS backup.
- Select Backup Vault.
-
Select the vault with the name you noted from
Backup_Vault_Name. -
Search for the
Recovery_Point_ARN. -
Delete the
Recovery_Point_ARNif not already deleted. - Navigate to the bucket containing the backup.
- Delete the directory having the name of the backup.
9.2. Restoring the Ansible Automation Platform deployment
Use the following procedures to ensure a smooth restore process.
A restored deployment contains the same VPC networking setup as the original backed up deployment. If a backed up deployment was deployed within an existing VPC, its restored deployment is also deployed into that VPC. If a backed up deployment was deployed with a new VPC, its restored deployment is also deployed with a new VPC.
9.2.1. AWS permissions
You must have the following permissions granted to restore the deployment you are planning to restore.
The following table contains a list of necessary IAM policies
| For deployment with a new VPC | For deployment with an existing VPC |
|---|---|
| autoscaling
| autoscaling
|
| backup
| backup
|
| cloudformation
| cloudformation
|
| ec2
| ec2
|
| elasticfilesystem
| elasticfilesystem
|
| elasticloadbalancing
| elasticloadbalancing
|
| iam
| iam
|
| kms
| kms
|
| rds
| rds
|
| s3
| s3
|
| secretsmanager
| secretsmanager
|
| sns
| sns
|
9.2.2. Setting the ansible-on-clouds-ops container image
The ansible-on-clouds-ops image tag should match the version of your foundation deployment. For example, if your foundation deployment version is 2.4.20230630-00, pull the ansible-on-clouds-ops image with tag 2.4.20230630.
Procedure
Pull the
ansible-on-clouds-opscontainer image with the same tag version as the foundation deployment.NoteBefore pulling the docker image, ensure you are logged in to registry.redhat.io using docker. Use the following command to login to registry.redhat.io.
$ docker login registry.redhat.io
For more information about registry login, see Registry Authentication
$ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel9:2.4.20230630 $ docker pull $IMAGE --platform=linux/amd64
9.2.3. Generating the restore data file
The following commands create a directory, and populate it with an empty data template that, when completed, is used during the restore.
Procedure
Create a folder to hold the configuration
$ mkdir command_generator_data
Populate the
$(pwd)/command_generator_datafolder with the configuration file template.NoteOn Linux, any file or directory created by the command generator is owned by
root:rootby default. To change the ownership of the files and directories, you can run thesudo chmodcommand after the files are created. For more information, read Command generator - Linux files owned by root.$ docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE \ command_generator_vars aws_restore_stack \ --output-data-file /data/restore.yml
After running these commands, a
$(pwd)/command_generator_data/restore.ymltemplate file is created. This template file resembles the following:aws_restore_stack: ansible_config_path: cloud_credentials_path: deployment_name: extra_vars: aws_backup_iam_role_arn: aws_backup_name: aws_backup_restore_point_arn: aws_backup_vault_name: aws_rds_db_snapshot_arn: aws_region: aws_s3_bucket: aws_ssm_bucket_name:
9.2.4. Updating the restore data file
You must populate the data file before triggering the restore. The following variables are parameters listed in the data file.
-
ansible_config_path(Optional) Only use if overriding with a customansible_config. -
cloud_credentials_pathis the path to your AWS credentials file. -
deployment_nameis the name you want for your restored deployment aws_backup_iam_role_arn(Optional) is the Amazon Resource Name (ARN) of the AWS IAM Role that has permissions to perform backup operations.NoteIf provided, the playbook defaults to this value over the value of the IAM role referenced in the backup file on S3. For further information, see Backing up the Ansible Automation Platform deployment
-
aws_backup_nameis the name of the backup folder stored on S3. This value was outputted after running the backup playbook. aws_backup_restore_point_arn(Optional) is the ARN of the of the EFS recovery point you want to use for restore.NoteIf provided, the playbook defaults to this value over the EFS restore point referenced in the backup file on S3. See AWS permissions.
You must also ensure that the provided restore point is from a version of Ansible on Clouds (AoC) that matches the version of the ansible-on-clouds-ops container image running the restore. For example, a restore point taken from a deployment on AoC version 2.4.20230630-00 needs to use the
ansible-on-clouds-opscontainer image version2.4.20230630.aws_backup_vault_name(Optional) is the name of the backup vault that holds the EFS recovery points.NoteIf provided, the playbook defaults to this value over the backup vault referenced in the backup file on S3 which was generated when running the backup operation. See AWS permissions. You must also ensure that this backup vault matches the one used to backup the deployment.
aws_rds_db_snapshot_arn(Optional) is the ARN of the RDS snapshot you want to use for restore.NoteIf provided, the playbook defaults to this value over the RDS snapshot referenced in the backup file on S3. You must also ensure that the provided RDS snapshot is from a version of Ansible on Clouds that matches the version of the
ansible-on-clouds-opscontainer image running the restore. For example, an RDS snapshot taken from a deployment on AoC version 2.4.20230630-00 needs to use theansible-on-clouds-opscontainer image version2.4.20230630.-
aws_regionis the region where the foundation deployment is deployed. -
aws_s3_bucketis the name of the S3 bucket where the backup files are stored. This must be the same bucket used for backup. aws_ssm_bucket_nameis the name of the S3 bucket where temporary configuration files for the AWS SSM are stored. You can use an existing bucket or create a new one.NoteThe
aws_ssm_bucket_nameparameter is ONLY used to store temporary config files. It does NOT need to be saved for use in other playbooks. Any valid existing bucket can be used. For more information on creating S3 buckets, read AWS Creating A Bucket in the AWS documentation.When you have populated the data file, it should resemble the following. The values in this file are provided as examples.
NoteThe optional values in this data file example have been removed. If you do not wish to use these optional values and want to use the default values for these variables, you 'must' also remove them for your data file like it was done in the example below. If you wish to use these optional variables, they 'must' be included in the data file and be assigned a value.
aws_restore_stack: cloud_credentials_path: ~/.aws/credentials deployment_name: AnsibleAutomationPlatform extra_vars: aws_backup_name: ansible-automation-platform-bucket-20230706T163309 aws_region: us-east-1 aws_s3_bucket: ansible-automation-platform-bucket aws_ssm_bucket_name: aap-ssm-bucket
9.2.5. Running the restore playbook
The following procedure runs the restore playbook as a container.
Procedure
To run the restore, run the command generator.
$ docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator aws_restore_stack --data-file /data/restore.yml
This generates the restore CLI command
---------------------------------------------- docker run --rm --env PLATFORM=AWS -v ~/.aws/credentials:/home/runner/.aws/credentials:ro --env ANSIBLE_CONFIG=../aws-ansible.cfg \ --env DEPLOYMENT_NAME=AnsibleAutomationPlatform --env GENERATE_INVENTORY=true --env CHECK_GENERATED_INVENTORY=false \ $IMAGE redhat.ansible_on_clouds.aws_restore_stack -e 'aws_foundation_stack_name=AnsibleAutomationPlatform \ aws_backup_name=ansible-automation-platform-bucket-20230706T163309 aws_region=us-east-1 \ aws_s3_bucket=ansible-automation-platform-bucket aws_ssm_bucket_name=aap-ssm-bucket' ===============================================
Run the generated command to trigger the restore.
$ docker run --rm --env PLATFORM=AWS -v ~/.aws/credentials:/home/runner/.aws/credentials:ro --env ANSIBLE_CONFIG=../aws-ansible.cfg \ --env DEPLOYMENT_NAME=AnsibleAutomationPlatform --env GENERATE_INVENTORY=true --env CHECK_GENERATED_INVENTORY=false \ $IMAGE redhat.ansible_on_clouds.aws_restore_stack -e 'aws_foundation_stack_name=AnsibleAutomationPlatform \ aws_backup_name=ansible-automation-platform-bucket-20230706T163309 aws_region=us-east-1 \ aws_s3_bucket=ansible-automation-platform-bucket aws_ssm_bucket_name=aap-ssm-bucket'
-
The playbook can take some time to create a restored deployment. When you have run the playbook successfully, a restored deployment is available in AWS CloudFormation matching the name provided for it in the
restore.ymlfile.
A restored deployment contains the same VPC networking setup as the original backed up deployment. If a backed up deployment was deployed within an existing VPC, its restored deployment is also deployed into that VPC. If a backed up deployment was deployed with a new VPC, its restored deployment is also deployed with a new VPC.