OpenShift APIs for Data Protection (OADP) FAQ

An important part of any platform used to host business and user workloads is data protection. Data protection may include operations including on-demand backup, scheduled backup and restore. These operations allow the objects within a cluster to be backed up to a storage provider, either locally or on a public cloud, and restore that cluster from the backup in the event of a failure or scheduled maintenance.

Red Hat has created OpenShift API for Data Protection, or OADP, for this purpose. OADP brings an API to the OpenShift Container Platform that Red Hat partners can leverage in creating a disaster recovery and data protection solution.

Frequently Asked Questions

What is OADP?

OADP (OpenShift APIs for Data Protection) is an operator that Red Hat has created to create backup and restore APIs in the OpenShift cluster.
You can read more about OADP in the following links:
OADP documentation
OADP blog posts

What APIs does the operator provide?

OADP provides the following APIs:

• Backup
• Restore
• Schedule
• BackupStorageLocation
• VolumeSnapshotLocation

Red Hat has not added, removed or modified any of the APIs as documented in the Velero upstream project. The Velero site has more details on the Velero API Types.

Is there an upstream project for OADP?

Yes. The OADP operator is developed in the oadp-operator upstream project.

What is the support status of the OADP operator?

Please refer to the OADP support policy

Is OADP a full end-to-end data protection solution?

OpenShift API for Data Protection (OADP) features provide options for backing up and restoring applications. You can find more detail regarding OADP's features in our documentation

What data can OADP protect?

OADP provides APIs to backup and restore OpenShift cluster resources (yaml files), internal images and persistent volume data.

What is the OADP operator installing?

The OADP operator will install Velero, and OpenShift plugins for Velero to use, for backup and restore operations.

Does OADP support CSI snapshots?

Yes, please refer to the documentation

How does OADP's Restic option manage incremental backups?

Velero looks for the most recent Restic backup for the current volume that is in the same backup location. If one is found, Velero then passes the Restic snapshot ID to the Restic CLI. This means that Restic will only retrieve files that have changed since the most recent backup, and uses the existing files for the rest.

Is there a recorded demo of OADP?

Yes! The OADP team did a great presentation and demonstration of OADP Check it out here. The first half is a very informative Q&A, followed by the demo.

What versions of OpenShift Container Platform can OADP be installed?

The OADP 1.0, and OADP 1.1 operators can be found within the embedded OperatorHub in the OpenShift web console, and are fully supported. Please refer to our support policy

Are there plans to include a data mover with OADP?

The data mover is in tech preview with OADP 1.1 and will be fully supported in OADP 1.2 which is expected to release early in 2023.

How do I determine the version of Velero OADP installed?

After OADP installation, the velero deployment it will contain the tag of the image. If you install OADP with the default config you will be using upstream tagged images with the version called out in the deployment. You can also check out the version matrix.

Where can I find examples of using OADP APIs for backup/restore?

The OADP operator page in the upstream oadp-operator project has examples that walk through usage.

Using S3 compatible storage that does not have an associated region

There are S3 compatible storage implementations that do not require a region to be setup. In these cases simply substitute a valid aws region like "us-east-1" in the DPA yaml configuration. For example the OADP with MCG documentation. Reference the velero issue

• A user should provide the:
  • s3Url: https://foo/storage
  • region: us-east-1

 backupLocations:
    - velero:
        config:
          profile: "default"
          region: us-east-1
          s3Url:  https://foo/storage <s3 endpoint>
          insecureSkipTLSVerify: "true"
          s3ForcePathStyle: "true"
        provider: aws
        default: true
        credential:
          key: cloud
          name: cloud-credentials 
        objectStorage:
          bucket: <bucket_name> 
          prefix: <prefix> 

Can OADP restore routes with base domain from the restore cluster?

OADP will restore routes with base domain from the restore cluster when the route being restored is a generated route

A generated route is a route that do not specify route.spec.host at creation and let OpenShift generates the hostname for the route. Generated route will have annotation "openshift.io/host.generated: 'true'". If you manually add this annotation to a route then unexpected behavior may occur during restore.

Can I turn off internal registry image backup?

If you experienced issues during backup or restore due to errors related to internal registry image (imagestreams) backup you can turn off image backup functionality like so in the DataProtectionApplication spec.

spec:
  backupImages: false // set this to disable image backup/restore

Set a backup to expire

When you create a backup, you can specify a TTL (time to live) by adding the flag --ttl . If Velero sees that an existing backup resource is expired, it removes:

• The backup resource
• The backup file from cloud object storage
• All PersistentVolume snapshots
• All associated Restores

Upstream Documentation with Details

Issues restoring an OADP backup: application unable to access data

• UID / GID range may have changed on the cluster where the application has been restored.
  • OADP does not backup and restore OpenShift UID/GID range metadata. If the backed application requires a specific UUID please ensure the range is available when restored. For more information on OpenShift's UID/GID range (please reference)[https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids]

VolSync 0.6.x and OADP 1.1.x annotation required

• For users that have upgraded to VolSync version >= 0.6.0 please note an annotation is required on the openshift-adp namespace for datamover operations to continue to work.
  Execute the following command to annotate the openshift-adp namespace with `volsync.backube/privileged-movers='true'

oc annotate --overwrite namespace/openshift-adp volsync.backube/privileged-movers='true'

pod volume backup failed: running Restic backup, stderr=Fatal: unable to open config file: blob.GetProperties: storage

Errors such as

pod volume backup failed: running Restic backup, stderr=Fatal: unable to open config file: blob.GetProperties: storage: service returned error: StatusCode=404, ErrorCode=404 The specified container does not exist

occurs when you delete restic folder in object storage. Try following:

oc get resticrepositories -n openshift-adp

If you see ones that corresponds to your object storage, delete it so velero recreates another restic repository.

Backing up data from one cluster and restoring to another cluster

• To successfully backup and restore data to two different clusters please ensure that in your DPA config on both clusters that:
  • The backup store location (BSL) and volume snapshot location have the same names and paths to restore resources to another cluster.
  • The same object storage location credentials must be shared across the clusters
  • The upstream Velero documentation is helpful in the case.
  • For Volume backup and restore please refer to the latest OADP documentation and the datamover sections.

Can OADP modify nodeSelector during restore?

• Not at this time. You can manually modify nodeSelectors after restore. OADP team or the community could implement in a future release a RestoreItemAction plugin that does this.

Disaster recovery - Using Schedules and Read-Only Backup Storage Locations

During disaster recovery, it is recommended that you set your backup location accessMode to ReadOnly to prevent addition/deletions to the backup storage location during the restore process.

You would set accessMode to readOnly like so in the DataProtectionApplication spec

...
spec:
  backupLocations:
    - velero:
        accessMode: ReadOnly
...

Proceed to restore from backup.

OADP Restore fails with ArgoCD

If ArgoCD is being used during a restore, it is possible to see the restore fail. This could be caused by a label used by ArgoCD app.kubernetes.io/instance. This label is used to identify which resources ArgoCD needs to manage, which can create conflict with OADP managing resources on restore.

To resolve this issue, you can set .spec.resourceTrackingMethod on the ArgoCD yaml to annotation+label or annotation. If issues still persist, then disable ArgoCD before restore, and then enable again once restore completes.

Please do let us know when the errors occur so we work to resolve the issue.

Can I install OADP into multiple OpenShift Projects to enable project owners?

We will be providing additional documentation to cover this use in the near future, however it is worth noting here. It is possible to install OADP into multiple namespaces to enable project owners to manage their own OADP instance. The deployments of OADP must all be at the same version, installing different versions of OADP on the same cluster is NOT supported.

• It is required that each individual deployment of OADP have a unique set of credentials and BackupStorageLocation configuration. The workflow has been validated with Restic and CSI.
• It is worth noting that by default each OADP deployment has cluster level access across namespaces. We recommend that OCP administrators review the security and RBAC settings carefully.

I am trying to use OADP with a ROSA cluster, and need help

We have recently updated the documentation for installing and configuring OADP with ROSA clusters. Please see the documentation here