Issue

Preface:

Because the ODF Operator and included resources are supported by the ODF team and the OADP Operator is supported by the Shift Storage team, the author of this solution collaborated with various teams/colleagues to produce this solution.

This solution was tested on OCP v4.17.17 and ODF v4.17.5 by storing objects in buckets, backing up with the OADP application, and deleting the db-noobaa-db-pg-0 PVC. This caused NooBaa to enter a Configuring phase and resulted in a state of not being able to access the bucket contents via s3. Once the db was restored, the objects were once again accessible, and read/write operations for both new/old objects was made possible again.

time="2025-01-23T19:46:45Z" level=error msg="⚠️  RPC: system.read_system() Response Error: Code=UNAUTHORIZED Message=account not found 661ff9fa3e37850029df061a"

Accidental db-noobaa-db-pg-0 PVC deletions are one of the most common instances of data loss regarding NooBaa. Once NooBaa entered this state, a restore was created in OADP using the backup ID. Performing the steps in this solution resulted in a successful restoration of NooBaa back to a Ready phase.

Best Practices:

The best practice for storing ODF/NooBaa (MCG) backups is to store them OUTSIDE of the cluster. For example, a NooBaa bucket in a different OCP cluster can store the backups or an AWS s3 bucket. Because not all customers have access to external environments for s3 backup storage (separate cluster, cloud/AWS), or if they're in a disconnected environment this solution was written to use Ceph's internal Rados Gateway (RGW) to store the backups.

Although this is not the best practice to use an internal object service for backup storage, the vast majority of observed data losses reveal that if the backups were simply stored in an internal s3 solution such RGW because an external solution was not available, a restore would've been possible. It's not recommended to deviate from best practices, however, all customers (including disconnected environments) have access to RGW, hence the reason why this solution is showing RGW (greater reach).

In Bare Metal/VM environments, RGW is created by default during the creation of the storagesystem/storagecluster. In other environments where the cephobjectstore (RGW) resource is not created, it can be manually created. See Enable use of the RGW on an OCS internal deployment for more information.

If AWS/an s3 compatible storage is available, just substitute the bucket name, external route (endpoint), access, and access secret key to reflect the backup destination bucket in the DataProtectionApplication YAML.