Regional disaster recovery (Regional-DR)

Regional-DR solution provides automated protection for block volumes, asynchronous replication, and protects business functionalities when a disaster strikes at a geographical location. In the public cloud this is similar to protecting from a region failure. For more information, see the planning guide and Regional-DR solution for OpenShift Data Foundation guide.

Multicluster monitoring in Red Hat Advanced Cluster Management console

Multicluster monitoring is a single simplified view of storage health and capacity spread across multiple clusters. This multicluster monitoring enables you to manage the storage capacity and monitor the OpenShift Data Foundation clusters from the Red Hat Advanced Cluster Management (RHACM) user interface. This monitoring capability applies to both DR and non-DR clusters. For more information, see Monitoring multicluster storage health.

Availability of Regional-DR asynchronously for CephFS volumes

Regional-DR solution now expands customer DR workload capabilities by adding Regional-DR tasks such as orchestration, failover, and relocate on CephFS volumes using the OpenShift console that is similar to the OpenShift Data Foundation experience with Regional-DR on Ceph RBD volumes. For more information, see the planning guide and Regional-DR solution guide.

async replication can no longer be set to 0

Previously, you could enter any value for Sync schedule. This meant you could set async replication to 0, which caused an error. With this update, a number input has been introduced that does not allow a value lower than 1. async replication now works correctly.

(BZ#2114501)

Deletion of Application now deletes pods and PVCs correctly

Previously, when deleting an application from the RHACM console, DRPC did not get deleted. Not deleting DRPC leads to not deleting the VRG as well as the VR. If the VRG/VR is not deleted, the PVC finalizer list will not be cleaned up, causing the PVC to stay in a Terminating state.

With this update, deleting an application from the RHACM console deletes the required dependent DRPC and related resources on the managed clusters, freeing up the PVCs as well for required garbage collection.

(BZ#2108716)

Deleting the internal VolumeReplicaitonGroup resource from where a workload failed over or relocated from no longer causes errors

Due to a bug in the disaster recovery (DR) reconciler, during deletion of the internal VolumeReplicaitonGroup resource on a managed cluster, from where a workload failed over or relocated from, a persistent volume claim (PVC) was attempted to be protected. The resulting cleanup operation did not complete and would report the PeerReady condition on the DRPlacementControl for the application to be False. This meant the application that was failed over or relocated, could not be relocated or failed over again because the DRPlacementControl resource was reporting its PeerReady condition as False.

With this update, during deletion of the internal VolumeReplicationGroup resource, a PVC is not attempted to be protected again, thereby avoiding the issue of a stalled cleanup. This results in DRPlacementControl reporting PeerReady as True post auto completion of the cleanup.

(BZ#2116605)

StorageCluster no longer goes into Error state while waiting for StorageClass creation

When an Red Hat OpenShift Data Foundation StorageCluster is created, it waits for the underlying pools to be created before the StorageClass is created. During this time, the cluster returns an error for the reconcile request until the pools are ready. Because of this error, the Phase of the StorageCluster is set to Error. With this update, this error is caught during pool creation, and the Phase of the StorageCluster is Progressing.

(BZ#2004027)

There is no longer an issue with bucket metadata when updating from RHCS 5.1 to a later version

RADOS Gateway (RGW) as shipped with Red Hat Ceph Storage (RHCS) version 5.1 inadvertently contained logic related to not-yet-GA support for dynamic bucket-index resharding in multisite replication setups. This logic was intentionally removed from RHCS 5.2. A side effect of this history is that sites which have upgraded to RHCS 5.1 cannot upgrade to RHCS 5.2, since version 5.2’s bucket metadata handling is not compatible with that of RHCS 5.1. This situation is now resolved with the upgrade to RHCS 5.3. As a result, RHCS 5.3 is able to operate on buckets created in all prior versions, including 5.1.

(BZ#2115645)

There is no longer a Pod Security Violation Alert when the ODF operator is installed

OpenShift Data Foundation version 4.11 introduced new POD Security Admission standards which give warnings on running of privileged pods. The ODF operator deployment uses a few pods which needed privileged access. Because of this, after the ODF operator was deployed, a Pod Security Violation alert started firing.

With this release, OLM now automatically labels the Namespace, which is prefixed by openshift-*, for relevant Pod security Admission standards.

(BZ#2110628)

Failover action reports RADOS block device image mount failed on the pod with RPC error still in use

Failing over a disaster recovery (DR) protected workload might result in pods using the volume on the failover cluster to be stuck in reporting RADOS block device (RBD) image is still in use. This prevents the pods from starting up for a long duration (upto several hours).

(BZ#2007376)

Failover action reports RADOS block device image mount failed on the pod with RPC error fsck

Failing over a disaster recovery (DR) protected workload may result in pods not starting with volume mount errors that state the volume has file system consistency check (fsck) errors. This prevents the workload from failing over to the failover cluster.

(BZ#2021460)

Creating an application namespace for the managed clusters

Application namespace needs to exist on RHACM managed clusters for disaster recovery (DR) related pre-deployment actions and hence is pre-created when an application is deployed at the RHACM hub cluster. However, if an application is deleted at the hub cluster and its corresponding namespace is deleted on the managed clusters, they reappear on the managed cluster.

Workaround: openshift-dr maintains a namespace manifestwork resource in the managed cluster namespace at the RHACM hub. These resources need to be deleted after the application deletion. For example, as a cluster administrator, execute the following command on the hub cluster: oc delete manifestwork -n <managedCluster namespace> <drPlacementControl name>-<namespace>-ns-mw.

(BZ#2059669)

RBD mirror scheduling is getting stopped for some images

The Ceph manager daemon gets blocklisted due to different reasons, which causes the scheduled RBD mirror snapshot from being triggered on the cluster where the image(s) are primary. All RBD images that are mirror enabled (hence DR protected) do not list a schedule when examined using rbd mirror snapshot schedule status -p ocs-storagecluster-cephblockpool, and hence are not actively mirrored to the peer site.

Workaround: Restart the Ceph manager deployment, on the managed cluster where the images are primary, to overcome the blocklist against the currently running instance, this can be done by scaling down and then later scaling up the ceph manager deployment as follows:

$ oc -n openshift-storage scale deployments/rook-ceph-mgr-a  --replicas=0
$ oc -n openshift-storage scale deployments/rook-ceph-mgr-a  --replicas=1

Result: Images that are DR enabled and denoted as primary on a managed cluster start reporting mirroring schedules when examined using rbd mirror snapshot schedule status -p ocs-storagecluster-cephblockpool

(BZ#2067095)

ceph df reports an invalid MAX AVAIL value when the cluster is in stretch mode

When a crush rule for a Red Hat Ceph Storage cluster has multiple "take" steps, the ceph df report shows the wrong maximum available size for the map. The issue will be fixed in an upcoming release.

(BZ#2100920)

Ceph does not recognize the global IP assigned by Globalnet

Ceph does not recognize global IP assigned by Globalnet, so disaster recovery solution cannot be configured between clusters with overlapping service CIDR using Globalnet. Due to this disaster recovery solution does not work when service CIDR overlaps.

(BZ#2102397)

Both the DRPCs protect all the persistent volume claims created on the same namespace

The namespaces that host multiple disaster recovery (DR) protected workloads, protect all the persistent volume claims (PVCs) within the namespace for each DRPlacementControl resource in the same namespace on the hub cluster that does not specify and isolate PVCs based on the workload using its spec.pvcSelector field.

This results in PVCs, that match the DRPlacementControl spec.pvcSelector across multiple workloads. Or, if the selector is missing across all workloads, replication management to potentially manage each PVC multiple times and cause data corruption or invalid operations based on individual DRPlacementControl actions.

Workaround: Label PVCs that belong to a workload uniquely, and use the selected label as the DRPlacementControl spec.pvcSelector to disambiguate which DRPlacementControl protects and manages which subset of PVCs within a namespace. It is not possible to specify the spec.pvcSelector field for the DRPlacementControl using the user interface, hence the DRPlacementControl for such applications must be deleted and created using the command line.

Result: PVCs are no longer managed by multiple DRPlacementControl resources and do not cause any operation and data inconsistencies.

(BZ#2111163)

MongoDB pod is in CrashLoopBackoff because of permission errors reading data in cephrbd volume

The OpenShift projects across different managed clusters have different security context constraints (SCC), which specifically differ in the specified UID range and/or FSGroups. This leads to certain workload pods and containers failing to start post failover or relocate operations within these projects, due to filesystem access errors in their logs.

Workaround: Ensure workload projects are created on all managed clusters with the same project-level SCC labels, allowing them to use the same filesystem context when failed over or relocated. Pods will no longer fail post-DR actions on filesystem-related access errors.

(BZ#2114573)

Application is stuck in Relocating state during relocate

Multicloud Object Gateway allowed multiple persistent volume (PV) objects of the same name or namespace to be added to the S3 store on the same path. Due to this, Ramen does not restore the PV because it detected multiple versions pointing to the same claimRef.

Workaround: Use S3 CLI or equivalent to clean up the duplicate PV objects from the S3 store. Keep only the one that has a timestamp closer to the failover or relocate time.

Result: The restore operation will proceed to completion and the failover or relocate operation proceeds to the next step.

(BZ#2120201)

Application is stuck in a FailingOver state when a zone is down

At the time of a failover or relocate, if none of the s3 stores are reachable then the failover or relocate process hangs. If the DR logs indicate that the S3 store is not reachable, then troubleshooting and getting the s3 store operational will allow the DR to proceed with the failover or relocate operation.

(BZ#2121680)

PeerReady state is set to true when a workload is failed over or relocated to the peer cluster until the cluster from where it was failed over or relocated from is cleaned up

After a disaster recovery (DR) action is initiated, the PeerReady condition is initially set to true for the duration when the workload is failed over or relocated to the peer cluster. After this it is set to false until the cluster from where it was failed over or relocated from is cleaned up for future actions. A user looking at DRPlacementControl status conditions for future actions may recognize this intermediate PeerReady state as a peer is ready for action and perform the same. This will result in the operation pending or failing and may require user intervention to recover from.

Workaround: Examine both Available and PeerReady states before performing any actions. Both should be true for a healthy DR state for the workload. Actions performed when both states are true will result in the requested operation progressing

(BZ#2138855)

Disaster recovery workloads remain stuck when deleted

When deleting a workload from a cluster, the corresponding pods might not terminate with events such as FailedKillPod. This might cause delay or failure in garbage collecting dependent DR resources such as the PVC, VolumeReplication, and VolumeReplicationGroup. It would also prevent a future deployment of the same workload to the cluster as the stale resources are not yet garbage collected.

Workaround: Reboot the worker node on which the pod is currently running and stuck in a terminating state. This results in successful pod termination and subsequently related DR API resources are also garbage collected.

(BZ#2159791)

Blocklisting can lead to Pods stuck in an error state

Blocklisting due to either network issues or a heavily overloaded or imbalanced cluster with huge tail latency spikes. Because of this, Pods get stuck in CreateContainerError with the message Error: relabel failed /var/lib/kubelet/pods/cb27938e-f66f-401d-85f0-9eb5cf565ace/volumes/kubernetes.io~csi/pvc-86e7da91-29f9-4418-80a7-4ae7610bb613/mount: lsetxattr /var/lib/kubelet/pods/cb27938e-f66f-401d-85f0-9eb5cf565ace/volumes/kubernetes.io~csi/pvc-86e7da91-29f9-4418-80a7-4ae7610bb613/mount/#ib_16384_0.dblwr: read-only file system.

Workaround: Reboot the node to which these pods are scheduled and failing by following these steps:

(BZ#2094320)

Poor performance of the stretch clusters on CephFS

Workloads with many small metadata operations might exhibit poor performance because of the arbitrary placement of metadata server (MDS) on multi-site Data Foundation clusters.

(BZ#1982116)

SELinux relabelling issue with a very high number of files

When attaching volumes to pods in Red Hat OpenShift Container Platform, the pods sometimes do not start or take an excessive amount of time to start. This behavior is generic and it is tied to how SELinux relabelling is handled by the Kubelet. This issue is observed with any filesystem based volumes having a very high file counts. In OpenShift Data Foundation, the issue is seen when using CephFS based volumes with a very high number of files. There are different ways to workaround this issue. Depending on your business needs you can choose one of the workarounds from the knowledgebase solution https://access.redhat.com/solutions/6221251.

(Jira#3327)

OpenShift Data Foundation dashboard crashes after upgrade

When OpenShift Container Platform and OpenShift Data Foundation are upgraded, the Data Foundation dashboard under the Storage section crashes with a "404: Page not found" error when dashboard link is clicked. This is because the pop-up that refreshes the console does not appear.

Workaround: Perform a hard refresh of the console. This brings back the dashboard and it will no longer crash.

(BZ#2157876)