Configuring OpenShift Data Foundation for Regional-DR with Advanced Cluster Management
DEVELOPER PREVIEW: Instructions about setting up OpenShift Data Foundation with Regional-DR capabilities. This solution is a Developer Preview feature and is not intended to be run in production environments.
Abstract
Making open source more inclusive
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.
Providing feedback on Red Hat documentation
We appreciate your input on our documentation. Do let us know how we can make it better. To give feedback:
For simple comments on specific passages:
- Make sure you are viewing the documentation in the Multi-page HTML format. In addition, ensure you see the Feedback button in the upper right corner of the document.
- Use your mouse cursor to highlight the part of text that you want to comment on.
- Click the Add Feedback pop-up that appears below the highlighted text.
- Follow the displayed instructions.
For submitting more complex feedback, create a Bugzilla ticket:
- Go to the Bugzilla website.
- In the Component section, choose documentation.
- Fill in the Description field with your suggestion for improvement. Include a link to the relevant part(s) of documentation.
- Click Submit Bug.
Chapter 1. Introduction to Regional-DR
Disaster recovery is the ability to recover and continue business critical applications from natural or human created disasters. It is the overall business continuance strategy of any major organization as designed to preserve the continuity of business operations during major adverse events.
Regional-DR capability provides volume persistent data and metadata replication across sites that are geographically dispersed. In the public cloud these would be similar to protecting from a region failure. Regional-DR ensures business continuity during the unavailability of a geographical region, accepting some loss of data in a predictable amount. This is usually expressed at Recovery Point Objective (RPO) and Recovery Time Objective (RTO).
- RPO is a measure of how frequently you take backups or snapshots of persistent data. In practice, the RPO indicates the amount of data that will be lost or need to be reentered after an outage.
- RTO is the amount of downtime a business can tolerate. The RTO answers the question, “How long can it take for our system to recover after we were notified of a business disruption?”
The intent of this guide is to detail the steps and commands necessary for configuring your infrastructure for enabling disaster recovery.
1.1. Components of Regional-DR solution
Regional-DR is composed of Red Hat Advanced Cluster Management for Kubernetes (RHACM) and OpenShift Data Foundation components to provide application and data mobility across OpenShift Container Platform clusters.
Red Hat Advanced Cluster Management for Kubernetes
Red Hat Advanced Cluster Management provides the ability to manage multiple clusters and application lifecycles. Hence, it serves as a control plane in a multi-cluster environment.
RHACM is split into two parts:
- RHACM Hub: includes component that run on the multi-cluster control plane.
- Managed clusters: includes components that run on the clusters that are managed.
For more information about this product, see RHACM documentation and the RHACM “Managing Applications” documentation.
OpenShift Data Foundation
OpenShift Data Foundation provides the ability to provision and manage storage for stateful applications in an OpenShift Container Platform cluster.
OpenShift Data Foundation is backed by Ceph as the storage provider, whose lifecycle is managed by Rook in the OpenShift Data Foundation component stack. Ceph-CSI provides the provisioning and management of Persistent Volumes for stateful applications.
OpenShift Data Foundation stack is now enhanced with the following abilities:
- Enable pools for mirroring
- Automatically mirror images across RBD block pools
- Provides csi-addons to manage per Persistent Volume Claim (PVC) mirroring
OpenShift DR
OpenShift DR is a disaster recovery orchestrator for stateful applications across a set of peer OpenShift clusters which are deployed and managed using RHACM and provides cloud-native interfaces to orchestrate the life-cycle of an application’s state on Persistent Volumes. These include:
- Protecting an application state relationship across OpenShift clusters
- Failing over an application’s state to a peer cluster
- Relocate an application’s state to the previously deployed cluster
OpenShift DR is split into three components:
ODF Multicluster Orchestrator: Installed on the multi-cluster control plane (RHACM Hub), it also performs the following actions:
- Creates a bootstrap token and exchanges this token between the managed clusters.
-
Enables mirroring for the default
CephBlockPoolon the managed clusters. - Creates an object bucket using Multicloud Object Gateway (MCG) on each managed cluster for PVC and PV metadata.
-
Creates a Secret for each new object bucket that has the keys for bucket access on the Hub cluster in the
openshift-dr-systemproject. -
Creates a VolumeReplicationClass on the Primary managed cluster and the Secondary managed cluster for each
schedulingIntervals(e.g. 5m, 15m, 30m). -
Modifies the
ramen-hub-operator-configConfigMap on the Hub cluster and adds the s3StoreProfiles entries.
- OpenShift DR Hub Operator: Installed on the hub cluster to manage failover and relocation for applications.
- OpenShift DR Cluster Operator: Installed on each managed cluster to manage the lifecycle of all PVCs of an application.
1.2. Regional-DR deployment workflow
This section provides an overview of the steps required to configure and deploy Regional-DR capabilities using OpenShift Data Foundation version 4.10 and RHACM latest version across two distinct OpenShift Container Platform clusters. In addition to two managed clusters, a third OpenShift Container Platform cluster will be required to deploy the Advanced Cluster Management.
To configure your infrastructure, perform the below steps in the order given:
- Ensure you meet each of the Regional-DR requirements which includes RHACM operator installation, creation or importing of OpenShift Container Platform into RHACM hub and network configuration. See Requirements for enabling Regional-DR.
- Install OpenShift Data Foundation 4.10 on Primary and Secondary managed clusters. See Installing OpenShift Data Foundation on managed clusters.
- Install the Openshift DR Hub Operator on the Hub cluster. See Installing OpenShift DR Hub Operator on Hub cluster.
- Configure multisite storage replication by creating the mirroring relationship between two OpenShift Data Foundation managed clusters. See Configuring multisite storage replication.
-
Create a mirroring StorageClass resource on each managed cluster that supports new
imageFeaturesfor block volumes that have mirroring enabled. See Creating mirroring StorageClass resource. Create the DRPolicy resource on the hub cluster which is used to deploy, failover, and relocate the workloads across managed clusters. See Creating Disaster Recovery Policy on Hub cluster.
NoteThere can be more than a single policy.
- Enable automatic installation of the OpenShift DR Cluster operator and automatic transfer of S3 secrets on the managed clusters. For instructions, see Enabling automatic install of OpenShift DR cluster operator and Enabling automatic transfer of S3 secrets on managed clusters.
- Create a sample application using RHACM console for testing failover and relocation testing. For instructions, see Creating sample application, application failover and relocating an application between managed clusters.
Chapter 2. Requirements for enabling Regional-DR
Disaster Recovery features supported by Red Hat OpenShift Data Foundation require all of the following prerequisites in order to successfully implement a Disaster Recovery solution:
Subscription requirements
- A valid Red Hat OpenShift Data Foundation Advanced entitlement
- A valid Red Hat Advanced Cluster Management for Kubernetes subscription
To know how subscriptions for OpenShift Data Foundation work, see knowledgebase article on OpenShift Data Foundation subscriptions.
You must have three OpenShift clusters that have network reachability between them:
- Hub cluster where Advanced Cluster Management for Kubernetes (RHACM operator), ODF Multicluster Orchestrator and OpenShift DR Hub controllers are installed.
- Primary managed cluster where OpenShift Data Foundation, OpenShift DR Cluster controller, and applications are installed.
- Secondary managed cluster where OpenShift Data Foundation, OpenShift DR Cluster controller, and applications are installed.
Ensure that RHACM operator and MultiClusterHub is installed on the Hub cluster. See RHACM installation guide for instructions.
- Login to the RHACM console using your OpenShift credentials.
Find the Route that has been created for the Advanced Cluster Manager console:
$ oc get route multicloud-console -n open-cluster-management -o jsonpath --template="https://{.spec.host}/multicloud/clusters{'\n'}"Example Output:
https://multicloud-console.apps.perf3.example.com/multicloud/clusters
After logging in using your OpenShift credentials, you should see your local cluster imported.
- Ensure that you have either imported or created the Primary managed cluster and the Secondary managed clusters using the RHACM console.
The managed clusters must have non-overlapping networks.
To connect the managed OpenShift cluster and service networks using the Submariner add-ons, you need to validate that the two clusters have non-overlapping networks by running the following commands for each of the managed clusters.
$ oc get networks.config.openshift.io cluster -o json | jq .spec
Example output for
cluster1(for example,ocp4perf1):{ "clusterNetwork": [ { "cidr": "10.5.0.0/16", "hostPrefix": 23 } ], "externalIP": { "policy": {} }, "networkType": "OpenShiftSDN", "serviceNetwork": [ "10.15.0.0/16" ] }Example output for
cluster2(for example,ocp4perf2):{ "clusterNetwork": [ { "cidr": "10.6.0.0/16", "hostPrefix": 23 } ], "externalIP": { "policy": {} }, "networkType": "OpenShiftSDN", "serviceNetwork": [ "10.16.0.0/16" ] }For more information, see Submariner add-ons documentation.
-
Ensure that the Managed clusters can connect using
Submariner add-ons. After identifying and ensuring that the cluster and service networks have non-overlapping ranges, install theSubmariner add-onsfor each managed cluster using the RHACM console andCluster sets. For instructions, see Submariner documentation.
Chapter 3. Installing OpenShift Data Foundation on managed clusters
Procedure
Install OpenShift Data Foundation version 4.10 on each of the managed clusters.
For information about the OpenShift Data Foundation deployment, refer to your infrastructure specific deployment guides (for example, AWS, VMware, Bare metal, Azure).
Validate the successful deployment on each managed cluster with the following command:
$ oc get storagecluster -n openshift-storage ocs-storagecluster -o jsonpath='{.status.phase}{"\n"}'and for the Multicloud Object Gateway (MCG):
$ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'If the status result is
Readyfor both queries on the Primary managed cluster and the Secondary managed cluster, then continue on to enabling mirroring on the managed clusters.
Chapter 4. Installing OpenShift DR Hub Operator on Hub cluster
Procedure
- On the Hub cluster, navigate to OperatorHub and use the search filter for OpenShift DR Hub Operator.
-
Follow the screen instructions to Install the operator into the project
openshift-dr-system. Verify that the operator Pod is in
Runningstate using the following command:$ oc get pods -n openshift-dr-system
Example output:
NAME READY STATUS RESTARTS AGE ramen-hub-operator-898c5989b-96k65 2/2 Running 0 4m14s
Chapter 5. Configuring multisite storage replication
Mirroring or replication is enabled on a per CephBlockPool basis within peer managed clusters and can then be configured on a specific subset of images within the pool. The rbd-mirror daemon is responsible for replicating image updates from the local peer cluster to the same image in the remote cluster.
These instructions detail how to create the mirroring relationship between two OpenShift Data Foundation managed clusters.
5.1. Installing OpenShift Data Foundation Multicluster Orchestrator
OpenShift Data Foundation Multicluster Orchestrator is a controller that is installed from OpenShift Container Platform’s OperatorHub on the Hub cluster. This Multicluster Orchestrator controller, along with the MirrorPeer custom resource, creates a bootstrap token and exchanges this token between the managed clusters.
Procedure
- Navigate to OperatorHub on the Hub cluster and use the keyword filter to search for ODF Multicluster Orchestrator.
- Click ODF Multicluster Orchestrator tile.
Keep all default settings and click Install.
The operator resources are installed in
openshift-operatorsand available to all namespaces.Verify that the ODF Multicluster Orchestrator has installed successfully.
- Validate successful installation by having the ability to select View Operator.
Verify that the operator Pod are in
Runningstate.$ oc get pods -n openshift-operators
Example output:
NAME READY STATUS RESTARTS AGE odfmo-controller-manager-65946fb99b-779v8 1/1 Running 0 5m3s
5.2. Creating mirror peer on hub cluster
Mirror Peer is a cluster-scoped resource to hold information about the managed clusters that will have a peer-to-peer relationship.
Prerequisites
- Ensure that ODF Multicluster Orchestrator is installed on the Hub cluster.
- You must have only two clusters per Mirror Peer.
-
Ensure that each cluster has uniquely identifiable cluster names such as
ocp4perf1andocp4perf2.
Procedure
Click ODF Multicluster Orchestrator to view the operator details.
You can also click View Operator after the Multicluster Orchestrator is installed successfully.
- Click on Mirror Peer API Create instance and then select YAML view.
Copy and save the following YAML to filename
mirror-peer.yamlafter replacing <cluster1> and <cluster2> with the correct names of your managed clusters in the RHACM console.apiVersion: multicluster.odf.openshift.io/v1alpha1 kind: MirrorPeer metadata: name: mirrorpeer-<cluster1>-<cluster2> spec: items: - clusterName: <cluster1> storageClusterRef: name: ocs-storagecluster namespace: openshift-storage - clusterName: <cluster2> storageClusterRef: name: ocs-storagecluster namespace: openshift-storage manageS3: true schedulingIntervals: - 5m - 15mNoteThe time values (e.g. 5m) for
schedulingIntervalswill be used to configure the desired interval for replicating persistent volumes. These values can be mapped to your Recovery Point Objective (RPO) for critical applications. Modify the values inschedulingIntervalsto be correct for your application requirements. The minimum value is1mand the default is5m.-
Copy the contents of your unique
mirror-peer.yamlfile into theYAML view. You must completely replace the original content. - Click Create at the bottom of the YAML view screen.
-
Verify that you can view Phase status as
ExchangedSecretbefore proceeding.
5.3. Validating Ceph mirroring on managed clusters
Perform the following validations on the Primary managed cluster and the Secondary managed cluster to check Ceph mirroring is active:
Verify that
mirroringis enabled on the defaultCeph block pool.$ oc get cephblockpool -n openshift-storage -o=jsonpath='{.items[?(@.metadata.ownerReferences[*].kind=="StorageCluster")].spec.mirroring.enabled}{"\n"}'Example output:
true
Verify that the
rbd-mirrorpod is up and running.$ oc get pods -o name -l app=rook-ceph-rbd-mirror -n openshift-storage
Example output:
pod/rook-ceph-rbd-mirror-a-6486c7d875-56v2v
Check the status of the
daemonhealth to ensure it is OK.$ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'Example output:
{"daemon_health":"OK","health":"OK","image_health":"OK","states":{}}NoteIt could take up to 10 minutes for the daemon_health and health fields to change from
WarningtoOK. If the status does not become OK after 10 minutes then use the Advanced Cluster Manager console to verify that thesubmariner add-onconnection is still in a healthy state.Verify that VolumeReplicationClass is created on the Primary managed cluster and the Secondary managed cluster for each schedulingIntervals listed in the MirrorPeer (e.g. 5m, 15m).
$ oc get volumereplicationclass
Example output:
NAME PROVISIONER rbd-volumereplicationclass-1625360775 openshift-storage.rbd.csi.ceph.com rbd-volumereplicationclass-539797778 openshift-storage.rbd.csi.ceph.com
NoteThe
VolumeReplicationClassis used to specify themirroringModefor each volume to be replicated as well as how often a volume or image is replicated (for example, every 5 minutes) from the local cluster to the remote cluster.
5.4. Validating object buckets and S3StoreProfiles
Perform the following validations on the Primary managed cluster and the Secondary managed cluster to check Ceph mirroring is active.
Procedure
Verify that there is a new Object Bucket Claim and corresponding Object Bucket in the Primary managed cluster and the Secondary managed cluster in the
openshift-storagenamespace.$ oc get obc,ob -n openshift-storage
Example output:
NAME STORAGE-CLASS PHASE AGE objectbucketclaim.objectbucket.io/odrbucket-21eb5332f6b6 openshift-storage.noobaa.io Bound 13m NAME STORAGE-CLASS CLAIM-NAMESPACE CLAIM-NAME RECLAIM-POLICY PHASE AGE objectbucket.objectbucket.io/obc-openshift-storage-odrbucket-21eb5332f6b6 openshift-storage.noobaa.io Delete Bound 13m
Verify that there are two new Secrets in the Hub cluster
openshift-dr-systemnamespace that contain the access and secret key for each new Object Bucket Class.$ oc get secrets -n openshift-dr-system | grep Opaque
Example output:
8b3fb9ed90f66808d988c7edfa76eba35647092 Opaque 2 16m af5f82f21f8f77faf3de2553e223b535002e480 Opaque 2 16m
The OBC and Secrets are written in the ConfigMap
ramen-hub-operator-configon the Hub cluster in the newly createds3StoreProfilessection.$ oc get cm ramen-hub-operator-config -n openshift-dr-system -o yaml | grep -A 14 s3StoreProfiles
Example output:
s3StoreProfiles: - s3Bucket: odrbucket-21eb5332f6b6 s3CompatibleEndpoint: https://s3-openshift-storage.apps.perf2.example.com s3ProfileName: s3profile-ocp4perf2-ocs-storagecluster s3Region: noobaa s3SecretRef: name: 8b3fb9ed90f66808d988c7edfa76eba35647092 namespace: openshift-dr-system - s3Bucket: odrbucket-21eb5332f6b6 s3CompatibleEndpoint: https://s3-openshift-storage.apps.perf1.example.com s3ProfileName: s3profile-ocp4perf1-ocs-storagecluster s3Region: noobaa s3SecretRef: name: af5f82f21f8f77faf3de2553e223b535002e480 namespace: openshift-dr-systemNoteRecord the names of the
s3ProfileName. They will be used in the DRPolicy resource.
Chapter 6. Creating mirroring StorageClass resource
You must create the block volumes with mirroring enabled using a new StorageClass that has additional imageFeatures required to enable faster image replication between managed clusters. The new features are exclusive-lock, object-map, and fast-diff. The default OpenShift Data Foundation StorageClass ocs-storagecluster-ceph-rbd does not include these features.
This resource must be created on the Primary managed cluster and the Secondary managed cluster.
Procedure
Save the following YAML to filename
ocs-storagecluster-ceph-rbdmirror.yaml.allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ocs-storagecluster-ceph-rbdmirror parameters: clusterID: openshift-storage csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage imageFeatures: layering,exclusive-lock,object-map,fast-diff imageFormat: "2" pool: ocs-storagecluster-cephblockpool provisioner: openshift-storage.rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate
Create the file on both the managed clusters.
$ oc create -f ocs-storagecluster-ceph-rbdmirror.yaml
Example output:
storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbdmirror created
Chapter 7. Configuring SSL access between S3 endpoints
Configure network (SSL) access between the s3 endpoints so that metadata can be stored on the alternate cluster in a MCG object bucket using a secure transport protocol and in the Hub cluster for verifying access to the object buckets.
If all of your OpenShift clusters are deployed using a signed and valid set of certificates for your environment then this section can be skipped.
Procedure
Extract the ingress certificate for the Primary managed cluster and save the output to
primary.crt.$ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crtExtract the ingress certificate for the Secondary managed cluster and save the output to
secondary.crt.$ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crtCreate a new ConfigMap to hold the remote cluster’s certificate bundle with filename
cm-clusters-crt.yamlon the Primary managed cluster, Secondary managed cluster, and the Hub cluster.NoteThere could be more or less than three certificates for each cluster as shown in this example file. Also, ensure that the certificate contents are correctly indented after you copy and paste from the
primary.crtandsecondary.crtfiles that were created before.apiVersion: v1 data: ca-bundle.crt: | -----BEGIN CERTIFICATE----- <copy contents of cert1 from primary.crt here> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <copy contents of cert2 from primary.crt here> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <copy contents of cert3 primary.crt here> -----END CERTIFICATE---- -----BEGIN CERTIFICATE----- <copy contents of cert1 from secondary.crt here> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <copy contents of cert2 from secondary.crt here> -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- <copy contents of cert3 from secondary.crt here> -----END CERTIFICATE----- kind: ConfigMap metadata: name: user-ca-bundle namespace: openshift-configCreate the ConfigMap file on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.
$ oc create -f cm-clusters-crt.yaml
Example output:
configmap/user-ca-bundle created
ImportantFor the Hub cluster to verify access to the object buckets using the DRPolicy resource, the same ConfigMap
cm-clusters-crt.yamlmust also be created on the Hub cluster.Patch default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.
$ oc patch proxy cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"user-ca-bundle"}}}'Example output:
proxy.config.openshift.io/cluster patched
Chapter 8. Creating Disaster Recovery Policy on Hub cluster
OpenShift DR uses Disaster Recovery Policy (DRPolicy) resources (cluster scoped) on the RHACM hub cluster to deploy, failover, and relocate workloads across managed clusters.
Prerequisites
- Ensure that there is a set of two clusters, which are peered for storage level replication and that CSI Volume Replication is enabled.
- Ensure that there is a scheduling interval that determines at what frequency data replication is performed which also serves as a coarse grained Recovery Point Objective (RPO) for the workload using the DRPolicy.
- Ensure that each cluster in the policy is assigned a S3 profile name, which is configured using the ConfigMap of the OpenShift DR cluster and hub operators.
Procedure
-
On the Hub cluster, navigate to Installed Operators in the
openshift-dr-systemproject and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl. - Click Create instance for DRPolicy and click YAML view.
Copy and save the following YAML to filename
drpolicy.yamlafter replacing <cluster1> and <cluster2> with the correct names of your managed clusters in ACM. Replace <string_value_1> and <string_value_2> with any values as long as they are unique (for example: east and west). TheschedulingIntervalshould be one of the values configured in the MirrorPeer earlier (for example: 5m).apiVersion: ramendr.openshift.io/v1alpha1 kind: DRPolicy metadata: name: odr-policy-5m spec: drClusterSet: - name: <cluster1> region: <string_value_1> s3ProfileName: s3profile-<cluster1>-ocs-storagecluster - name: <cluster2> region: <string_value_2> s3ProfileName: s3profile-<cluster2>-ocs-storagecluster schedulingInterval: 5mNoteThere is no need to specify a namespace to create this resource because DRPolicy is a cluster-scoped resource.
-
Copy the contents of your unique
drpolicy.yamlfile into the YAML view. You must completely replace the original content. Click Create on the YAML view screen.
ImportantThe DRPolicy
schedulingIntervalmust match one of the values configured in MirroPeer resource (e.g. 5m). To use one of the otherschedulingIntervalsfor volume replication configured in the MirrorPeer requires creating additional DRPolicy resources with the new values (i.e., 15m). Make sure to change the DRPolicynameto be unique and useful in identifying the replication interval (e.g. odr-policy-15m).Verify that the DRPolicy is created successfully by running the command on the Hub cluster for each DRPolicy resource created. This example is for
odr-policy-5m:$ oc get drpolicy odr-policy-5m -n openshift-dr-system -o jsonpath='{.status.conditions[].reason}{"\n"}'Example output:
Succeeded
Chapter 9. Enabling automatic install of OpenShift DR cluster operator
Once the DRPolicy is created successfully, the OpenShift DR Cluster operator can be installed on the Primary managed cluster and Secondary managed cluster in the openshift-dr-system namespace.
Procedure
Edit the ConfigMag
ramen-hub-operator-configon the Hub cluster to adddeploymentAutomationEnabled=trueas follows:$ oc edit configmap ramen-hub-operator-config -n openshift-dr-system
apiVersion: v1 data: ramen_manager_config.yaml: | apiVersion: ramendr.openshift.io/v1alpha1 drClusterOperator: deploymentAutomationEnabled: true ## <-- Add to enable installation of ODR Cluster operator on managed clusters catalogSourceName: redhat-operators catalogSourceNamespaceName: openshift-marketplace channelName: stable-4.10 clusterServiceVersionName: odr-cluster-operator.v4.10.0 namespaceName: openshift-dr-system packageName: odr-cluster-operator [...]Verify that the installation was successful in the Primary managed cluster and the Secondary managed cluster do the following command:
$ oc get csv,pod -n openshift-dr-system
Example output:
NAME DISPLAY VERSION REPLACES PHASE clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.10.0 Openshift DR Cluster Operator 4.10.0 Succeeded NAME READY STATUS RESTARTS AGE pod/ramen-dr-cluster-operator-5564f9d669-f6lbc 2/2 Running 0 5m32s
You can also go to OperatorHub on each of the managed clusters and verify if the
OpenShift DR Cluster Operatoris installed.
Chapter 10. Enabling automatic transfer of s3Secrets to managed clusters
Follow this procedure to enable auto transfer of s3Secrets to the required OpenShift DR cluster components. It updates the OpenShift DR cluster namespace with the s3Secrets that are required to access the s3Profiles in the OpenShift DR config map.
Procedure
Edit the ConfigMag
ramen-hub-operator-configon the Hub cluster to adds3SecretDistributionEnabled=trueas follows:$ oc edit configmap ramen-hub-operator-config -n openshift-dr-system
apiVersion: v1 data: ramen_manager_config.yaml: | apiVersion: ramendr.openshift.io/v1alpha1 drClusterOperator: deploymentAutomationEnabled: true s3SecretDistributionEnabled: true ## <-- Add to enable automatic transfer of s3secrets catalogSourceName: redhat-operators catalogSourceNamespaceName: openshift-marketplace channelName: stable-4.10 clusterServiceVersionName: odr-cluster-operator.v4.10.0 namespaceName: openshift-dr-system packageName: odr-cluster-operator [...]Verify that transfer of secrets was successful by running this command in both managed clusters.
$ oc get secrets -n openshift-dr-system | grep Opaque
Example output:
8b3fb9ed90f66808d988c7edfa76eba35647092 Opaque 2 11m af5f82f21f8f77faf3de2553e223b535002e480 Opaque 2 11m
Chapter 11. Creating a sample application
In order to test failover from the Primary managed cluster to the Secondary managed cluster and back again we need a simple application. Use the sample application called busybox as an example.
Procedure
Create a namespace or project on the Hub cluster for a
busyboxsample application.$ oc new-project busybox-sample
NoteA different project name other than
busybox-samplecan be used if desired. Make sure when deploying the sample application via the Advanced Cluster Manager console to use the same project name as what is created in this step.Create DRPlacementControl resource
DRPlacementControl is an API available after the OpenShift DR Hub Operator is installed on the Hub cluster. It is broadly an Advanced Cluster Manager PlacementRule reconciler that orchestrates placement decisions based on data availability across clusters that are part of a DRPolicy.
-
On the Hub cluster, navigate to Installed Operators in the
busybox-sampleproject and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl. -
Create an instance for DRPlacementControl and then go to the YAML view. Make sure the
busybox-sampleproject is selected. Save the following YAML to filename
busybox-drpc.yamlafter replacing <cluster1> with the correct name of your managed cluster in Advanced Cluster Manager. ModifydrPolicyRefname for the DRPolicy that has the desired replication interval.apiVersion: ramendr.openshift.io/v1alpha1 kind: DRPlacementControl metadata: labels: app: busybox-sample name: busybox-drpc spec: drPolicyRef: name: odr-policy-5m ## <-- Modify to specify desired DRPolicy and RPO placementRef: kind: PlacementRule name: busybox-placement preferredCluster: <cluster1> pvcSelector: matchLabels: appname: busybox-
Copy the contents of your unique
busybox-drpc.yamlfile into the YAML view (completely replacing original content). Click Create on the YAML view screen.
You can also create this resource using the following CLI command:
$ oc create -f busybox-drpc.yaml -n busybox-sample
Example output:
drplacementcontrol.ramendr.openshift.io/busybox-drpc created
ImportantThis resource must be created in the
busybox-samplenamespace (or whatever namespace you created earlier).
-
On the Hub cluster, navigate to Installed Operators in the
Create Placement Rule resource that defines the target clusters where resource templates can be deployed. Use placement rules to facilitate the multicluster deployment of your applications.
Copy and save the following YAML to filename
busybox-placementrule.yaml.apiVersion: apps.open-cluster-management.io/v1 kind: PlacementRule metadata: labels: app: busybox-sample name: busybox-placement spec: clusterConditions: - status: "True" type: ManagedClusterConditionAvailable clusterReplicas: 1 schedulerName: ramenCreate the Placement Rule resource for the
busybox-sampleapplication.$ oc create -f busybox-placementrule.yaml -n busybox-sample
Example output:
placementrule.apps.open-cluster-management.io/busybox-placement created
ImportantThis resource must be created in the
busybox-samplenamespace (or whatever namespace you created earlier).
Create sample application using RHACM console
Log in to the RHACM console using your OpenShift credentials if not already logged in.
$ oc get route multicloud-console -n open-cluster-management -o jsonpath --template="https://{.spec.host}/multicloud/applications{'\n'}"Example Output:
https://multicloud-console.apps.perf3.example.com/multicloud/applications
- Navigate to Applications and click Create application.
- Select type as Subscription.
-
Enter your application Name (for example,
busybox) and Namespace (for example,busybox-sample). -
In Repository location for resources section, select Repository type
Git. Enter the Git repository URL for the sample application, the github Branch and Path where the resources
busyboxPod and PVC will be created.Use the sample application repository as
https://github.com/RamenDR/ocm-ramen-sampleswhere the Branch ismainand Path isbusybox-odr.ImportantMake sure that the new StorageClass
ocs-storagecluster-ceph-rbdmirroris created as detailed in section Create Mirroring StorageClass resource before proceeding.Verify that it is created using the following command:
oc get storageclass | grep rbdmirror | awk '{print $1}'Example output:
ocs-storagecluster-ceph-rbdmirror
- Scroll down the form to the section Select clusters to deploy to and click Select an existing placement configuration.
-
Select an Existing Placement Rule (for example,
busybox-placement) from the drop-down list. Click Save.
On the follow-on screen scroll to the bottom. You should see that there are all Green checkmarks on the application topology.
NoteTo get more information, click on any of the topology elements and a window will appear on the right of the topology view.
Verify the sample application deployment and replication.
Now that the
busyboxapplication has been deployed to your preferred Cluster (specified in the DRPlacementControl) the deployment can be validated.Login to your managed cluster where
busyboxwas deployed by RHACM.$ oc get pods,pvc -n busybox-sample
Example output:
NAME READY STATUS RESTARTS AGE pod/busybox 1/1 Running 0 6m NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/busybox-pvc Bound pvc-a56c138a-a1a9-4465-927f-af02afbbff37 1Gi RWO ocs-storagecluster-ceph-rbd 6m
Verify that the replication resources are also created for the
busyboxPVC.$ oc get volumereplication,volumereplicationgroup -n busybox-sample
Example output:
NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE volumereplication.replication.storage.openshift.io/busybox-pvc 6m odf-rbd-volumereplicationclass busybox-pvc primary Primary NAME AGE volumereplicationgroup.ramendr.openshift.io/busybox-drpc 6m
Verify that the
busyboxvolume has been replicated to the alternate cluster by running the following command on both the Primary managed cluster and the Secondary managed cluster.$ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'Example output:
{"daemon_health":"OK","health":"OK","image_health":"OK","states":{"replaying":2}}NoteBoth managed clusters should have the exact same output with a new status of
"states":{"replaying":2}`.
11.1. Deleting sample application
You can delete the sample application busybox using the RHACM console.
The instructions to delete the sample application should not be executed until the failover and failback (relocate) testing is completed and the application is ready to be removed from RHACM and the managed clusters.
Procedure
- On the RHACM console, navigate to Applications.
-
Search for the sample application to be deleted (for example,
busybox). - Click the Action Menu (⋮) next to the application you want to delete.
Click Delete application.
When Delete application is selected a new screen will appear asking if the application related resources should also be deleted.
- Select Remove application related resources checkbox to delete the Subscription and PlacementRule.
- Click Delete. This will delete the busybox application on the Primary managed cluster (or whatever cluster the application was running on).
In addition to the resources deleted using the RHACM console, the
DRPlacementControlmust also be deleted immediately after deleting thebusyboxapplication.-
Login to the OpenShift Web console for the Hub cluster and navigate to Installed Operators for the project
busybox-sample. - Click OpenShift DR Hub Operator and then click DRPlacementControl tab.
-
Click the Action Menu (⋮) next to the
busyboxapplication DRPlacementControl that you want to delete. - Click Delete DRPlacementControl.
- Click Delete.
-
Login to the OpenShift Web console for the Hub cluster and navigate to Installed Operators for the project
This process can be used to delete any application with a DRPlacementControl resource. The DRPlacementControl resource can also be deleted in the application namespace using CLI.
Chapter 12. Application failover between managed clusters
This section provides instructions on how to failover the busybox sample application. The failover method for Regional-DR is application based. Each application that is to be protected in this manner must have a corresponding DRPlacementControl resource and a PlacementRule resource created in the application namespace as shown in the Create Sample Application for DR testing section.
Procedure
- On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
- Click DRPlacementControl tab.
-
Click DRPC
busybox-drpcand then the YAML view. Add the
actionandfailoverClusterdetails as shown in below screenshot. ThefailoverClustershould be the ACM cluster name for the Secondary managed cluster.DRPlacementControl add action Failover
- Click Save.
Verify that the application
busyboxis now running in the Secondary managed cluster, the failover clusterocp4perf2specified in the YAML file.$ oc get pods,pvc -n busybox-sample
Example output:
NAME READY STATUS RESTARTS AGE pod/busybox 1/1 Running 0 35s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/busybox-pvc Bound pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb 5Gi RWO ocs-storagecluster-ceph-rbd 35s
Verify that
busyboxis no longer running on the Primary managed cluster.$ oc get pods,pvc -n busybox-sample
Example output:
No resources found in busybox-sample namespace.
Be aware of known Regional-DR issues as documented in Known Issues section of Release Notes.
Chapter 13. Relocating an application between managed clusters
A relocation operation is very similar to failover. Relocate is application based and uses the DRPlacementControl to trigger the relocation. The main difference for relocation is that a resync is issued to make sure any new application data saved on the Secondary managed cluster is immediately, not waiting for the mirroring schedule interval, replicated to the Primary managed cluster.
Procedure
- On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
- Click DRPlacementControl tab.
-
Click DRPC
busybox-drpcand then the YAML view. Modify action to
RelocateDRPlacementControl modify action to Relocate
- Click Save.
Verify if the application
busyboxis now running in the Primary managed cluster.The failback is to the preferredClusterocp4perf1as specified in the YAML file, which is where the application was running before the failover operation.$ oc get pods,pvc -n busybox-sample
Example output:
NAME READY STATUS RESTARTS AGE pod/busybox 1/1 Running 0 60s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/busybox-pvc Bound pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb 5Gi RWO ocs-storagecluster-ceph-rbd 61s
Verify if
busyboxis running in the Secondary managed cluster. The busybox application should no longer be running on this managed cluster.$ oc get pods,pvc -n busybox-sample
Example output:
No resources found in busybox-sample namespace.
Be aware of known Regional-DR issues as documented in Known Issues section of Release Notes.