Procedure

1. Save the following YAML to filename rbd-volumereplicationclass.yaml.

   apiVersion: replication.storage.openshift.io/v1alpha1
   kind: VolumeReplicationClass
   metadata:
     name: odf-rbd-volumereplicationclass
   spec:
     provisioner: openshift-storage.rbd.csi.ceph.com
     parameters:
       mirroringMode: snapshot
       schedulingInterval: "5m"  # <-- Must be the same as scheduling interval in the DRPolicy
       replication.storage.openshift.io/replication-secret-name: rook-csi-rbd-provisioner
       replication.storage.openshift.io/replication-secret-namespace: openshift-storage

2. Create the file on both the managed clusters.

   $ oc create -f rbd-volumereplicationclass.yaml

   Example output:

   volumereplicationclass.replication.storage.openshift.io/odf-rbd-volumereplicationclass created

Procedure

1. Save the following YAML to filename ocs-storagecluster-ceph-rbdmirror.yaml.

   allowVolumeExpansion: true
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: ocs-storagecluster-ceph-rbdmirror
   parameters:
     clusterID: openshift-storage
     csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
     csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
     csi.storage.k8s.io/fstype: ext4
     csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
     csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
     csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
     csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
     imageFeatures: layering,exclusive-lock,object-map,fast-diff
     imageFormat: "2"
     pool: ocs-storagecluster-cephblockpool
   provisioner: openshift-storage.rbd.csi.ceph.com
   reclaimPolicy: Delete
   volumeBindingMode: Immediate

2. Create the file on both the managed clusters.

   $ oc create -f ocs-storagecluster-ceph-rbdmirror.yaml

   Example output:

   storageclass.storage.k8s.io/ocs-storagecluster-ceph-rbdmirror created

Procedure

1. On each managed cluster, navigate to OperatorHub and filter for OpenShift DR Cluster Operator.

2. Follow the screen instructions to install the operator into the project openshift-dr-system.

   Note

   The OpenShift DR Cluster Operator must be installed on both the Primary managed cluster and Secondary managed cluster.

3. Configure SSL access between the s3 endpoints so that metadata can be stored on the alternate cluster in a MCG object bucket using a secure transport protocol and in the Hub cluster for verifying access to the object buckets.

   Note

   If all of your OpenShift clusters are deployed using a signed and trusted set of certificates for your environment then this section can be skipped.

   1. Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

      $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > primary.crt

   2. Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

      $ oc get cm default-ingress-cert -n openshift-config-managed -o jsonpath="{['data']['ca-bundle\.crt']}" > secondary.crt

   3. Create a new ConfigMap to hold certificate bundle of the remote cluster with file name cm-clusters-crt.yaml on the Primary managed cluster, Secondary managed cluster and the Hub cluster.

      Note

      There could be more or less than three certificates for each cluster as shown in this example file.

      apiVersion: v1
      data:
        ca-bundle.crt: |
          -----BEGIN CERTIFICATE-----
          <copy contents of cert1 from primary.crt here>
          -----END CERTIFICATE-----
      
          -----BEGIN CERTIFICATE-----
          <copy contents of cert2 from primary.crt here>
          -----END CERTIFICATE-----
      
          -----BEGIN CERTIFICATE-----
          <copy contents of cert3 primary.crt here>
          -----END CERTIFICATE----
      
          -----BEGIN CERTIFICATE-----
          <copy contents of cert1 from secondary.crt here>
          -----END CERTIFICATE-----
      
          -----BEGIN CERTIFICATE-----
          <copy contents of cert2 from secondary.crt here>
          -----END CERTIFICATE-----
      
          -----BEGIN CERTIFICATE-----
          <copy contents of cert3 from secondary.crt here>
          -----END CERTIFICATE-----
      kind: ConfigMap
      metadata:
        name: user-ca-bundle
        namespace: openshift-config

   4. Run the following command on the Primary managed cluster, Secondary managed cluster, and the Hub cluster to create the file.

      $ oc create -f cm-clusters-crt.yaml

      Example output:

      configmap/user-ca-bundle created

      Important

      For the Hub cluster to verify access to the object buckets using the DRPolicy resource, the same ConfigMap, cm-clusters-crt.yaml, must be created on the Hub cluster.

   5. Modify default Proxy cluster resource.

      1. Copy and save the following content into the new YAML file proxy-ca.yaml.

         apiVersion: config.openshift.io/v1
         kind: Proxy
         metadata:
           name: cluster
         spec:
           trustedCA:
             name: user-ca-bundle

      2. Apply this new file to the default proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

         $ oc apply -f proxy-ca.yaml

         Example output:

         proxy.config.openshift.io/cluster configured

4. Retrieve Multicloud Object Gateway (MCG) keys and external S3 endpoint.

   1. Check if MCG is installed on the Primary managed cluster and the Secondary managed cluster, and if Phase is Ready.

      $ oc get noobaa -n openshift-storage

      Example output:

      NAME     MGMT-ENDPOINTS                   S3-ENDPOINTS                    IMAGE                                                                                                 PHASE   AGE
      noobaa   ["https://10.70.56.161:30145"]   ["https://10.70.56.84:31721"]   quay.io/rhceph-dev/mcg-core@sha256:c4b8857ee9832e6efc5a8597a08b81730b774b2c12a31a436e0c3fadff48e73d   Ready   27h

   2. Copy the following YAML file to filename odrbucket.yaml.

      apiVersion: objectbucket.io/v1alpha1
      kind: ObjectBucketClaim
      metadata:
        name: odrbucket
        namespace: openshift-dr-system
      spec:
        generateBucketName: "odrbucket"
        storageClassName: openshift-storage.noobaa.io

   3. Create a MCG bucket odrbucket on both the Primary managed cluster and the Secondary managed cluster.

      $ oc create -f odrbucket.yaml

      Example output:

      objectbucketclaim.objectbucket.io/odrbucket created

   4. Extract the odrbucket OBC access key for each managed cluster as their base-64 encoded values by using the following command.

      $ oc get secret odrbucket -n openshift-dr-system -o jsonpath='{.data.AWS_ACCESS_KEY_ID}{"\n"}'

      Example output:

      cFpIYTZWN1NhemJjbEUyWlpwN1E=

   5. Extract the odrbucket OBC secret key for each managed cluster as their base-64 encoded values by using the following command.

      $ oc get secret odrbucket -n openshift-dr-system -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}{"\n"}'

      Example output:

      V1hUSnMzZUoxMHRRTXdGMU9jQXRmUlAyMmd5bGwwYjNvMHprZVhtNw==

5. Create S3 Secrets for Managed clusters.

   Now that the necessary MCG information has been extracted there must be new Secrets created on the Primary managed cluster and the Secondary managed cluster. These new Secrets stores the MCG access and secret keys for both managed clusters.

   Note

   OpenShift DR requires one or more S3 stores to store relevant cluster data of a workload from the managed clusters and to orchestrate a recovery of the workload during failover or relocate actions. These instructions are applicable for creating the necessary object bucket(s) using Multicloud Gateway (MCG). MCG should already be installed as a result of installing OpenShift Data Foundation.

   1. Copy the following S3 secret YAML format for the Primary managed cluster to filename odr-s3secret-primary.yaml.

      apiVersion: v1
      data:
        AWS_ACCESS_KEY_ID: <primary cluster base-64 encoded access key>
        AWS_SECRET_ACCESS_KEY: <primary cluster base-64 encoded secret access key>
      kind: Secret
      metadata:
        name: odr-s3secret-primary
        namespace: openshift-dr-system

      Replace <primary cluster base-64 encoded access key> and <primary cluster base-64 encoded secret access key> with actual values retrieved in an earlier step.

   2. Create this secret on the Primary managed cluster and the Secondary managed cluster.

      $ oc create -f odr-s3secret-primary.yaml

      Example output:

      secret/odr-s3secret-primary created

   3. Copy the following S3 secret YAML format for the Secondary managed cluster to filename odr-s3secret-secondary.yaml.

      apiVersion: v1
      data:
        AWS_ACCESS_KEY_ID: <secondary cluster base-64 encoded access key>
        AWS_SECRET_ACCESS_KEY: <secondary cluster base-64 encoded secret access key>
      kind: Secret
      metadata:
        name: odr-s3secret-secondary
        namespace: openshift-dr-system

      Replace <secondary cluster base-64 encoded access key> and <secondary cluster base-64 encoded secret access key> with actual values retrieved in step 4.

   4. Create this secret on the Primary managed cluster and the Secondary managed cluster.

      $ oc create -f odr-s3secret-secondary.yaml

      Example output:

      secret/odr-s3secret-secondary created

      Important

      The values for the access and secret key must be base-64 encoded. The encoded values for the keys were retrieved in an earlier step.

6. Configure OpenShift DR Cluster Operator ConfigMaps on each of the managed clusters.

   1. Search for the external S3 endpoint s3CompatibleEndpoint or route for MCG on each managed cluster by using the following command.

      $ oc get route s3 -n openshift-storage -o jsonpath --template="https://{.spec.host}{'\n'}"

      Example output:

      https://s3-openshift-storage.apps.perf1.example.com

      Important

      The unique s3CompatibleEndpoint route or s3-openshift-storage.apps.<primary clusterID>.<baseDomain> and s3-openshift-storage.apps.<secondary clusterID>.<baseDomain> must be retrieved for both the Primary managed cluster and Secondary managed cluster respectively.

   2. Search for the odrbucket OBC bucket name.

      $ oc get configmap odrbucket -n openshift-dr-system -o jsonpath='{.data.BUCKET_NAME}{"\n"}'

      Example output:

      odrbucket-2f2d44e4-59cb-4577-b303-7219be809dcd

      Important

      The unique s3Bucket name odrbucket-<your value1> and odrbucket-<your value2> must be retrieved on both the Primary managed cluster and Secondary managed cluster respectively.

   3. Modify the ConfigMap ramen-dr-cluster-operator-config to add the new content.

      $ oc edit configmap ramen-dr-cluster-operator-config -n openshift-dr-system

   4. Add the following new content starting at s3StoreProfiles to the ConfigMap on the Primary managed cluster and the Secondary managed cluster.

      [...]
      data:
        ramen_manager_config.yaml: |
          apiVersion: ramendr.openshift.io/v1alpha1
          kind: RamenConfig
      [...]
          ramenControllerType: "dr-cluster"
          ### Start of new content to be added
          s3StoreProfiles:
          - s3ProfileName: s3-primary
            s3CompatibleEndpoint: https://s3-openshift-storage.apps.<primary clusterID>.<baseDomain>
            s3Region: primary
            s3Bucket: odrbucket-<your value1>
            s3SecretRef:
              name: odr-s3secret-primary
              namespace: openshift-dr-system
          - s3ProfileName: s3-secondary
            s3CompatibleEndpoint: https://s3-openshift-storage.apps.<secondary clusterID>.<baseDomain>
            s3Region: secondary
            s3Bucket: odrbucket-<your value2>
            s3SecretRef:
              name: odr-s3secret-secondary
              namespace: openshift-dr-system
      [...]

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the search filter for OpenShift DR Hub Operator.
2. Follow the screen instructions to Install the operator into the project openshift-dr-system.

3. Create S3 secrets for the Hub cluster using the following S3 secret YAML format for the Primary managed cluster.

   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: <primary cluster base-64 encoded access key>
     AWS_SECRET_ACCESS_KEY: <primary cluster base-64 encoded secret access key>
   kind: Secret
   metadata:
     name: odr-s3secret-primary
     namespace: openshift-dr-system

   Run the following command to create this secret on the Hub cluster.

   $ oc create -f odr-s3secret-primary.yaml

   Example output:

   secret/odr-s3secret-primary created

4. Create S3 secrets using the following S3 secret YAML format for the Secondary managed cluster.

   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: <secondary cluster base-64 encoded access key>
     AWS_SECRET_ACCESS_KEY: <secondary cluster base-64 encoded secret access key>
   kind: Secret
   metadata:
     name: odr-s3secret-secondary
     namespace: openshift-dr-system

   Run the following command to create this secret on the Hub cluster.

   $ oc create -f odr-s3secret-secondary.yaml

   Example output:

   secret/odr-s3secret-secondary created

5. Configure ConfigMap for the OpenShift DR Hub Operator.

   After the operator is successfully created, a new ConfigMap called ramen-hub-operator-config is created.

   1. Run the following command to edit the file.

      $ oc edit configmap ramen-hub-operator-config -n openshift-dr-system

   2. Add the following new content starting at s3StoreProfiles to the ConfigMap on the Hub cluster.

      [...]
      apiVersion: v1
      data:
        ramen_manager_config.yaml: |
          apiVersion: ramendr.openshift.io/v1alpha1
          kind: RamenConfig
      [...]
          ramenControllerType: "dr-hub"
          ### Start of new content to be added
          s3StoreProfiles:
          - s3ProfileName: s3-primary
            s3CompatibleEndpoint: https://s3-openshift-storage.apps.<primary clusterID>.<baseDomain>
            s3Region: primary
            s3Bucket: odrbucket-<your value1>
            s3SecretRef:
              name: odr-s3secret-primary
              namespace: openshift-dr-system
          - s3ProfileName: s3-secondary
            s3CompatibleEndpoint: https://s3-openshift-storage.apps.<secondary clusterID>.<baseDomain>
            s3Region: secondary
            s3Bucket: odrbucket-<your value2>
            s3SecretRef:
              name: odr-s3secret-secondary
              namespace: openshift-dr-system
      [...]

      Note

      Make sure to replace <primary clusterID>, <secondary clusterID>, baseDomain, odrbucket-<your value1>, and odrbucket-<your value2> variables with exact same values as used for the ramen-cluster-operator-config ConfigMap on the managed clusters.

Procedure

1. On the Hub cluster, navigate to Installed Operators in the openshift-dr-system project and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.
2. Click Create instance for DRPolicy and click YAML view.

3. Copy and save the following YAML to filename drpolicy.yaml after replacing <cluster1> and <cluster2> with the correct names of your managed clusters in RHACM.

   apiVersion: ramendr.openshift.io/v1alpha1
   kind: DRPolicy
   metadata:
     name: odr-policy-5m
   spec:
     drClusterSet:
     - name: <cluster1>
       s3ProfileName: s3-primary
     - name: <cluster2>
       s3ProfileName: s3-secondary
     schedulingInterval: 5m

   Note

   There is no need to specify a namespace to create this resource because DRPolicy is a cluster-scoped resource.

4. Copy the contents of your unique drpolicy.yaml file into the YAML view. You must completely replace the original content.

5. Click Create on the YAML view screen.

   Important

   The DRPolicy scheduling interval must match the interval configured in the Creating VolumeReplicationClass resource section.

6. Verify that the DRPolicy is created successfully by running the command:

   $ oc get drpolicy odr-policy-5m -n openshift-dr-system -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

Procedure

1. Create a namespace or project on the Hub cluster for a busybox sample application.

   $ oc new-project busybox-sample

   Note

   A different project name other than busybox-sample can be used if desired. Make sure when deploying the sample application via the Advanced Cluster Manager console to use the same project name as what is created in this step.

2. Create DRPlacementControl resource

   DRPlacementControl is an API available after the OpenShift DR Hub Operator is installed on the Hub cluster. It is broadly an Advanced Cluster Manager PlacementRule reconciler that orchestrates placement decisions based on data availability across clusters that are part of a DRPolicy.

   1. On the Hub cluster, navigate to Installed Operators in the busybox-sample project and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.
   2. Create an instance for DRPlacementControl and then go to the YAML view. Make sure the busybox-sample project is selected.

   3. Copy and save the following YAML to filename busybox-drpc.yaml after replacing <cluster1> with the correct name of your managed cluster in Advanced Cluster Manager.

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRPlacementControl
      metadata:
        labels:
          app: busybox-sample
        name: busybox-drpc
      spec:
        drPolicyRef:
          name: odr-policy-5m
        placementRef:
          kind: PlacementRule
          name: busybox-placement
        preferredCluster: <cluster1>
        pvcSelector:
          matchLabels:
            appname: busybox

   4. Copy the contents of your unique busybox-drpc.yaml file into the YAML view (completely replacing original content).

   5. Click Create on the YAML view screen.

      You can also create this resource using the following CLI command:

      $ oc create -f busybox-drpc.yaml -n busybox-sample

      Example output:

      drplacementcontrol.ramendr.openshift.io/busybox-drpc created

      Important

      This resource must be created in the busybox-sample namespace (or whatever namespace you created earlier).

3. Create Placement Rule resource that defines the target clusters where resource templates can be deployed. Use placement rules to facilitate the multicluster deployment of your applications.

   1. Copy and save the following YAML to filename busybox-placementrule.yaml.

      apiVersion: apps.open-cluster-management.io/v1
      kind: PlacementRule
      metadata:
        labels:
          app: busybox-sample
        name: busybox-placement
      spec:
        clusterConditions:
        - status: "True"
          type: ManagedClusterConditionAvailable
        clusterReplicas: 1
        schedulerName: ramen

   2. Create the Placement Rule resource for the busybox-sample application.

      $ oc create -f busybox-placementrule.yaml -n busybox-sample

      Example output:

      placementrule.apps.open-cluster-management.io/busybox-placement created

      Important

      This resource must be created in the busybox-sample namespace (or whatever namespace you created earlier).

4. Create sample application using RHACM console

   1. Log in to the RHACM console using your OpenShift credentials if not already logged in.

      $ oc get route multicloud-console -n open-cluster-management -o jsonpath --template="https://{.spec.host}/multicloud/applications{'\n'}"

      Example Output:

      https://multicloud-console.apps.perf3.example.com/multicloud/applications

   2. Navigate to Applications and click Create application.
   3. Select type as Subscription.
   4. Enter your application Name (for example, busybox) and Namespace (for example, busybox-sample).
   5. In Repository location for resources section, select Repository type Git.

   6. Enter the Git repository URL for the sample application, the github Branch and Path where the resources busybox Pod and PVC will be created.

      Use the sample application repository as https://github.com/RamenDR/ocm-ramen-samples where the Branch is main and Path is busybox-odr.

      Important

      Make sure that the new StorageClass ocs-storagecluster-ceph-rbdmirror is created as detailed in section [Create Mirroring StorageClass resource] before proceeding.

      Verify that it is created using the following command:

      oc get storageclass | grep rbdmirror | awk '{print $1}'

      Example output:

      ocs-storagecluster-ceph-rbdmirror

   7. Scroll down the form to the section Select clusters to deploy to and click Select an existing placement configuration.
   8. Select an Existing Placement Rule (for example, busybox-placement) from the drop-down list.

   9. Click Save.

      When the application is created, the application details page is displayed. You can scroll down to Resource topology section. There should be Green check marks on the elements and application in the topology.

      Note

      To view more information, click on any of the topology elements and a window will appear in the right of the topology view.

5. Verify the sample application deployment and replication.

   Now that the busybox application has been deployed to your preferred Cluster (specified in the DRPlacementControl) the deployment can be validated.

   1. Logon to your managed cluster where busybox was deployed by RHACM.

      $ oc get pods,pvc -n busybox-sample

      Example output:

      NAME          READY   STATUS    RESTARTS   AGE
      pod/busybox   1/1     Running   0          6m
      
      NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
      persistentvolumeclaim/busybox-pvc   Bound    pvc-a56c138a-a1a9-4465-927f-af02afbbff37   1Gi        RWO            ocs-storagecluster-ceph-rbd   6m

   2. Verify that the replication resources are also created for the busybox PVC.

      $ oc get volumereplication,volumereplicationgroup -n busybox-sample

      Example output:

      NAME                                                             AGE   VOLUMEREPLICATIONCLASS           PVCNAME       DESIREDSTATE   CURRENTSTATE
      volumereplication.replication.storage.openshift.io/busybox-pvc   6m   odf-rbd-volumereplicationclass   busybox-pvc   primary        Primary
      
      NAME                                                       AGE
      volumereplicationgroup.ramendr.openshift.io/busybox-drpc   6m

   3. Verify that the busybox volume has been replicated to the alternate cluster by running the following command on both the Primary managed cluster and the Secondary managed cluster.

      $ oc get cephblockpool ocs-storagecluster-cephblockpool -n openshift-storage -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'

      Example output:

      {"daemon_health":"OK","health":"OK","image_health":"OK","states":{"replaying":2}}

      Note

      Both managed clusters should have the exact same output with a new status of "states":{"replaying":2}`.

Procedure

1. On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
2. Click DRPlacementControl tab.
3. Click DRPC busybox-drpc and then the YAML view.

4. Add the action and failoverCluster details as shown in below screenshot. The failoverCluster should be the ACM cluster name for the Secondary managed cluster.

   DRPlacementControl add action Failover

   

5. Click Save.

6. Verify that the application busybox is now running in the Secondary managed cluster, the failover cluster ocp4perf2 specified in the YAML file.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   NAME          READY   STATUS    RESTARTS   AGE
   pod/busybox   1/1     Running   0          35s
   
   NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/busybox-pvc   Bound    pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb   5Gi        RWO            ocs-storagecluster-ceph-rbd   35s

7. Verify that busybox is no longer running on the Primary managed cluster.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   No resources found in busybox-sample namespace.

Procedure

1. On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
2. Click DRPlacementControl tab.
3. Click DRPC busybox-drpc and then the YAML view.

4. Modify action to Relocate

   DRPlacementControl modify action to Relocate

   

5. Click Save.

6. Verify if the application busybox is now running in the Primary managed cluster, failover cluster ocp4perf2 specified in the YAML file.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   NAME          READY   STATUS    RESTARTS   AGE
   pod/busybox   1/1     Running   0          60s
   
   NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/busybox-pvc   Bound    pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb   5Gi        RWO            ocs-storagecluster-ceph-rbd   61s

7. Verify if busybox is running in the Secondary managed cluster. The busybox application should no longer be running on this managed cluster.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   No resources found in busybox-sample namespace.