Procedure

1. Check the current election strategy being used by the monitors with the ceph mon dump command. By default in a ceph cluster, the connectivity is set to classic.

   ceph mon dump | grep election_strategy

   Example output:

   dumped monmap epoch 9
   election_strategy: 1

2. Change the monitor election to connectivity.

   ceph mon set election_strategy connectivity

3. Run the previous ceph mon dump command again to verify the election_strategy value.

   $ ceph mon dump | grep election_strategy

   Example output:

   dumped monmap epoch 10
   election_strategy: 3

   To know more about the different election strategies, see Configuring monitor election strategy.

4. Set the location for all our Ceph monitors:

   ceph mon set_location ceph1 datacenter=DC1
   ceph mon set_location ceph2 datacenter=DC1
   ceph mon set_location ceph4 datacenter=DC2
   ceph mon set_location ceph5 datacenter=DC2
   ceph mon set_location ceph7 datacenter=DC3

5. Verify that each monitor has its appropriate location.

   $ ceph mon dump

   Example output:

   epoch 17
   fsid dd77f050-9afe-11ec-a56c-029f8148ea14
   last_changed 2022-03-04T07:17:26.913330+0000
   created 2022-03-03T14:33:22.957190+0000
   min_mon_release 16 (pacific)
   election_strategy: 3
   0: [v2:10.0.143.78:3300/0,v1:10.0.143.78:6789/0] mon.ceph1; crush_location {datacenter=DC1}
   1: [v2:10.0.155.185:3300/0,v1:10.0.155.185:6789/0] mon.ceph4; crush_location {datacenter=DC2}
   2: [v2:10.0.139.88:3300/0,v1:10.0.139.88:6789/0] mon.ceph5; crush_location {datacenter=DC2}
   3: [v2:10.0.150.221:3300/0,v1:10.0.150.221:6789/0] mon.ceph7; crush_location {datacenter=DC3}
   4: [v2:10.0.155.35:3300/0,v1:10.0.155.35:6789/0] mon.ceph2; crush_location {datacenter=DC1}

6. Create a CRUSH rule that makes use of this OSD crush topology by installing the ceph-base RPM package in order to use the crushtool command:

   $ dnf -y install ceph-base

   To know more about CRUSH ruleset, see Ceph CRUSH ruleset.

7. Get the compiled CRUSH map from the cluster:

   $ ceph osd getcrushmap > /etc/ceph/crushmap.bin

8. Decompile the CRUSH map and convert it to a text file in order to be able to edit it:

   $ crushtool -d /etc/ceph/crushmap.bin -o /etc/ceph/crushmap.txt

9. Add the following rule to the CRUSH map by editing the text file /etc/ceph/crushmap.txt at the end of the file.

   $ vim /etc/ceph/crushmap.txt

   rule stretch_rule {
           id 1
           type replicated
           min_size 1
           max_size 10
           step take DC1
           step chooseleaf firstn 2 type host
           step emit
           step take DC2
           step chooseleaf firstn 2 type host
           step emit
   }
   
   # end crush map

   Note

   The rule id has to be unique. In the example, we only have one more crush rule with id 0 hence we are using id 1. If your deployment has more rules created, then use the next free id.

   The CRUSH rule declared contains the following information:

   • Rule name:

     • Description: A unique whole name for identifying the rule.
     • Value: stretch_rule

   • id:

     • Description: A unique whole number for identifying the rule.
     • Value: 1

   • type:

     • Description: Describes a rule for either a storage drive replicated or erasure-coded.
     • Value: replicated

   • min_size:

     • Description: If a pool makes fewer replicas than this number, CRUSH will not select this rule.
     • Value: 1

   • max_size:

     • Description: If a pool makes more replicas than this number, CRUSH will not select this rule.
     • Value: 10

   • step take DC1

     • Description: Takes a bucket name (DC1), and begins iterating down the tree.

   • step chooseleaf firstn 2 type host

     • Description: Selects the number of buckets of the given type, in this case is two different hosts located in DC1.

   • step emit

     • Description: Outputs the current value and empties the stack. Typically used at the end of a rule, but may also be used to pick from different trees in the same rule.

   • step take DC2

     • Description: Takes a bucket name (DC2), and begins iterating down the tree.

   • step chooseleaf firstn 2 type host

     • Description: Selects the number of buckets of the given type, in this case, is two different hosts located in DC2.

   • step emit

     • Description: Outputs the current value and empties the stack. Typically used at the end of a rule, but may also be used to pick from different trees in the same rule.

10. Compile the new CRUSH map from the file /etc/ceph/crushmap.txt and convert it to a binary file called /etc/ceph/crushmap2.bin:

    $ crushtool -c /etc/ceph/crushmap.txt -o /etc/ceph/crushmap2.bin

11. Inject the new crushmap we created back into the cluster:

    $ ceph osd setcrushmap -i /etc/ceph/crushmap2.bin

    Example output:

    17

    Note

    The number 17 is a counter and it will increase (18,19, and so on) depending on the changes you make to the crush map.

12. Verify that the stretched rule created is now available for use.

    ceph osd crush rule ls

    Example output:

    replicated_rule
    stretch_rule

13. Enable the stretch cluster mode.

    $ ceph mon enable_stretch_mode ceph7 stretch_rule datacenter

    In this example, ceph7 is the arbiter node, stretch_rule is the crush rule we created in the previous step and datacenter is the dividing bucket.

14. Verify all our pools are using the stretch_rule CRUSH rule we have created in our Ceph cluster:

    $ for pool in $(rados lspools);do echo -n "Pool: ${pool}; ";ceph osd pool get ${pool} crush_rule;done

    Example output:

    Pool: device_health_metrics; crush_rule: stretch_rule
    Pool: cephfs.cephfs.meta; crush_rule: stretch_rule
    Pool: cephfs.cephfs.data; crush_rule: stretch_rule
    Pool: .rgw.root; crush_rule: stretch_rule
    Pool: default.rgw.log; crush_rule: stretch_rule
    Pool: default.rgw.control; crush_rule: stretch_rule
    Pool: default.rgw.meta; crush_rule: stretch_rule
    Pool: rbdpool; crush_rule: stretch_rule

    This indicates that a working Red Hat Ceph Storage stretched cluster with arbiter mode is now available.

1. Install the latest OpenShift Data Foundation on each of the managed clusters.

2. After installing the operator, create StorageSystem using the option Connect with external storage platform.

   For detailed instructions, refer to Deploying OpenShift Data foundation in external mode.

3. Validate the successful deployment of OpenShift Data foundation:

   1. on each managed cluster with the following command:

      $ oc get storagecluster -n openshift-storage ocs-external-storagecluster -o jsonpath='{.status.phase}{"\n"}'

   2. For the Multicloud Gateway (MCG):

      $ oc get noobaa -n openshift-storage noobaa -o jsonpath='{.status.phase}{"\n"}'

   If the status result is Ready for both queries on the Primary managed cluster and the Secondary managed cluster, then continue with the next step.

Procedure

1. On the Hub cluster, navigate to OperatorHub and use the search filter for OpenShift DR Hub Operator.
2. Follow the screen instructions to Install the operator into the project openshift-dr-system.

3. Verify that the operator Pod is in Running state using the following command:

   $ oc get pods -n openshift-dr-system

   Example output:

   NAME                                 	READY   STATUS    RESTARTS   AGE
   ramen-hub-operator-898c5989b-96k65   	2/2     Running   0          4m14s

Procedure

1. On the Hub cluster, navigate to Installed Operators in the openshift-dr-system project and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.
2. Click Create instance for DRPolicy and click YAML view.

3. Save the following YAML to filename drpolicy.yaml after replacing <cluster1> and <cluster2> with the correct names of your managed clusters in RHACM. Replace <string_value> with any value (i.e. metro).

   apiVersion: ramendr.openshift.io/v1alpha1
   kind: DRPolicy
   metadata:
     name: odr-policy
   spec:
     drClusterSet:
     - name: <cluster1>
       region: <string_value>
       s3ProfileName: s3-primary
       clusterFence: Unfenced
     - name: <cluster2>
       region: <string_value>
       s3ProfileName: s3-secondary
       clusterFence: Unfenced

   Note

   There is no need to specify a namespace to create this resource because DRPolicy is a cluster-scoped resource.

4. Copy the contents of your unique drpolicy.yaml file into the YAML view. You must completely replace the original content.
5. Click Create on the YAML view screen.

6. To validate that the DRPolicy is created successfully and that the MCG object buckets can be accessed using the Secrets created earlier, run this command on the Hub cluster:

   $ oc get drpolicy odr-policy -n openshift-dr-system -o jsonpath='{.status.conditions[].reason}{"\n"}'

   Example output:

   Succeeded

Procedure

1. Edit the ConfigMag ramen-hub-operator-config on the Hub cluster and modify the value of deploymentAutomationEnabled=false to true as follows:

   $ oc edit configmap ramen-hub-operator-config -n openshift-dr-system

   apiVersion: v1
   data:
     ramen_manager_config.yaml: |
     [...]
       drClusterOperator:
         deploymentAutomationEnabled: true  ## <-- Change value to "true" if it is set to "false"
         channelName: stable-4.10
         packageName: odr-cluster-operator
         namespaceName: openshift-dr-system
         catalogSourceName: redhat-operators
         catalogSourceNamespaceName: openshift-marketplace
         clusterServiceVersionName: odr-cluster-operator.v4.10.0
   [...]

2. Verify that the installation was successful in the Primary managed cluster and the Secondary managed cluster do the following command:

   $ oc get csv,pod -n openshift-dr-system

   Example output:

   NAME                                                                      DISPLAY                         VERSION   REPLACES   PHASE
   clusterserviceversion.operators.coreos.com/odr-cluster-operator.v4.10.0   Openshift DR Cluster Operator   4.10.0               Succeeded
   
   NAME                                             READY   STATUS    RESTARTS   AGE
   pod/ramen-dr-cluster-operator-5564f9d669-f6lbc   2/2     Running   0          5m32s

   You can also go to OperatorHub on each of the managed clusters and verify if the OpenShift DR Cluster Operator is installed.

Procedure

1. Edit the ConfigMag ramen-hub-operator-config on the Hub cluster to add s3SecretDistributionEnabled=true as follows:

   $ oc edit configmap ramen-hub-operator-config -n openshift-dr-system

   apiVersion: v1
   data:
     ramen_manager_config.yaml: |
       apiVersion: ramendr.openshift.io/v1alpha1
       drClusterOperator:
         deploymentAutomationEnabled: true
         s3SecretDistributionEnabled: true  ## <-- Add to enable automatic transfer of s3secrets
         catalogSourceName: redhat-operators
         catalogSourceNamespaceName: openshift-marketplace
         channelName: stable-4.10
         clusterServiceVersionName: odr-cluster-operator.v4.10.0
         namespaceName: openshift-dr-system
         packageName: odr-cluster-operator
   [...]

2. Verify that transfer of secrets was successful by running this command in both managed clusters.

   $ oc get secrets -n openshift-dr-system | grep Opaque

   Example output:

   8b3fb9ed90f66808d988c7edfa76eba35647092   Opaque        2      11m
   af5f82f21f8f77faf3de2553e223b535002e480   Opaque        2      11m

Procedure

1. Create a namespace or project on the Hub cluster for a busybox sample application.

   $ oc new-project busybox-sample

   Note

   A different project name other than busybox-sample can be used if desired. Make sure when deploying the sample application via the Advanced Cluster Manager console to use the same project name as what is created in this step.

2. Create DRPlacementControl resource

   DRPlacementControl is an API available after the OpenShift DR Hub Operator is installed on the Hub cluster. It is broadly an Advanced Cluster Manager PlacementRule reconciler that orchestrates placement decisions based on data availability across clusters that are part of a DRPolicy.

   1. On the Hub cluster, navigate to Installed Operators in the busybox-sample project and click on OpenShift DR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.
   2. Create an instance for DRPlacementControl and then go to the YAML view. Make sure the busybox-sample project is selected.

   3. Copy and save the following YAML to filename busybox-drpc.yaml after replacing <cluster1> with the correct name of your managed cluster in Advanced Cluster Manager.

      apiVersion: ramendr.openshift.io/v1alpha1
      kind: DRPlacementControl
      metadata:
        labels:
          app: busybox-sample
        name: busybox-drpc
      spec:
        drPolicyRef:
          name: odr-policy
        placementRef:
          kind: PlacementRule
          name: busybox-placement
        preferredCluster: <cluster1>
        pvcSelector:
          matchLabels:
            appname: busybox

   4. Copy the contents of your unique busybox-drpc.yaml file into the YAML view (completely replacing original content).

   5. Click Create on the YAML view screen.

      You can also create this resource using the following CLI command:

      $ oc create -f busybox-drpc.yaml -n busybox-sample

      Example output:

      drplacementcontrol.ramendr.openshift.io/busybox-drpc created

      Important

      This resource must be created in the busybox-sample namespace (or whatever namespace you created earlier).

3. Create Placement Rule resource that defines the target clusters where resource templates can be deployed. Use placement rules to facilitate the multicluster deployment of your applications.

   1. Copy and save the following YAML to filename busybox-placementrule.yaml.

      apiVersion: apps.open-cluster-management.io/v1
      kind: PlacementRule
      metadata:
        labels:
          app: busybox-sample
        name: busybox-placement
      spec:
        clusterConditions:
        - status: "True"
          type: ManagedClusterConditionAvailable
        clusterReplicas: 1
        schedulerName: ramen

   2. Create the Placement Rule resource for the busybox-sample application.

      $ oc create -f busybox-placementrule.yaml -n busybox-sample

      Example output:

      placementrule.apps.open-cluster-management.io/busybox-placement created

      Important

      This resource must be created in the busybox-sample namespace (or whatever namespace you created earlier).

4. Create sample application using RHACM console

   1. Log in to the RHACM console using your OpenShift credentials if not already logged in.

      $ oc get route multicloud-console -n open-cluster-management -o jsonpath --template="https://{.spec.host}/multicloud/applications{'\n'}"

      Example Output:

      https://multicloud-console.apps.perf3.example.com/multicloud/applications

   2. Navigate to Applications and click Create application.
   3. Select type as Subscription.
   4. Enter your application Name (for example, busybox) and Namespace (for example, busybox-sample).
   5. In Repository location for resources section, select Repository type Git.

   6. Enter the Git repository URL for the sample application, the github Branch and Path where the resources busybox Pod and PVC will be created.

      Use the sample application repository as https://github.com/RamenDR/ocm-ramen-samples where the Branch is main and Path is busybox-odr-metro.

   7. Scroll down the form to the section Select clusters to deploy to and click Select an existing placement configuration.
   8. Select an Existing Placement Rule (for example, busybox-placement) from the drop-down list.

   9. Click Save.

      On the follow-on screen scroll to the bottom. You should see that there are all Green checkmarks on the application topology.

      Note

      To get more information, click on any of the topology elements and a window will appear on the right of the topology view.

5. Validating the sample application deployment and replication.

   Now that the busybox application has been deployed to your preferred Cluster (specified in the DRPlacementControl) the deployment can be validated.

   1. Login to your managed cluster where busybox was deployed by RHACM.

      $ oc get pods,pvc -n busybox-sample

      Example output:

      NAME          READY   STATUS    RESTARTS   AGE
      pod/busybox   1/1     Running   0          6m
      
      NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
      persistentvolumeclaim/busybox-pvc   Bound    pvc-a56c138a-a1a9-4465-927f-af02afbbff37   1Gi        RWO            ocs-storagecluster-ceph-rbd   6m

   2. Verify that the replication resources are also created for the busybox PVC.

      $ oc get volumereplicationgroup -n busybox-sample

      Example output:

      NAME                                                       AGE
      volumereplicationgroup.ramendr.openshift.io/busybox-drpc   6m

Procedure

1. Create NetworkFence resource and enable Fencing.

   Specify the list of CIDR blocks or IP addresses on which network fencing operation will be performed. In our case, this will be the EXTERNAL-IP of every OpenShift node in the cluster that needs to be fenced from using the external RHCS cluster.

   1. Execute this command to get the IP addresses for the Primary managed cluster.

      $ oc get nodes -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="ExternalIP")].address}{"\n"}{end}'

      Example output:

      10.70.56.118
      10.70.56.193
      10.70.56.154
      10.70.56.242
      10.70.56.136
      10.70.56.99

      Note

      Collect the current IP addresses of all OpenShift nodes before there is a site outage. Best practice would be to create the NetworkFence YAML file and have it available and up-to-date for a disaster recovery event.

      The IP addresses for all nodes will be added to the NetworkFence example resource as shown below. This example is for six nodes but there could be more nodes in your cluster.

      apiVersion: csiaddons.openshift.io/v1alpha1
      kind: NetworkFence
      metadata:
        name: network-fence-<cluster1>
      spec:
        driver: openshift-storage.rbd.csi.ceph.com
        cidrs:
          -  <IP_Address1>/32
          -  <IP_Address2>/32
          -  <IP_Address3>/32
          -  <IP_Address4>/32
          -  <IP_Address5>/32
          -  <IP_Address6>/32
          [...]
        secret:
          name: rook-csi-rbd-provisioner
          namespace: openshift-storage
        parameters:
          clusterID: openshift-storage

   2. For the YAML file example above, modify the IP addresses and provide the correct <cluster1> to be the cluster name found in RHACM for the Primary managed cluster. Save this to filename network-fence-<cluster1>.yaml.

      Important

      The NetworkFence must be created from the opposite managed cluster where the application is currently running prior to failover. In this case, that is the Secondary managed cluster.

      $ oc create -f network-fence-<cluster1>.yaml

      Example output:

      networkfences.csiaddons.openshift.io/network-fence-ocp4perf1 created

      Important

      After the NetworkFence is created, all communication from applications to the OpenShift Data Foundation storage will fail and some Pods will be in an unhealthy state (For example: CreateContainerError, CrashLoopBackOff) on the cluster that is now fenced.

   3. In the same cluster as where the NetworkFence was created, verify that the status is Succeeded. Modify <cluster1> to be correct.

      export NETWORKFENCE=network-fence-<cluster1>
      oc get networkfences.csiaddons.openshift.io/$NETWORKFENCE -n openshift-dr-system -o jsonpath='{.status.result}{"\n"}'

      Example output:

      Succeeded

2. Modify DRPolicy for the fenced cluster.

   1. Edit the DRPolicy on the Hub cluster and change <cluster1> (for example: ocp4perf1) from Unfenced to ManuallyFenced.

      $ oc edit drpolicy odr-policy

      Example output:

      [...]
      spec:
        drClusterSet:
        - clusterFence: ManuallyFenced  ## <-- Modify from Unfenced to ManuallyFenced
          name: ocp4perf1
          region: metro
          s3ProfileName: s3-primary
        - clusterFence: Unfenced
          name: ocp4perf2
          region: metro
          s3ProfileName: s3-secondary
      [...]

      Example output:

      drpolicy.ramendr.openshift.io/odr-policy edited

   2. Validate the DRPolicy status in the Hub cluster has changed to Fenced for the Primary managed cluster.

      $ oc get drpolicies.ramendr.openshift.io odr-policy -o yaml | grep -A 6 drClusters

      Example output:

        drClusters:
          ocp4perf1:
            status: Fenced
            string: ocp4perf1
          ocp4perf2:
            status: Unfenced
            string: ocp4perf2

3. Modify DRPlacementControl to failover

   1. On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
   2. Click DRPlacementControl tab.
   3. Click DRPC busybox-drpc and then the YAML view.

   4. Add the action and failoverCluster details as shown in below screenshot. The failoverCluster should be the ACM cluster name for the Secondary managed cluster.

      DRPlacementControl add action Failover

      

   5. Click Save.

4. Verify that the application busybox is now running in the Secondary managed cluster, the failover cluster ocp4perf2 specified in the YAML file.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   NAME          READY   STATUS    RESTARTS   AGE
   pod/busybox   1/1     Running   0          35s
   
   NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
   persistentvolumeclaim/busybox-pvc   Bound    pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb   5Gi        RWO            ocs-storagecluster-ceph-rbd   35s

5. Verify that busybox is no longer running on the Primary managed cluster.

   $ oc get pods,pvc -n busybox-sample

   Example output:

   No resources found in busybox-sample namespace.

Procedure

1. Remove NetworkFence resource and disable Fencing.

   Before a failback or relocate action can be successful the NetworkFence for the Primary managed cluster must be deleted.

   1. Execute this command in the Secondary managed cluster and modify <cluster1> to be correct for the NetworkFence YAML filename created in the prior section.

      $ oc delete -f network-fence-<cluster1>.yaml

      Example output:

      networkfence.csiaddons.openshift.io "network-fence-ocp4perf1" deleted

   2. Reboot OpenShift Container Platform nodes that were Fenced.

      This step is required because some application Pods on the prior fenced cluster, in this case the Primary managed cluster, are in an unhealthy state (For example: CreateContainerError, CrashLoopBackOff). This can be most easily fixed by rebooting all worker OpenShift nodes one at a time.

      Note

      The OpenShift Web Console dashboards and Overview page can also be used to assess the health of applications and the external storage. The detailed OpenShift Data Foundation dashboard is found by navigating to Storage → Data Foundation.

   3. Verify all Pods are in a healthy state by running this command on the Primary managed cluster after all OpenShift nodes have rebooted and are in a Ready status. The output for this query should be zero Pods.

      $ oc get pods -A | egrep -v 'Running|Completed'

      Example output:

      NAMESPACE                                          NAME                                                              READY   STATUS      RESTARTS       AGE

      Important

      If there are Pods still in an unhealthy status because of severed storage communication, troubleshoot and resolve before continuing. Because the storage cluster is external to OpenShift, it also has to be properly recovered after a site outage for OpenShift applications to be healthy.

2. Modify DRPolicy to Unfenced status.

   In order for the ODR HUB operator to know the NetworkFence has been removed for the Primary managed cluster the DRPolicy must be modified for the newly Unfenced cluster.

   1. Edit the DRPolicy on the Hub cluster and change <cluster1> (example ocp4perf1) from ManuallyFenced to Unfenced.

      $ oc edit drpolicy odr-policy

      Example output:

      [...]
      spec:
        drClusterSet:
        - clusterFence: Unfenced  ## <-- Modify from ManuallyFenced to Unfenced
          name: ocp4perf1
          region: metro
          s3ProfileName: s3-primary
        - clusterFence: Unfenced
          name: ocp4perf2
          region: metro
          s3ProfileName: s3-secondary
      [...]

      Example output:

      drpolicy.ramendr.openshift.io/odr-policy edited

   2. Verify that the status of DRPolicy in the Hub cluster has changed to Unfenced for the Primary managed cluster.

      $ oc get drpolicies.ramendr.openshift.io odr-policy -o yaml | grep -A 6 drClusters

      Example output:

        drClusters:
          ocp4perf1:
            status: Unfenced
            string: ocp4perf1
          ocp4perf2:
            status: Unfenced
            string: ocp4perf2

3. Modify DRPlacementControl to failback

   1. On the Hub cluster navigate to Installed Operators and then click Openshift DR Hub Operator.
   2. Click DRPlacementControl tab.
   3. Click DRPC busybox-drpc and then the YAML view.

   4. Modify action to Relocate.

      DRPlacementControl modify action to Relocate

      

   5. Click Save.

   6. Verify if the application busybox is now running in the Primary managed cluster.The failback is to the preferredCluster ocp4perf1 as specified in the YAML file, which is where the application was running before the failover operation.

      $ oc get pods,pvc -n busybox-sample

      Example output:

      NAME          READY   STATUS    RESTARTS   AGE
      pod/busybox   1/1     Running   0          60s
      
      NAME                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
      persistentvolumeclaim/busybox-pvc   Bound    pvc-79f2a74d-6e2c-48fb-9ed9-666b74cfa1bb   5Gi        RWO            ocs-storagecluster-ceph-rbd   61s

   7. Verify if busybox is running in the Secondary managed cluster. The busybox application should no longer be running on this managed cluster.

      $ oc get pods,pvc -n busybox-sample

      Example output:

      No resources found in busybox-sample namespace.