Procedure

1. Click Administration → Custom Resource Definitions from the left pane of the OpenShift Web Console.
2. On the Custom Resource Definitions page, click ClusterLogging.
3. On the Custom Resource Definition Overview page, select View Instances from the Actions menu or click the Instances Tab.

4. On the Cluster Logging page, click Create Cluster Logging.

   You might have to refresh the page to load the data.

5. In the YAML, replace the code with the following:

   apiVersion: "logging.openshift.io/v1"
   kind: "ClusterLogging"
   metadata:
     name: "instance"
     namespace: "openshift-logging"
   spec:
     managementState: "Managed"
     logStore:
       type: "elasticsearch"
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: ocs-storagecluster-ceph-rbd
           size: 200G
         redundancyPolicy: "SingleRedundancy"
     visualization:
       type: "kibana"
       kibana:
         replicas: 1
     curation:
       type: "curator"
       curator:
         schedule: "30 3 * * *"
     collection:
       logs:
         type: "fluentd"
         fluentd: {}

6. Click Save.

Verification steps

1. Verify that the Persistent Volume Claims are bound to the elasticsearch pods.

   1. Go to Storage → Persistent Volume Claims.
   2. Set the Project dropdown to openshift-logging.

   3. Verify that Persistent Volume Claims are visible with a state of Bound, attached to elasticsearch-* pods.

      Figure 2.1. Cluster logging created and bound

      

2. Verify that the new cluster logging is being used.

   1. Click Workload → Pods.
   2. Set the Project to openshift-logging.
   3. Verify that the new elasticsearch-* pods appear with a state of Running.
   4. Click the new elasticsearch-* pod to view pod details.
   5. Scroll down to Volumes and verify that the elasticsearch volume has a Type that matches your new Persistent Volume Claim, for example, elasticsearch-elasticsearch-cdm-9r624biv-3.
   6. Click the Persistent Volume Claim name and verify the storage class name in the PersistenVolumeClaim Overview page.

Procedure

1. Add the following lines to your application YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <obc-name>
   spec:
     generateBucketName: <obc-bucket-name>
     storageClassName: noobaa

   These lines are the Object Bucket Claim itself.

   1. Replace <obc-name> with the a unique Object Bucket Claim name.
   2. Replace <obc-bucket-name> with a unique bucket name for your Object Bucket Claim.

2. You can add more lines to the YAML file to automate the use of the Object Bucket Claim. The example below is the mapping between the bucket claim result, which is a configuration map with data and a secret with the credentials. This specific job will claim the Object Bucket from NooBaa, which will create a bucket and an account.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: testjob
   spec:
     template:
       spec:
         restartPolicy: OnFailure
         containers:
           - image: <your application image>
             name: test
             env:
               - name: BUCKET_NAME
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_NAME
               - name: BUCKET_HOST
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_HOST
               - name: BUCKET_PORT
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_PORT
               - name: AWS_ACCESS_KEY_ID
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_ACCESS_KEY_ID
               - name: AWS_SECRET_ACCESS_KEY
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_SECRET_ACCESS_KEY

   1. Replace all instances of <obc-name> with your Object Bucket Claim name.
   2. Replace <your application image> with your application image.

3. Apply the updated YAML file:

   # oc apply -f <yaml.file>

   1. Replace <yaml.file> with the name of your YAML file.

4. To view the new configuration map, run the following:

   # oc get cm <obc-name>

   1. Replace obc-name with the name of your Object Bucket Claim.

      You can expect the following environment variables in the output:

      • BUCKET_HOST - Endpoint to use in the application

      • BUCKET_PORT - The port available for the application

        • The port is related to the BUCKET_HOST. For example, if the BUCKET_HOST is https://my.example.com, and the BUCKET_PORT is 443, the endpoint for the object service would be https://my.example.com:443.
      • BUCKET_NAME - Requested or generated bucket name
      • AWS_ACCESS_KEY_ID - Access key that is part of the credentials
      • AWS_SECRET_ACCESS_KEY - Secret access key that is part of the credentials

Procedure

1. Use the command-line interface to generate the details of a new bucket and credentials. Run the following command:

   # noobaa obc create <obc-name> -n openshift-storage

   Replace <obc-name> with a unique Object Bucket Claim name, for example, myappobc.

   Additionally, you can use the --app-namespace option to specify the namespace where the Object Bucket Claim configuration map and secret will be created, for example, myapp-namespace.

   Example output:

   INFO[0001] ✅ Created: ObjectBucketClaim "test21obc"

   The MCG command-line-interface has created the necessary configuration and has informed OpenShift about the new OBC.

2. Run the following command to view the Object Bucket Claim:

   # oc get obc -n openshift-storage

   Example output:

   NAME        STORAGE-CLASS                 PHASE   AGE
   test21obc   openshift-storage.noobaa.io   Bound   38s

3. Run the following command to view the YAML file for the new Object Bucket Claim:

   # oc get obc test21obc -o yaml -n openshift-storage

   Example output:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     generation: 2
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     resourceVersion: "40756"
     selfLink: /apis/objectbucket.io/v1alpha1/namespaces/openshift-storage/objectbucketclaims/test21obc
     uid: 64f04cba-f662-11e9-bc3c-0295250841af
   spec:
     ObjectBucketName: obc-openshift-storage-test21obc
     bucketName: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     generateBucketName: test21obc
     storageClassName: openshift-storage.noobaa.io
   status:
     phase: Bound

4. Inside of your openshift-storage namespace, you can find the configuration map and the secret to use this Object Bucket Claim. The CM and the secret have the same name as the Object Bucket Claim. To view the secret:

   # oc get -n openshift-storage secret test21obc -o yaml

   Example output:

   Example output:
   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: c0M0R2xVanF3ODR3bHBkVW94cmY=
     AWS_SECRET_ACCESS_KEY: Wi9kcFluSWxHRzlWaFlzNk1hc0xma2JXcjM1MVhqa051SlBleXpmOQ==
   kind: Secret
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40751"
     selfLink: /api/v1/namespaces/openshift-storage/secrets/test21obc
     uid: 65117c1c-f662-11e9-9094-0a5305de57bb
   type: Opaque

   The secret gives you the S3 access credentials.

5. To view the configuration map:

   # oc get -n openshift-storage cm test21obc -o yaml

   Example output:

   apiVersion: v1
   data:
     BUCKET_HOST: 10.0.171.35
     BUCKET_NAME: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     BUCKET_PORT: "31242"
     BUCKET_REGION: ""
     BUCKET_SUBREGION: ""
   kind: ConfigMap
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40752"
     selfLink: /api/v1/namespaces/openshift-storage/configmaps/test21obc
     uid: 651c6501-f662-11e9-9094-0a5305de57bb

   The configuration map contains the S3 endpoint information for your application.

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*

3. Verify the health of OpenShift Container Storage cluster using the persistent storage dashboard:

   1. Click Home → Dashboards from the left pane of the OpenShift Web Console and click the Persistent Storage tab.
   2. In the Health card, verify that the cluster health is displayed as ocs-storagecluster is healthy.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create <backing-store-type> <backingstore_name> --access-key=<AWS ACCESS KEY> --secret-key=<AWS SECRET ACCESS KEY> --target-bucket <bucket-name>

   1. Replace <backing-store-type> with your relevant backing store type: aws-s3, google-cloud-store, azure-blob, or s3-compatible.
   2. Replace <backingstore_name> with the name of the backingstore.
   3. Replace <AWS ACCESS KEY> and <AWS SECRET ACCESS KEY> with an AWS access key ID and secret access key you created for this purpose.

   4. Replace <bucket-name> with an existing AWS bucket name. This argument tells NooBaa which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "aws-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-aws-resource"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     AWS_ACCESS_KEY_ID: <AWS ACCESS KEY ID ENCODED IN BASE64>
     AWS_SECRET_ACCESS_KEY: <AWS SECRET ACCESS KEY ENCODED IN BASE64>

   1. You must supply and encode your own AWS access key ID and secret access key using Base64, and use the results in place of <AWS ACCESS KEY ID ENCODED IN BASE64> and <AWS SECRET ACCESS KEY ENCODED IN BASE64>.
   2. Replace <backingstore-secret-name> with a unique name.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: noobaa
   spec:
     awsS3:
       secret:
         name: <backingstore-secret-name>
         namespace: noobaa
       targetBucket: <bucket-name>
     type: <backing-store-type>

   1. Replace <bucket-name> with an existing AWS bucket name. This argument tells NooBaa which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   2. Replace <backingstore-secret-name> with the name of the secret created in the previous step.
   3. Replace <backing-store-type> with your relevant backing store type: aws-s3, google-cloud-store, azure-blob, or s3-compatible.

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create s3-compatible rgw-resource --access-key=<RGW ACCESS KEY> --secret-key=<RGW SECRET KEY> --target-bucket=<bucket-name> --endpoint=http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:80

   1. To get the <RGW ACCESS KEY> and <RGW SECRET KEY>, run the following command using your RGW user secret name:

      oc get secret <RGW USER SECRET NAME> -o yaml

   2. Decode the access key ID and the access key from Base64 and keep them.
   3. Replace <RGW USER ACCESS KEY> and <RGW USER SECRET ACCESS KEY> with the appropriate, decoded data from the previous step.

   4. Replace <bucket-name> with an existing RGW bucket name. This argument tells NooBaa which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "rgw-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-rgw-resource"

1. Create a CephObjectStore user. This also creates a secret containing the RGW credentials:

   apiVersion: ceph.rook.io/v1
   kind: CephObjectStoreUser
   metadata:
     name: <RGW-Username>
     namespace: openshift-storage
   spec:
     store: ocs-storagecluster-cephobjectstore
     displayName: "<Display-name>"

   1. Replace <RGW-Username> and <Display-name> with a unique username and display name.

2. Apply the following YAML for an S3-Compatible backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <backingstore-name>
     namespace: openshift-storage
   spec:
     s3Compatible:
       endpoint: http://rook-ceph-rgw-ocs-storagecluster-cephobjectstore.openshift-storage.svc.cluster.local:80
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       signatureVersion: v4
       targetBucket: <RGW-bucket-name>
     type: s3-compatible

   1. Replace <backingstore-secret-name> with the name of the secret that was created with CephObjectStore in the previous step.
   2. Replace <bucket-name> with an existing RGW bucket name. This argument tells NooBaa which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

1. From the MCG command-line interface, run the following command to create a bucket class with a mirroring policy:

   $ noobaa bucketclass create mirror-to-aws --backingstores=azure-resource,aws-resource --placement Mirror

2. Set the newly created bucket class to a new bucket claim, generating a new bucket that will be mirrored between two locations:

   $ noobaa obc create  mirrored-bucket --bucketclass=mirror-to-aws

1. Apply the following YAML. This YAML is a hybrid example that mirrors data between local Ceph storage and AWS:

   apiVersion: noobaa.io/v1alpha1
   kind: BucketClass
   metadata:
     name: hybrid-class
     labels:
       app: noobaa
   spec:
     placementPolicy:
       tiers:
         - tier:
             mirrors:
               - mirror:
                   spread:
                     - cos-east-us
               - mirror:
                   spread:
                     - noobaa-test-bucket-for-ocp201907291921-11247_resource

2. Add the following lines to your standard Object Bucket Claim (OBC):

   additionalConfig:
     bucketclass: mirror-to-aws

   For more information about OBCs, see Section 2.4, “Object Bucket Claim”.

1. In your OpenShift Storage console, navigate to Dashboards → Object Service → select the noobaa link:

   

2. Click the buckets icon on the left side. You will see a list of your buckets:

   
3. Cick the bucket you want to update.

4. Click Edit Tier 1 Resources:

   

5. Select Mirror and check the relevant resources you want to use for this bucket. In the following example, we mirror data between on prem Ceph RGW to AWS:

   
6. Click Save.

Procedure

1. Identify the node that needs to be replaced.

2. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

3. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

4. Delete the node using the following command:

   $ oc delete nodes <node_name>

5. Create a new AWS machine instance with the required infrastructure. See Infrastructure requirements.
6. Create a new OpenShift Container Platform node using the new AWS machine instance.

7. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

8. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

9. Click Compute → Nodes, confirm if the new node is in Ready state.

10. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels.
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

11. Restart the mgr pod to update the OpenShift Container Storage with the new hostname.

    $ oc delete pod rook-ceph-mgr-xxxx

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the node that needs to be replaced. Take a note of its Machine Name.

3. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

4. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

5. Click Compute → Machines. Search for the required machine.
6. Besides the required machine, click the Action menu (⋮) → Delete Machine.
7. Click Delete to confirm the machine deletion. A new machine is automatically created.

8. Wait for new machine to start and transition into Running state.

   Important

   This activity may take at least 5-10 minutes or more.

9. Click Compute → Nodes, confirm if the new node is in Ready state.

10. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

11. Restart the mgr pod to update the OpenShift Container Storage with the new hostname.

    $ oc delete pod rook-ceph-mgr-xxxx

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Identify the AWS machine instance of the node that needs to be replaced.
2. Log in to AWS and terminate the identified AWS machine instance.
3. Create a new AWS machine instance with the required infrastructure. See Infrastructure requirements.
4. Create a new OpenShift Container Platform node using the new AWS machine instance.

5. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

6. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

7. Click Compute → Nodes, confirm if the new node is in Ready state.

8. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

9. Restart the mgr pod to update the OpenShift Container Storage with the new hostname.

   $ oc delete pod rook-ceph-mgr-xxxx

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Log in to OpenShift Web Console and click Compute → Nodes.
2. Identify the faulty node and click on its Machine Name.
3. Click Actions → Edit Annotations, and click Add More.
4. Add machine.openshift.io/exclude-node-draining and click Save.
5. Click Actions → Delete Machine, and click Delete.

6. A new machine is automatically created, wait for new machine to start.

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

7. Click Compute → Nodes, confirm if the new node is in Ready state.

8. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

9. [Optional]: If the failed AWS instance is not removed automatically, terminate the instance from AWS console.

10. Restart the mgr pod to update the OpenShift Container Storage with the new hostname.

    $ oc delete pod rook-ceph-mgr-xxxx

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Identify the node and its VM that needs to be replaced.

2. Mark the node as unschedulable using the following command:

   $ oc adm cordon <node_name>

3. Drain the node using the following command:

   $ oc adm drain <node_name> --force --delete-local-data --ignore-daemonsets

   Important

   This activity may take at least 5-10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when the new node is labeled and functional.

4. Delete the node using the following command:

   $ oc delete nodes <node_name>

5. Log in to vSphere and terminate the identified VM.

   Important

   VM should be deleted only from the inventory and not from the disk.

6. Create a new VM on vSphere with the required infrastructure. See Infrastructure requirements.
7. Create a new OpenShift Container Platform worker node using the new VM.

8. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

9. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

10. Click Compute → Nodes, confirm if the new node is in Ready state.

11. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node:

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

12. Restart the mgr pod to update the OpenShift Container Storage with the new hostname.

    $ oc delete pod rook-ceph-mgr-xxxx

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.

Procedure

1. Identify the node and its VM that needs to be replaced.

2. Delete the node using the following command:

   $ oc delete nodes <node_name>

3. Log in to vSphere and terminate the identified VM.

   Important

   VM should be deleted only from the inventory and not from the disk.

4. Create a new VM on vSphere with the required infrastructure. See Infrastructure requirements.
5. Create a new OpenShift Container Platform worker node using the new VM.

6. Check for certificate signing requests (CSRs) related to OpenShift Container Platform that are in Pending state:

   $ oc get csr

7. Approve all required OpenShift Container Platform CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

8. Click Compute → Nodes, confirm if the new node is in Ready state.

9. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

10. Restart the mgr pod to update the OpenShift Container Storage with the new hostname.

    $ oc delete pod rook-ceph-mgr-xxxx

Verification steps

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all other required OpenShift Container Storage pods are in Running state.
4. If verification steps fail, kindly contact Red Hat Support.