Chapter 4. Scaling storage nodes

Procedure

1. Navigate to the OpenShift Web Console.
2. Click on Operators on the left navigation bar.
3. Select Installed Operators.

4. In the window, click OpenShift Container Storage Operator:

   

5. In the top navigation bar, scroll right and click Storage Cluster tab.

   
6. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.

7. Select Add Capacity from the options menu.

   

   From this dialog box, you can set the requested additional capacity and the storage class. Add capacity shows the capacity selected at the time of installation and allows to add the capacity only in this increment. On AWS, the storage class should be set to gp2. On VMWare, the storage class should be set to thin.

   Note

   The effectively provisioned capacity will be three times as much as what you see in the Raw Capacity field because OpenShift Container Storage uses a replica count of 3.

8. Click Add. You can see the status of the storage cluster after it reaches the Ready state. You might need to wait a couple of minutes after you see the Ready state.

Verification steps

1. Navigate to Overview → Persistent Storage tab, then check the Capacity breakdown card.

   
2. Note that the capacity increases based on your selections.

Procedure

1. To add storage capacity to OpenShift Container Platform nodes with OpenShift Container Storage installed, you need to

   1. Find the unique by-id identifier for available devices that you want to add, that is, a minimum of one device per worker node. Follow procedure finding available storage devices.

      Note

      Make sure you perform this process for all the existing nodes (minimum of 3) for which you want to add storage.

   2. Add unique device by-id.

      $ oc edit -n local-storage localvolume local-block

      Example output:

      spec:
        logLevel: Normal
        managementState: Managed
        nodeSelector:
          nodeSelectorTerms:
          - matchExpressions:
            - key: cluster.ocs.openshift.io/openshift-storage
              operator: In
              values:
              - ""
        storageClassDevices:
        - devicePaths:
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402P51P0GGN
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402LM1P0GGN
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402M21P0GGN
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402B71P0GGN    # newly  added device by-id
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402A31P0GGN    # newly  added device by-id
          - /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T7_PHLF733402Q71P0GGN    # newly  added device by-id
          storageClassName: localblock
          volumeMode: Block

      Make sure to save the changes after editing the CR.

      localvolume.local.storage.openshift.io/local-block edited

      You can see that in this CR new devices using by-id have been added. Each device maps to nvme1n1 on one of the three worker nodes.

      • nvme-INTEL_SSDPE2KX010T7_PHLF733402B71P0GGN
      • nvme-INTEL_SSDPE2KX010T7_PHLF733402A31P0GGN
      • nvme-INTEL_SSDPE2KX010T7_PHLF733402Q71P0GGN

2. Display PVs with storageclass name used in localVolume CR.

   $ oc get pv | grep localblock | grep Available

   Example output:

   local-pv-5ee61dcc   894Gi   RWO     Delete  Available   localblock      2m35s
   local-pv-b1fa607a   894Gi   RWO     Delete  Available   localblock      2m27s
   local-pv-e971c51d   894Gi   RWO     Delete  Available   localblock      2m22s
   ...

   There are three more available PVs of same size which will be used for new OSDs.

3. To expand storage capacity, increase the count by 1 for StorageDeviceSets in StorageCluster CR.

   $ oc edit storageclusters.ocs.openshift.io -n openshift-storage

   Example output:

   spec:
     monDataDirHostPath: /var/lib/rook
     storageDeviceSets:
     - config: {}
       count: 2    # <-- increase this count by 1
       dataPVCTemplate:
         metadata:
           creationTimestamp: null
         spec:
           accessModes:
           - ReadWriteOnce
           resources:
             requests:
               storage: 894Gi
           storageClassName: localblock
           volumeMode: Block
         status: {}
       name: ocs-deviceset
       placement: {}
       replica: 3
       resources: {}
     version: 4.4.0

   Make sure to save the changes after editing the CR.

   storagecluster.ocs.openshift.io/ocs-storagecluster edited

   Important

   To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

4. Verify that there are three new OSDs running and their corresponding new PVCs are created.

   $ oc get -n openshift-storage pods -l app=rook-ceph-osd

   Example output:

   NAME                               READY   STATUS    RESTARTS   AGE
   rook-ceph-osd-0-77c4fdb758-qshw4   1/1     Running   0          1h
   rook-ceph-osd-1-8645c5fbb6-656ks   1/1     Running   0          1h
   rook-ceph-osd-2-86895b854f-r4gt6   1/1     Running   0          1h
   rook-ceph-osd-3-dc7f787dd-gdnsz    1/1     Running   0          10m
   rook-ceph-osd-4-554b5c46dd-hbf9t   1/1     Running   0          10m
   rook-ceph-osd-5-5cf94c4448-k94j6   1/1     Running   0          10m

   In the above example, osd-3, osd-4, and osd-5 are the newly added pods to the OpenShift Container Storage cluster.

   $ oc get pvc -n openshift-storage |grep localblock

   Example output:

   ocs-deviceset-0-0-qc29m   Bound    local-pv-fc5562d3    894Gi   RWO  localblock  1h
   ocs-deviceset-0-1-qdmrl   Bound    local-pv-b1fa607a    894Gi   RWO  localblock  10m
   ocs-deviceset-1-0-mpwmk   Bound    local-pv-58cdd0bc    894Gi   RWO  localblock  1h
   ocs-deviceset-1-1-85892   Bound    local-pv-e971c51d    894Gi   RWO  localblock  10m
   ocs-deviceset-2-0-rll47   Bound    local-pv-29d8ad8d    894Gi   RWO  localblock  1h
   ocs-deviceset-2-1-cgth2   Bound    local-pv-5ee61dcc    894Gi   RWO  localblock  10m

   In the above example, we see three new PVCs are created.

Procedure

1. Navigate to Compute → Machine Sets.
2. On the machine set where you want to add nodes, select Edit Count.
3. Add the amount of nodes, and click Save.
4. Click Compute → Nodes and confirm if the new node is in Ready state.

5. Apply the OpenShift Container Storage label to the new node.

   1. For the new node, Action menu (⋮) → Edit Labels.
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

Procedure

1. Depending on whether you are adding a node on an AWS user provisioned infrastructure or a VMware user-provisioned infrastructure, perform the following steps:

   • For AWS

     1. Create a new AWS machine instance with the required infrastructure. See Supported Infrastructure and Platforms.
     2. Create a new OpenShift Container Platform node using the new AWS machine instance.

   • For VMware:

     1. Create a new VM on vSphere with the required infrastructure. See Supported Infrastructure and Platforms.
     2. Create a new OpenShift Container Platform worker node using the new VM.

2. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state:

   $ oc get csr

3. Approve all required OpenShift Container Storage CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

4. Click Compute → Nodes, confirm if the new node is in Ready state.

5. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Note

     It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

Procedure

1. Depending on whether you are adding a node on bare metal, Amazon EC2, or VMware infrastructure, perform the following steps:

   • For Amazon EC2

     1. Create a new Amazon EC2 I3 machine instance with the required infrastructure. See Creating a MachineSet in AWS and Supported Infrastructure and Platforms.
     2. Create a new OpenShift Container Platform node using the new Amazon EC2 I3 machine instance.

   • For VMware:

     1. Create a new VM on vSphere with the required infrastructure. See Supported Infrastructure and Platforms.
     2. Create a new OpenShift Container Platform worker node using the new VM.

   • For bare metal:

     1. Get a new bare metal machine with the required infrastructure. See Supported Infrastructure and Platforms.
     2. Create a new OpenShift Container Platform node using the new bare metal machine.

2. Check for certificate signing requests (CSRs) related to OpenShift Container Storage that are in Pending state:

   $ oc get csr

3. Approve all required OpenShift Container Storage CSRs for the new node:

   $ oc adm certificate approve <Certificate_Name>

4. Click Compute → Nodes, confirm if the new node is in Ready state.

5. Apply the OpenShift Container Storage label to the new node using any one of the following:

   From User interface

   1. For the new node, click Action Menu (⋮) → Edit Labels
   2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

   From Command line interface

   • Execute the following command to apply the OpenShift Container Storage label to the new node:

     $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

     Note

     It is recommended to add 3 nodes each in different zones. You must add 3 nodes and perform this procedure for all of them.

1. Execute the following command and verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*