Chapter 4. Scaling storage nodes

To scale the storage capacity of OpenShift Container Storage, you can do either of the following:

  • Scale up storage nodes - Add storage capacity to the existing Red Hat OpenShift Container Storage worker nodes
  • Scale out storage nodes - Add new worker nodes to increase the storage capacity

4.1. Requirements for scaling storage nodes

Before you proceed to scale the storage nodes, refer to the following sections to understand the node requirements for your specific Red Hat OpenShift Container Storage instance:

Warning

Always ensure that you have plenty of storage capacity.

If storage ever fills completely, it is not possible to add capacity or delete or migrate content away from the storage to free up space. Completely full storage is very difficult to recover.

Capacity alerts are issued when cluster storage capacity reaches 75% (near-full) and 85% (full) of total capacity. Always address capacity warnings promptly, and review your storage regularly to ensure that you do not run out of storage space.

If you do run out of storage space completely, contact Red Hat Customer Support.

4.1.1. Supported Deployments for Red Hat OpenShift Container Storage

  • User-provisioned infrastructure:

    • Amazon Web Services (AWS)
    • VMware

  • Installer-provisioned infrastructure:

    • Amazon Web Services (AWS)

4.2. Scaling up storage by adding capacity to your OpenShift Container Storage nodes

Use this procedure to add storage capacity and performance to your configured Red Hat OpenShift Container Storage worker nodes.

Prerequisites

  • A running OpenShift Container Storage Platform
  • Administrative privileges on the OpenShift Web Console

Procedure

  1. Navigate to the OpenShift Web Console.
  2. Click on Operators on the left navigation bar.
  3. Select Installed Operators.

  4. In the window, click OpenShift Container Storage Operator:

    ocs installed operators

  5. In the top navigation bar, scroll right and click Storage Cluster tab.

    OCS Storage Cluster overview reachit
  6. The visible list should have only one item. Click (⋮) on the far right to extend the options menu.

  7. Select Add Capacity from the options menu.

    OCS add capacity dialog menu

    From this dialog box, you can set the requested additional capacity and the storage class. The size should always be set in multiples of 2TiB. On AWS, the storage class should be set to gp2. On VMWare, the storage class should be set to thin.

    Note

    The effectively provisioned capacity will be three times as much as you put into the Requested Capacity field because OpenShift Container Storage uses a replica count of 3.

  8. Once you are done with your setting, click Add. You will not see the status of the storage cluster until it reaches Ready. You might need to wait a couple of minutes after you see the Ready state.

Verification steps

  1. Navigate to Dashboards -→ Persistent Storage tab, then check the Capacity card.

    ocs add capacity expansion verification

  2. Note that the capacity increases based on your selections.

    ocs add capacity expansion verification capacity card
Important

As of OpenShift Container Storage 4.2, cluster reduction, whether by reducing OSDs or nodes, is not supported.

4.3. Scaling out storage capacity

To scale out storage capacity, you need to perform the following steps:

  • Add a new node
  • Verify that the new node is added successfully
  • Scale up the storage capacity

4.3.1. Adding a node

You can add additional nodes to increase the storage capacity when existing worker nodes are already running at their maximum supported OSDs, which is 3 OSDs of size 2TiB.

Depending on the type of your deployment, you can choose one of the following procedures to add a storage node:

4.3.1.1. Adding a node on an AWS installer-provisioned infrastructure

Prerequisites

  • A running OpenShift Container Storage Platform
  • Access to the OpenShift Web Console

Procedure

  1. Navigate to the OpenShift Web Console.
  2. Navigate to Compute → Machine Sets.
  3. On the machine set where you want to add nodes, click the Action menu (⋮) → Edit Count beside the required machine.
  4. Add the amount of nodes, and click Save.

  5. Click Compute → Nodes and confirm if the new node is in Ready state.

    Note

    It might take some time for the new node to reach the Ready state.

  6. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface
    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.
    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node: 

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
      Note

      It is recommended to add 3 worker nodes. For AWS deployments with three availability zones, make sure that you add each node to a different availability zone and perform this procedure on all the nodes. However, for AWS deployments with one availability zone, you add the 3 worker nodes to the available zone and perform this procedure on that zone.

Verification steps

To verify that the new node is added, see Section 4.3.2, “Verifying the addition of a new node”.

4.3.1.2. Adding a node on an AWS or a VMware user-provisioned infrastructure

Prerequisites

  • A running OpenShift Container Storage Platform
  • Access to the OpenShift Web Console

Procedure

  1. Depending on whether you are adding a node on an AWS user provisioned infrastructure or a VMware user-provisioned infrastructure, perform the following steps:

    • For AWS

      1. Create a new AWS machine instance with the required infrastructure. See Infrastructure requirements.
      2. Create a new OpenShift Container Platform node using the new AWS machine instance.

    • For VMware:

      1. Create a new VM on vSphere with the required infrastructure. See Infrastructure requirements.
      2. Create a new OpenShift Container Platform worker node using the new VM.

  2. Check for certificate signing requests (CSRs) that are in Pending state: 

    $ oc get csr

  3. Approve all required CSRs for the new node: 

    $ oc adm certificate approve <Certificate_Name>
  4. Click Compute → Nodes, confirm if the new node is in Ready state.

  5. Apply the OpenShift Container Storage label to the new node using any one of the following:

    From User interface
    1. For the new node, click Action Menu (⋮) → Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.
    From Command line interface

    • Execute the following command to apply the OpenShift Container Storage label to the new node: 

      $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""
      Note

      It is recommended to add 3 worker nodes. For AWS deployments with three availability zones, make sure that you add each node to a different availability zone and perform this procedure on all the nodes. However, for AWS deployments with one availability zone and VMware deployments that have only one availability zone, you add the 3 worker nodes to the available zone and perform this procedure on that zone.

Verification steps

To verify that the new node is added, see Section 4.3.2, “Verifying the addition of a new node”.

4.3.2. Verifying the addition of a new node

  1. Execute the following command and verify that the new node is present in the output: 

    $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

  2. Click Workloads → Pods, confirm that at least the following pods on the new node are in Running state:

    • csi-cephfsplugin-*
    • csi-rbdplugin-*

  3. Verify the health of OpenShift Container Storage cluster using the persistent storage dashboard:

    1. Click Home → Dashboards from the left pane of the OpenShift Web Console and click the Persistent Storage tab.
    2. In the Health card, verify that the cluster health is displayed as ocs-storagecluster is healthy.

4.3.3. Scaling up storage capacity

To scale up storage capacity, perform the steps in the procedure, Scaling up storage by adding capacity to your OpenShift Container Storage nodes.