Chapter 1. Deploying OpenShift Container Storage

OpenShift Container Storage 4.3, installation is supported only on existing Red Hat OpenShift Container Platform (OCP) worker nodes. Follow the instructions in Section 1.1, “Deploying Red Hat OpenShift Container Storage on an existing Red Hat OpenShift Container Platform” to deploy OpenShift Container Storage.

Note

When you install OpenShift Container Storage in a restricted network environment, you need to apply a custom Network Time Protocol (NTP) configuration to the nodes, because by default, internet connectivity is assumed in OpenShift Container Platform and chronyd is configured to use *.rhel.pool.ntp.org servers. See https://access.redhat.com/solutions/4828941 and Configuring chrony time service for more details.

1.1. Deploying Red Hat OpenShift Container Storage on an existing Red Hat OpenShift Container Platform

The deployment process consists of two main parts:

  1. Install the OpenShift Container Storage Operator by following the instructions in Section 1.1.1, “Installing Red Hat OpenShift Container Storage Operator using the Operator Hub”.
  2. Create the OpenShift Container Storage service by following the instructions in Section 1.1.2, “Creating an OpenShift Container Storage service”.

For Red Hat Enterprise Linux based hosts in a user provisioned infrastructure (UPI), you need to enable container access to the underlying file system by following the instructions in Section 1.1.3, “Enabling file system access for containers on Red Hat Enterprise Linux based nodes”.

1.1.1. Installing Red Hat OpenShift Container Storage Operator using the Operator Hub

You can install Red Hat OpenShift Container Storage using the Red Hat OpenShift Container Platform Operator Hub on Amazon Web Services (AWS) and VMware vSphere platforms. For information about the hardware and software requirements, see Planning your deployment.

Prerequisites

  • You must be logged into OpenShift Container Platform (OCP) cluster.
  • You must have at least three worker nodes in OCP cluster.
  • You must create a namespace called openshift-storage as follows:

    1. Click Administration → Namespaces in the left pane of the OpenShift Web Console.
    2. Click Create Namespace.
    3. In the Create Namespace dialog box, enter openshift-storage for Name and openshift.io/cluster-monitoring=true for Labels. This label is required to get the dashboards.
    4. Select No restrictions option for Default Network Policy.
    5. Click Create.
Note

When you need to override the cluster-wide default node selector for OpenShift Container Storage, you can use the following command in command line interface to specify a blank node selector for the openshift-storage namespace:

$ oc annotate namespace openshift-storage openshift.io/node-selector=

Procedure

  1. Click Operators → OperatorHub in the left pane of the OpenShift Web Console.

    Figure 1.1. List of operators in the Operator Hub

    Screenshot of list of operators in the Operator Hub of the OpenShift Web Console.
  2. Search for OpenShift Container Storage Operator from the list of operators and click on it.
  3. On the OpenShift Container Storage Operator page, click Install.
  4. On the Create Operator Subscription page, the Installation Mode, Update Channel, and Approval Strategy options can be set.

    Figure 1.2. Create Operator Subscription page

    Screenshot of create operator subscription page.
    1. Select A specific namespace on the cluster for the Installation Mode option.

      • Select openshift-storage namespace from the drop down menu.
    2. Select stable-4.3 as the update channel based on your requirement.
    3. Select an Approval Strategy:

      • Automatic specifies that you want OpenShift Container Platform to upgrade OpenShift Container Storage automatically.
      • Manual specifies that you want to have control to upgrade OpenShift Container Storage manually.
  5. Click Subscribe.

    Figure 1.3. Installed operators

    Screenshot of the installed operators.

    The Installed Operators page is displayed with the status of the operator.

Verification steps

  • Verify that the lib-bucket-provisioner and OpenShift Container Storage Operator show the Status as Succeeded.

1.1.2. Creating an OpenShift Container Storage service

You need to create a new OpenShift Container Storage service after you install OpenShift Container Storage operator on a user provisioned cloud for both Amazon Web Services (AWS) and VMware vSphere platforms.

Prerequisites

Procedure

  1. Click OperatorsInstalled Operators from the left pane of the OpenShift Web Console to view the installed operators.
  2. On the Installed Operator page, select openshift-storage from the Project drop down list to switch to the openshift-storage project.
  3. Click OpenShift Container Storage Operator.

    OpenShift Container Storage operator creates a OCSInitialization resource automatically.

  4. On the OpenShift Container Storage Operator page, scroll right and click the Storage Cluster tab.

    Figure 1.4. OpenShift Container Storage Operator page

    Screenshot of OpenShift Container Storage operator page.
  5. On the OCS Cluster Services page, click Create OCS Cluster Service.

    Figure 1.5. Create New OCS Service page

    Screenshot of create new OCS service page.
  6. On the Create New OCS Service page, perform the following:

    1. Select at least three worker nodes from the available list of nodes for the use of OpenShift Container Storage service. Ensure that the nodes are in different Location.
    2. Storage Class is set by default to gp2 for AWS and thin for VMware.
    3. Select OCS Service Capacity from drop down list.

      Note

      Once you select the initial storage capacity here, you can add more capacity only in this increment.

  7. Click Create.

    The Create button is enabled only after you select three nodes. A new storage cluster of three volumes will be created with one volume per worker node. The default configuration uses a replication factor of 3.

Verification steps

1.1.3. Enabling file system access for containers on Red Hat Enterprise Linux based nodes

Deploying OpenShift Container Platform on a Red Hat Enterprise Linux base in a user provisioned infrastructure (UPI) does not automatically provide container access to the underlying Ceph file system. This is a bug tracked by RHSTOR-787.

Note

This process is not necessary for hosts based on Red Hat Enterprise Linux CoreOS.

Procedure

Perform the following steps on each node in your cluster.

  1. Log in to the Red Hat Enterprise Linux based node and open a terminal.
  2. Verify that the node has access to the rhel-7-server-extras-rpms repository.

    # subscription-manager repos --list-enabled | grep rhel-7-server

    If you do not see both rhel-7-server-rpms and rhel-7-server-extras-rpms in the output, or if there is no output, run the following commands to enable each repository.

    # subscription-manager repos --enable=rhel-7-server-rpms
    # subscription-manager repos --enable=rhel-7-server-extras-rpms
  3. Install the required packages.

    # yum install -y policycoreutils container-selinux
  4. Persistently enable container use of the Ceph file system in SELinux.

    # setsebool -P container_use_cephfs on
  5. Verify that containers can now access OpenShift Container Storage hosted on this node.

1.2. Installing OpenShift Container Storage using local storage devices

Use this section to install OpenShift Container Storage on bare metal, Amazon EC2, and VMware infrastructures where OpenShift Container Platform is already installed.

Important

Installing OpenShift Container Storage on bare metal, Amazon EC2, and VMware using local storage operator is a Technology Preview feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

To install OpenShift Container Storage using local storage devices, perform the following steps:

1.2.1. Requirements for installing OpenShift Container Storage using local storage devices

  • You must have at least three OpenShift Container Platform worker nodes in the cluster with locally attached storage devices on each of them.

    • Each of the three worker nodes must have at least one raw block device available to be used by OpenShift Container Storage.
    • For minimum starting node requirements, see Node Requirements section in Planning guide.
  • The devices to be used must be empty, that is, there should be no PVs, VGs, or LVs remaining on the disks.
  • You must have a minimum of three labeled nodes.

    • Each worker node that has local storage devices to be used by OpenShift Container Storage must have a specific label to deploy OpenShift Container Storage pods. To label the nodes, use the following command:

      $ oc label nodes <NodeName> cluster.ocs.openshift.io/openshift-storage=''
  • There should not be any storage providers managing locally mounted storage on the storage nodes that would conflict with the use of Local Storage Operator for Red Hat OpenShift Container Storage.

1.2.2. Installing Local Storage Operator

Use this procedure to install Local Storage Operator from the Operator Hub before creating OpenShift Container Storage clusters on local storage devices in Amazon EC2, VMware, and bare metal infrastructures.

Prerequisites

  • Create a namespace called local-storage as follows:

    1. Click Administration → Namespaces in the left pane of the OpenShift Web Console.
    2. Click Create Namespace.
    3. In the Create Namespace dialog box, enter local-storage for Name.
    4. Select No restrictions option for Default Network Policy.
    5. Click Create.

Procedure

  1. Click Operators → OperatorHub in the left pane of the OpenShift Web Console.
  2. Search for Local Storage Operator from the list of operators and click on it.
  3. Click Install.

    Figure 1.6. Create Operator Subscription page

    Screenshot of Operator Subscription page.
  4. Select A specific namespace on the cluster for the Installation Mode option.

    • Select local-storage namespace from the drop down menu.
  5. Select a desired value for the Update Channel option.
  6. Select the desired Approval Strategy.
  7. Click Subscribe.
  8. Verify that the Local Storage Operator show the Status as Succeeded.

1.2.3. Finding available storage devices

Use this procedure to identify the device name for each of the three or more worker nodes that you have labeled with OpenShift Container Storage label, cluster.ocs.openshift.io/openshift-storage='' before creating PVs for bare metal, Amazon EC2, or VMware storage devices.

Procedure

  1. List and verify the name of the worker nodes with the OpenShift Container Storage label.

    $ oc get nodes -l cluster.ocs.openshift.io/openshift-storage=

    Example output:

    NAME                                        STATUS   ROLES    AGE     VERSION
    ip-10-0-135-71.us-east-2.compute.internal    Ready    worker   6h45m   v1.16.2
    ip-10-0-145-125.us-east-2.compute.internal   Ready    worker   6h45m   v1.16.2
    ip-10-0-160-91.us-east-2.compute.internal    Ready    worker   6h45m   v1.16.2
  2. Log in to each worker node that is used for OpenShift Container Storage resources and find the unique by-id device name for each available raw block device.

    $ oc debug node/<Nodename>

    Example output:

    $ oc debug node/ip-10-0-135-71.us-east-2.compute.internal
    Starting pod/ip-10-0-135-71us-east-2computeinternal-debug ...
    To use host binaries, run `chroot /host`
    Pod IP: 10.0.135.71
    If you don't see a command prompt, try pressing enter.
    sh-4.2# chroot /host
    sh-4.4# lsblk
    NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    xvda                         202:0    0   120G  0 disk
    |-xvda1                      202:1    0   384M  0 part /boot
    |-xvda2                      202:2    0   127M  0 part /boot/efi
    |-xvda3                      202:3    0     1M  0 part
    `-xvda4                      202:4    0 119.5G  0 part
      `-coreos-luks-root-nocrypt 253:0    0 119.5G  0 dm   /sysroot
    nvme0n1                      259:0    0   1.7T  0 disk
    nvme1n1                      259:1    0   1.7T  0 disk

    In this example, the local devices available are nvme0n1 and nvme1n1.

  3. Find the unique by-id device name depending on the hardware serial number for each device.

    sh-4.4# ls -l /dev/disk/by-id/
    total 0
    lrwxrwxrwx. 1 root root 10 Mar 17 16:24 dm-name-coreos-luks-root-nocrypt -> ../../dm-0
    lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC -> ../../nvme0n1
    lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-Amazon_EC2_NVMe_Instance_Storage_AWS60382E5D7441494EC -> ../../nvme1n1
    lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-nvme.1d0f-4157533130333832453544373434313439344543-416d617a6f6e20454332204e564d6520496e7374616e63652053746f72616765-00000001 -> ../../nvme0n1
    lrwxrwxrwx. 1 root root 13 Mar 17 16:24 nvme-nvme.1d0f-4157533630333832453544373434313439344543-416d617a6f6e20454332204e564d6520496e7374616e63652053746f72616765-00000001 -> ../../nvme1n1

    In this example, all the OpenShift Container Storage worker nodes are of the Amazon EC2 type i3.2xlarge. So, all the three worker nodes have the same type of machine but the by-id identifier is unique for every local device. The lsblk command shows the last two devices, nvme0n1 and nvme1n1 with a size of 1.7 TB each.

    For each worker node that has the OpenShift Container Storage label (a minimum of three), you need to find the unique by-id device names. In this example, the by-id device names are:

    • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC
    • nvme-Amazon_EC2_NVMe_Instance_Storage_AWS60382E5D7441494EC
Note

You must repeat finding the device name by-id for all the other nodes that have the storage devices to be used by OpenShift Container Storage. See https://access.redhat.com/solutions/4928841 for more details.

1.2.4. Creating OpenShift Container Storage cluster on Amazon EC2 storage optimized - i3en.2xlarge instance type

Use this procedure to create OpenShift Container Storage cluster on Amazon EC2 (storage optimized - i3en.2xlarge instance type) infrastructure, which involves:

  1. Creating PVs by using the LocalVolume CR
  2. Creating a new StorageClass

The Amazon EC2 storage optimized - i3.2xlarge instance type includes two non-volatile memory express (NVMe) disks. The example in this procedure illustrates the use of both the disks that the instance type comes with.

Warning

It is not recommended to use ephemeral storage of Amazon EC2 for OpenShift Container Storage persistent data, because stopping all the three nodes can cause data loss. It is recommended to use ephemeral storage only in scenarios such as the following:

  • Cloud burst where data is copied from another location for a specific data crunching, which is limited in time
  • Development or testing environment

When you are using the ephemeral storage of Amazon EC2, it is recommended to:

  • Use three availability zones to decrease the risk of losing all the data
  • Limit the number of users with ec2:StopInstances permissions to avoid instance shutdown by mistake

Prerequisites

  • Ensure that all the requirements in the Requirements for installing OpenShift Container Storage using local storage devices section are met.
  • Verify your OpenShift Container Platform worker nodes are labeled for OpenShift Container Storage, which is used as the nodeSelector.

    oc get nodes -l cluster.ocs.openshift.io/openshift-storage -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}'

    Example output:

    ip-10-0-135-71.us-east-2.compute.internal
    ip-10-0-145-125.us-east-2.compute.internal
    ip-10-0-160-91.us-east-2.compute.internal

Procedure

  1. Create local persistent volumes (PVs) on the storage nodes using LocalVolume custom resource (CR).

    Example of LocalVolume CR local-storage-block.yaml using OpenShift Storage Container label as node selector and by-id device identifier:

    apiVersion: local.storage.openshift.io/v1
    kind: LocalVolume
    metadata:
      name: local-block
      namespace: local-storage
      labels:
        app: ocs-storagecluster
    spec:
      nodeSelector:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cluster.ocs.openshift.io/openshift-storage
                operator: In
                values:
                  - ''
      storageClassDevices:
        - storageClassName: localblock
          volumeMode: Block
          devicePaths:
            - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441494EC   # <-- modify this line
            - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84FE3E9   # <-- modify this line
            - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE4   # <-- modify this line
            - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS10382E5D7441464EP   # <-- modify this line
            - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS1F45C01D7E84F43E7   # <-- modify this line
            - /dev/disk/by-id/nvme-Amazon_EC2_NVMe_Instance_Storage_AWS136BC945B4ECB9AE8   # <-- modify this line

    Each Amazon EC2 instance has two disks and this example uses both the disks.

  2. Create the LocalVolume CR.

    $ oc create -f local-storage-block.yaml

    Example output:

    localvolume.local.storage.openshift.io/local-block created
  3. Check if the pods are created.

    $ oc -n local-storage get pods

    Example output:

    NAME                                      READY   STATUS    RESTARTS   AGE
    local-block-local-diskmaker-59rmn         1/1     Running   0          15m
    local-block-local-diskmaker-6n7ct         1/1     Running   0          15m
    local-block-local-diskmaker-jwtsn         1/1     Running   0          15m
    local-block-local-provisioner-6ssxc       1/1     Running   0          15m
    local-block-local-provisioner-swwvx       1/1     Running   0          15m
    local-block-local-provisioner-zmv5j       1/1     Running   0          15m
    local-storage-operator-7848bbd595-686dg   1/1     Running   0          15m
  4. Check if the PVs are created.

    You must see a new PV for each of the local storage devices on the three worker nodes. Refer the example in the Finding available storage devices section that shows two available storage devices per worker node with a size 1.7 TB for each node.

    $ oc get pv

    Example output:

    NAME               CAPACITY ACCESS MODES  RECLAIM POLICY STATUS     CLAIM   STORAGECLASS  REASON   AGE
    local-pv-1a46bc79  2328Gi   RWO           Delete         Available          localblock             14m
    local-pv-429d90ee  2328Gi   RWO           Delete         Available          localblock             14m
    local-pv-4d0a62e3  2328Gi   RWO           Delete         Available          localblock             14m
    local-pv-55c05d76  2328Gi   RWO           Delete         Available          localblock             14m
    local-pv-5c7b0990  2328Gi   RWO           Delete         Available          localblock             14m
    local-pv-a6b283b   2328Gi   RWO           Delete         Available          localblock             14m
  5. Check if a new StorageClass is created due to the creation of LocalVolume CR. This StorageClass is used while creating StorageCluster to create PVCs.

    $ oc get sc | grep localblock

    Example output:

    NAME            PROVISIONER                    AGE
    localblock      kubernetes.io/no-provisioner   7m46s
  6. Create StorageCluster CR that uses the localblock StorageClass and the three PVs that are created.

    Example StorageCluster CR ocs-cluster-service.yaml using monDataDirHostPath and localblock StorageClass.

    apiVersion: ocs.openshift.io/v1
    kind: StorageCluster
    metadata:
      name: ocs-storagecluster
      namespace: openshift-storage
    spec:
      manageNodes: false
      resources:
        mds:
          limits:
            cpu: 3
          requests:
            cpu: 1
        noobaa-core:
          limits:
            cpu: 2
            memory: 8Gi
          requests:
            cpu: 1
            memory: 8Gi
        noobaa-db:
          limits:
            cpu: 2
            memory: 8Gi
          requests:
            cpu: 1
            memory: 8Gi
      monDataDirHostPath: /var/lib/rook
      storageDeviceSets:
        - count: 2
          dataPVCTemplate:
            spec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 2328Gi
              storageClassName: localblock
              volumeMode: Block
          name: ocs-deviceset
          placement: {}
          portable: false
          replica: 3
          resources: {}
    Important

    To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

  7. Create StorageCluster CR.

    $ oc create -f ocs-cluster-service.yaml

    Example output

    storagecluster.ocs.openshift.io/ocs-cluster-service created

1.2.5. Creating OpenShift Container Storage cluster on VMware

Use this procedure to create storage cluster on VMware infrastructure.

VMware supports the following three types of local storage:

  • Virtual machine disk (VMDK)
  • Raw device mapping (RDM)
  • VMDirectPath I/O

Prerequisites

  • Ensure that all the requirements in the Requirements for installing OpenShift Container Storage using local storage devices section are met.
  • You must have three worker nodes with the same storage size and type attached to each node to use local storage devices on VMware.
  • Verify your OpenShift Container Platform worker nodes are labeled for OpenShift Container Storage:

    oc get nodes -l cluster.ocs.openshift.io/openshift-storage -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}'

To identify storage devices on each node, refer to Finding available storage devices.

Procedure

  1. Create the LocalVolume CR for block PVs.

    Example of LocalVolume CR local-storage-block.yaml using OpenShift Container Storage label as node:

    apiVersion: local.storage.openshift.io/v1
    kind: LocalVolume
    metadata:
      name: local-block
      namespace: local-storage
      labels:
        app: ocs-storagecluster
    spec:
      nodeSelector:
        nodeSelectorTerms:
        - matchExpressions:
            - key: cluster.ocs.openshift.io/openshift-storage
              operator: In
              values:
              - ""
      storageClassDevices:
        - storageClassName: localblock
          volumeMode: Block
          devicePaths:
            - /dev/disk/by-id/scsi-36000c2991c27c2e5ba7c47d1e4352de2   # <-- modify this line
            - /dev/disk/by-id/scsi-36000c29682ca9e347926406711f3dc4e   # <-- modify this line
            - /dev/disk/by-id/scsi-36000c296aaf03a9b1e4b01d086bc6348   # <-- modify this line
  2. Create LocalVolume CR for block PVs.

    $ oc create -f local-storage-block.yaml

    Example output:

    localvolume.local.storage.openshift.io/local-block created
  3. Check if the pods are created.

    $ oc -n local-storage get pods

    Example output:

    NAME                                      READY   STATUS    RESTARTS   AGE
    local-block-local-diskmaker-5brzv         1/1     Running   0          31s
    local-block-local-diskmaker-8sxcs         1/1     Running   0          31s
    local-block-local-diskmaker-s7s9p         1/1     Running   0          31s
    local-block-local-provisioner-9cbw8       1/1     Running   0          31s
    local-block-local-provisioner-cpddv       1/1     Running   0          31s
    local-block-local-provisioner-f6h7h       1/1     Running   0          31s
    local-storage-operator-75b9776b75-vwdzh   1/1     Running   0          2m47s
  4. Check the new localblock StorageClass.

    $ oc get sc | grep localblock

    Example output:

    NAME            PROVISIONER                     AGE
    localblock      kubernetes.io/no-provisioner    8m38s
  5. Check the PVs that are created with the Available status.

    $ oc get pv

    Example output:

    NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
    local-pv-150fdc87   100Gi      RWO            Delete           Available           localblock              2m11s
    local-pv-183bfc0a   100Gi      RWO            Delete           Available           localblock              2m11s
    local-pv-b2f5cb25   100Gi      RWO            Delete           Available           localblock              2m21s

    In this example, three PVs are used for OSD storage (100 GB).

  6. Create StorageCluster CR ocs-cluster-service-VMware.yaml that uses the monDataDirHostPath and localblock StorageClass.

    apiVersion: ocs.openshift.io/v1
    kind: StorageCluster
    metadata:
      name: ocs-storagecluster
      namespace: openshift-storage
    spec:
      manageNodes: false
      monDataDirHostPath: /var/lib/rook
      storageDeviceSets:
      - count: 1
        dataPVCTemplate:
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 100Gi
            storageClassName: localblock
            volumeMode: Block
        name: ocs-deviceset
        placement: {}
        portable: false
        replica: 3
        resources: {}
    Important

    To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

  7. Create StorageCluster CR.

    $ oc create -f ocs-cluster-service-VMware.yaml

    Example output:

    storagecluster.ocs.openshift.io/ocs-storagecluster created

1.2.6. Creating OpenShift Container Storage cluster on bare metal

Prerequisites

  • Ensure that all the requirements in the Requirements for installing OpenShift Container Storage using local storage devices section are met.
  • You must have three worker nodes with the same storage size and type attached to each node (for example, 2TB NVMe hard drive) to use local storage devices on bare metal.
  • Verify your OpenShift Container Platform worker nodes are labeled for OpenShift Container Storage:

    oc get nodes -l cluster.ocs.openshift.io/openshift-storage -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}'

To identify storage devices on each node, refer to Finding available storage devices.

Procedure

  1. Create LocalVolume CR for block PVs.

    Example of LocalVolume CR local-storage-block.yaml using OCS label as node selector.

    apiVersion: local.storage.openshift.io/v1
    kind: LocalVolume
    metadata:
      name: local-block
      namespace: local-storage
      labels:
        app: ocs-storagecluster
    spec:
      nodeSelector:
        nodeSelectorTerms:
        - matchExpressions:
            - key: cluster.ocs.openshift.io/openshift-storage
              operator: In
              values:
              - ""
      storageClassDevices:
        - storageClassName: localblock
          volumeMode: Block
          devicePaths:
            - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY81260978128A   # <-- modify this line
            - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY80440W5U128A   # <-- modify this line
            - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYB85AABDE128A   # <-- modify this line
            - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY0A60CB81128A   # <-- modify this line
            - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPY0093D45E128A   # <-- modify this line
            - /dev/disk/by-id/nvme-INTEL_SSDPEKKA128G7_BTPYE46F6060128A   # <-- modify this line
  2. Create the LocalVolume CR for block PVs.

    $ oc create -f local-storage-block.yaml
  3. Check if the pods are created.

    $ oc -n local-storage get pods
  4. Check if the PVs are created.

    $ oc get pv
  5. Check for the new localblock StorageClass.

    oc get sc | grep localblock

    Example output:

    NAME            PROVISIONER                     AGE
    localblock     kubernetes.io/no-provisioner    10m20s
  6. Create StorageCluster CR cluster-service-metal.yaml using monDataDirHostPath and localblock storage classes.

    apiVersion: ocs.openshift.io/v1
    kind: StorageCluster
    metadata:
      name: ocs-storagecluster
      namespace: openshift-storage
    spec:
      manageNodes: false
      monDataDirHostPath: /var/lib/rook
      storageDeviceSets:
      - count: 2
        dataPVCTemplate:
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 2Ti
            storageClassName: localblock
            volumeMode: Block
        name: ocs-deviceset
        placement: {}
        portable: false
        replica: 3
        resources: {}
    Important

    To ensure that the OSDs have a guaranteed size across the nodes, the storage size for storageDeviceSets must be specified as less than or equal to the size of the desired PVs created on the nodes.

  7. Create the StorageCluster CR.

    $ oc create -f cluster-service-metal.yaml

    Example output:

    storagecluster.ocs.openshift.io/ocs-storagecluster created