Chapter 3. Deploy OpenShift Data Foundation using IBM FlashSystem

OpenShift Data Foundation can use IBM FlashSystem storage available for consumption through OpenShift Container Platform clusters. You need to install the OpenShift Data Foundation operator and then create an OpenShift Data Foundation cluster for IBM FlashSystem storage.

3.1. Installing Red Hat OpenShift Data Foundation Operator

You can install Red Hat OpenShift Data Foundation Operator using the Red Hat OpenShift Container Platform Operator Hub.

Prerequisites

  • Access to an OpenShift Container Platform cluster using an account with cluster-admin and Operator installation permissions.
  • For additional resource requirements, see the Planning your deployment guide.
Important

  • When you need to override the cluster-wide default node selector for OpenShift Data Foundation, you can use the following command in the command line interface to specify a blank node selector for the openshift-storage namespace (create openshift-storage namespace in this case): 

    $ oc annotate namespace openshift-storage openshift.io/node-selector=

Procedure

  1. Log in to the OpenShift Web Console.
  2. Click Operators → OperatorHub.
  3. Scroll or type OpenShift Data Foundation into the Filter by keyword box to find the OpenShift Data Foundation Operator.
  4. Click Install.

  5. Set the following options on the Install Operator page:

    1. Update Channel as stable-4.10.
    2. Installation Mode as A specific namespace on the cluster.
    3. Installed Namespace as Operator recommended namespace openshift-storage. If Namespace openshift-storage does not exist, it is created during the operator installation.

    4. Select Approval Strategy as Automatic or Manual.

      If you select Automatic updates, then the Operator Lifecycle Manager (OLM) automatically upgrades the running instance of your Operator without any intervention.

      If you select Manual updates, then the OLM creates an update request. As a cluster administrator, you must then manually approve that update request to update the Operator to a newer version.

    5. Ensure that the Enable option is selected for the Console plugin.
    6. Click Install.

Verification steps

  • After the operator is successfully installed, a pop-up with a message, Web console update is available appears on the user interface. Click Refresh web console from this pop-up for the console changes to reflect.

  • In the Web Console:

    • Navigate to Installed Operators and verify that the OpenShift Data Foundation Operator shows a green tick indicating successful installation.
    • Navigate to Storage and verify if Data Foundation dashboard is available.

3.2. Creating an OpenShift Data Foundation Cluster for external IBM FlashSystem storage

You need to create a new OpenShift Data Foundation cluster after you install the OpenShift Data Foundation operator on the OpenShift Container Platform.

Prerequisites

  • For Red Hat Enterprise Linux® operating system, ensure that there is iSCSI connectivity and then configure Linux multipath devices on the host.
  • For Red Hat Enterprise Linux CoreOS or when the packages are already installed, configure Linux multipath devices on the host.
  • Ensure to configure each worker with storage connectivity according to your storage system instructions. For the latest supported FlashSystem products and versions, see the Installing section within your Spectrum Virtualize family product documentation in IBM Documentation.

Procedure

  1. In the OpenShift Web Console, click Operators → Installed Operators to view all the installed operators.

    Ensure that the Project selected is openshift-storage.

  2. Click OpenShift Data Foundation and then click Create StorageSystem.

  3. In the Backing storage page, select the following options:

    1. Select Full deployment for the Deployment type option.
    2. Select Connect an external storage platform from the available options.
    3. Select IBM FlashSystem Storage from the Storage platform list.
    4. Click Next.

  4. In the Create storage class page, provide the following information:

    1. Enter a name for the storage class.

      When creating block storage persistent volumes, select the storage class <storage_class_name> for best performance. The storage class allows direct I/O path to the FlashSystem.

    2. Enter the followin g details of IBM FlashSystem connection:

      • IP address
      • User name
      • Password
      • Pool name
    3. Select thick or thin for the Volume mode.
    4. Click Next.

  5. In the Capacity and nodes page, provide the necessary details:

    1. Select a value for Requested capacity.

      The available options are 0.5 TiB, 2 TiB, and 4 TiB. The requested capacity is dynamically allocated on the infrastructure storage class.

    2. Select at least three nodes in three different zones.

      It is recommended to start with at least 14 CPUs and 34 GiB of RAM per node. If the nodes selected do not match the OpenShift Data Foundation cluster requirement of an aggregated 30 CPUs and 72 GiB of RAM, a minimal cluster will be deployed. For minimum starting node requirements, see the Resource requirements section in the Planning guide.

    3. Click Next.

  6. Optional: In the Security and network page, provide the necessary details:

    1. To enable encryption, select Enable data encryption for block and file storage.

    2. Choose any one or both Encryption level:

      • Cluster-wide encryption to encrypt the entire cluster (block and file).
      • StorageClass encryption to create encrypted persistent volume (block only) using encryption enabled storage class.

    3. Select the Connect to an external key management service checkbox. This is optional for cluster-wide encryption.

      1. Key Management Service Provider is set to Vault by default.
      2. Enter Vault Service Name, host Address of Vault server ('https://<hostname or ip>'), Port number, and Token.

    4. Expand Advanced Settings to enter additional settings and certificate details based on your Vault configuration:

      1. Enter the Key Value secret path in the Backend Path that is dedicated and unique to OpenShift Data Foundation.
      2. Optional: Enter TLS Server Name and Vault Enterprise Namespace.
      3. Provide CA Certificate, Client Certificate, and Client Private Key by uploading the respective PEM encoded certificate file.
    5. Click Save.

    6. Select Default (SDN) if you are using a single network or Custom (Multus) if you are using multiple network interfaces.

      1. Select a Public Network Interface from the dropdown.
      2. Select a Cluster Network Interface from the dropdown. NOTE: If you are using only one additional network interface, select the single NetworkAttachementDefinition, that is, ocs-public-cluster for the Public Network Interface, and leave the Cluster Network Interface blank.
    7. Click Next.

  7. In the Review and create page, review if all the details are correct:

    • To modify any configuration settings, click Back to go back to the previous configuration page.
  8. Click Create StorageSystem.

Verification Steps

Verifying the state of the pods
  1. Click Workloads → Pods from the left pane of the OpenShift Web Console.

  2. Select openshift-storage from the Project drop-down list.

    Note

    If the Show default projects option is disabled, use the toggle button to list all the default projects.

    Table 3.1. Pods corresponding to OpenShift Data Foundation components

    ComponentCorresponding pods

    OpenShift Data Foundation Operator

    • ocs-operator-* (1 pod on any worker node)
    • ocs-metrics-exporter-* (1 pod on any worker node)
    • odf-operator-controller-manager-* (1 pod on any worker node)
    • odf-console-* (1 pod on any worker node)
    • csi-addons-controller-manager-* (1 pod on any worker node)

    ibm-storage-odf-operator

    • ibm-storage-odf-operator-* (2 pods on any worker nodes)
    • ibm-odf-console-*

    ibm-flashsystem-storage

    ibm-flashsystem-storage-* (1 pod on any worker node)

    rook-ceph Operator

    rook-ceph-operator-* (1 pod on any worker node)

    Multicloud Object Gateway

    • noobaa-operator-* (1 pod on any worker node)
    • noobaa-core-* (1 pod on any worker node)
    • noobaa-db-pg-* (1 pod on any worker node)
    • noobaa-endpoint-* (1 pod on any worker node)

    CSI

    • ibm-block-csi-* (1 pod on any worker node)
Verifying that the OpenShift Data Foundation cluster is healthy
  1. In the Web Console, click Storage → Data Foundation.
  2. In the Status card of the Overview tab, verify that Storage System has a green tick mark.
  3. In the Details card, verify that the cluster information is displayed.

For more information on the health of OpenShift Data Foundation cluster using the Block and File dashboard, see Monitoring OpenShift Data Foundation.

Verfifying that the Multicloud Object Gateway is healthy
  1. In the Web Console, click Storage → Data Foundation.
  2. In the Status card of the Overview tab, click Storage System and then click the storage system link from the pop up.
  3. In the Status card of the Object tab, verify that both Object Service and Data Resiliency have a green tick.
  4. In the Details card, verify that the MCG information is displayed.

For more information on the health of OpenShift Data Foundation cluster using the object dashboard, see Monitoring OpenShift Data Foundation.

Verifying that IBM FlashSystem is connected and the storage cluster is ready
  • Run the following command to verify if the OpenShift Data Foundation cluster is connected to the external IBM FlashSystem. 
$ oc get flashsystemclusters.odf.ibm.com
NAME                     AGE   PHASE   CREATED AT
ibm-flashsystemcluster   35s           2021-09-23T07:44:52Z
Verifying the StorageSystem of the storage
  • Run the following command to verify the storageSystem of IBM FlashSystem storage cluster. 
$ oc get storagesystems.odf.openshift.io
NAME                                   STORAGE-SYSTEM-KIND                       STORAGE-SYSTEM-NAME
ibm-flashsystemcluster-storagesystem   flashsystemcluster.odf.ibm.com/v1alpha1   ibm-flashsystemcluster
ocs-storagecluster-storagesystem       storagecluster.ocs.openshift.io/v1        ocs-storagecluster
Verifying the subscription of the IBM operator
  • Run the following command to verify the subscription: 
$ oc get subscriptions.operators.coreos.com
NAME                                                                      PACKAGE                    SOURCE                CHANNEL
ibm-block-csi-operator-stable-certified-operators-openshift-marketplace   ibm-block-csi-operator     certified-operators   stable
ibm-storage-odf-operator                                                  ibm-storage-odf-operator   odf-catalogsource     stable-v1
noobaa-operator-alpha-odf-catalogsource-openshift-storage                 noobaa-operator            odf-catalogsource     alpha
ocs-operator-alpha-odf-catalogsource-openshift-storage                    ocs-operator               odf-catalogsource     alpha
odf-operator                                                              odf-operator               odf-catalogsource     alpha
Verifying the CSVs
  • Run the following command to verify that the CSVs are in the succeeded state. 
$ oc get csv
NAME                              DISPLAY                                     VERSION   REPLACES                        PHASE
ibm-block-csi-operator.v1.6.0     Operator for IBM block storage CSI driver   1.6.0     ibm-block-csi-operator.v1.5.0   Succeeded
ibm-storage-odf-operator.v0.2.1   IBM Storage ODF operator                    0.2.1                                     Installing
noobaa-operator.v5.9.0            NooBaa Operator                             5.9.0                                     Succeeded
ocs-operator.v4.10.0               OpenShift Container Storage                 4.10.0                                     Succeeded
odf-operator.v4.10.0               OpenShift Data Foundation                   4.10.0                                     Succeeded
Verifying the IBM operator and CSI pods
  • Run the following command to verify the IBM operator and CSI pods: 
$ oc get pods
NAME                                                              READY   STATUS              RESTARTS   AGE
5cb2b16ec2b11bf63dbe691d44a63535dc026bb5315d5075dc6c398b3c58l94   0/1     Completed           0          10m
7c806f6568f85cf10d72508261a2535c220429b54dbcf87349b9b4b9838fctg   0/1     Completed           0          8m47s
c4b05566c04876677a22d39fc9c02512401d0962109610e85c8fb900d3jd7k2   0/1     Completed           0          10m
c5d1376974666727b02bf25b3a4828241612186744ef417a668b4bc1759rzts   0/1     Completed           0          10m
ibm-block-csi-operator-7b656d6cc8-bqnwp                           1/1     Running             0          8m3s
ibm-odf-console-97cb7c84c-r52dq                                   0/1     ContainerCreating   0          8m4s
ibm-storage-odf-operator-57b8bc47df-mgkc7                         1/2     ImagePullBackOff    0          94s
noobaa-operator-7698579d56-x2zqs                                  1/1     Running             0          9m37s
ocs-metrics-exporter-94b57d764-zq2g2                              1/1     Running             0          9m32s
ocs-operator-5d96d778f6-vxlq5                                     1/1     Running             0          9m33s
odf-catalogsource-j7q72                                           1/1     Running             0          10m
odf-console-8987868cd-m7v29                                       1/1     Running             0          9m35s
odf-operator-controller-manager-5dbf785564-rwsgq                  2/2     Running             0          9m35s
rook-ceph-operator-68b4b976d8-dlc6w                               1/1     Running             0          9m32s