Chapter 4. Configuring persistent storage

Procedure

1. Copy an existing Azure MachineSet custom resource (CR) and edit it by running the following command:

   $ oc edit machineset <machine-set-name>

   where <machine-set-name> is the machine set that you want to provision machines with ultra disks.

2. Add the following lines in the positions indicated:

   apiVersion: machine.openshift.io/v1beta1
   kind: MachineSet
   spec:
     template:
       spec:
         metadata:
           labels:
             disk: ultrassd 1
         providerSpec:
           value:
             ultraSSDCapability: Enabled 2

   1

   Specify a label to use to select a node that is created by this machine set. This procedure uses disk.ultrassd for this value.

   2

   These lines enable the use of ultra disks.

3. Create a machine set using the updated configuration by running the following command:

   $ oc create -f <machine-set-name>.yaml

4. Create a storage class that contains the following YAML definition:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: ultra-disk-sc 1
   parameters:
     cachingMode: None
     diskIopsReadWrite: "2000" 2
     diskMbpsReadWrite: "320" 3
     kind: managed
     skuname: UltraSSD_LRS
   provisioner: disk.csi.azure.com 4
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer 5

   1

   Specify the name of the storage class. This procedure uses ultra-disk-sc for this value.

   2

   Specify the number of IOPS for the storage class.

   3

   Specify the throughput in MBps for the storage class.

   4

   For Azure Kubernetes Service (AKS) version 1.21 or later, use disk.csi.azure.com. For earlier versions of AKS, use kubernetes.io/azure-disk.

   5

   Optional: Specify this parameter to wait for the creation of the pod that will use the disk.

5. Create a persistent volume claim (PVC) to reference the ultra-disk-sc storage class that contains the following YAML definition:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: ultra-disk 1
   spec:
     accessModes:
     - ReadWriteOnce
     storageClassName: ultra-disk-sc 2
     resources:
       requests:
         storage: 4Gi 3

   1

   Specify the name of the PVC. This procedure uses ultra-disk for this value.

   2

   This PVC references the ultra-disk-sc storage class.

   3

   Specify the size for the storage class. The minimum value is 4Gi.

6. Create a pod that contains the following YAML definition:

   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx-ultra
   spec:
     nodeSelector:
       disk: ultrassd 1
     containers:
     - name: nginx-ultra
       image: alpine:latest
       command:
         - "sleep"
         - "infinity"
       volumeMounts:
       - mountPath: "/mnt/azure"
         name: volume
     volumes:
       - name: volume
         persistentVolumeClaim:
           claimName: ultra-disk 2

   1

   Specify the label of the machine set that enables the use of ultra disks. This procedure uses disk.ultrassd for this value.

   2

   This pod references the ultra-disk PVC.

Verification

1. Validate that the machines are created by running the following command:

   $ oc get machines

   The machines should be in the Running state.

2. For a machine that is running and has a node attached, validate the partition by running the following command:

   $ oc debug node/<node-name> -- chroot /host lsblk

   In this command, oc debug node/<node-name> starts a debugging shell on the node <node-name> and passes a command with --. The passed command chroot /host provides access to the underlying host OS binaries, and lsblk shows the block devices that are attached to the host OS machine.

Procedure

1. Save your object definition to a file.

   cinder-persistentvolume.yaml

   apiVersion: "v1"
   kind: "PersistentVolume"
   metadata:
     name: "pv0001" 1
   spec:
     capacity:
       storage: "5Gi" 2
     accessModes:
       - "ReadWriteOnce"
     cinder: 3
       fsType: "ext3" 4
       volumeID: "f37a03aa-6212-4c62-a805-9ce139fab180" 5

   1

   The name of the volume that is used by persistent volume claims or pods.

   2

   The amount of storage allocated to this volume.

   3

   Indicates cinder for Red Hat OpenStack Platform (RHOSP) Cinder volumes.

   4

   The file system that is created when the volume is mounted for the first time.

   5

   The Cinder volume to use.

   Important

   Do not change the fstype parameter value after the volume is formatted and provisioned. Changing this value can result in data loss and pod failure.

2. Create the object definition file you saved in the previous step.

   $ oc create -f cinder-persistentvolume.yaml

Procedure

1. Create a service account and add it to the SCC:

   $ oc create serviceaccount <service_account>

   $ oc adm policy add-scc-to-user <new_scc> -z <service_account> -n <project>

2. In your application’s deployment configuration, provide the service account name and securityContext:

   apiVersion: v1
   kind: ReplicationController
   metadata:
     name: frontend-1
   spec:
     replicas: 1  1
     selector:    2
       name: frontend
     template:    3
       metadata:
         labels:  4
           name: frontend 5
       spec:
         containers:
         - image: openshift/hello-openshift
           name: helloworld
           ports:
           - containerPort: 8080
             protocol: TCP
         restartPolicy: Always
         serviceAccountName: <service_account> 6
         securityContext:
           fsGroup: 7777 7

   1

   The number of copies of the pod to run.

   2

   The label selector of the pod to run.

   3

   A template for the pod that the controller creates.

   4

   The labels on the pod. They must include labels from the label selector.

   5

   The maximum name length after expanding any parameters is 63 characters.

   6

   Specifies the service account you created.

   7

   Specifies an fsGroup for the pods.

Procedure

1. In the OpenShift Container Platform console, click Storage → Persistent Volume Claims.
2. In the persistent volume claims overview, click Create Persistent Volume Claim.

3. Define the required options on the resulting page.

   1. Select the thin storage class.
   2. Enter a unique name for the storage claim.
   3. Select the access mode to determine the read and write access for the created storage claim.
   4. Define the size of the storage claim.
4. Click Create to create the persistent volume claim and generate a persistent volume.

Procedure (CLI)

1. You can define a VMware vSphere PersistentVolumeClaim by creating a file, pvc.yaml, with the following contents:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: pvc 1
   spec:
     accessModes:
     - ReadWriteOnce 2
     resources:
       requests:
         storage: 1Gi 3

   1

   A unique name that represents the persistent volume claim.

   2

   The access mode of the persistent volume claim. With ReadWriteOnce, the volume can be mounted with read and write permissions by a single node.

   3

   The size of the persistent volume claim.

2. Enter the following command to create the PersistentVolumeClaim object from the file:

   $ oc create -f pvc.yaml

Procedure

1. Create the openshift-local-storage project:

   $ oc adm new-project openshift-local-storage

2. Optional: Allow local storage creation on infrastructure nodes.

   You might want to use the Local Storage Operator to create volumes on infrastructure nodes in support of components such as logging and monitoring.

   You must adjust the default node selector so that the Local Storage Operator includes the infrastructure nodes, and not just worker nodes.

   To block the Local Storage Operator from inheriting the cluster-wide default selector, enter the following command:

   $ oc annotate namespace openshift-local-storage openshift.io/node-selector=''

3. Optional: Allow local storage to run on the management pool of CPUs in single-node deployment.

   Use the Local Storage Operator in single-node deployments and allow the use of CPUs that belong to the management pool. Perform this step on single-node installations that use management workload partitioning.

   To allow Local Storage Operator to run on the management CPU pool, run following commands:

   $ oc annotate namespace openshift-local-storage workload.openshift.io/allowed='management'

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → OperatorHub.
3. Type Local Storage into the filter box to locate the Local Storage Operator.
4. Click Install.
5. On the Install Operator page, select A specific namespace on the cluster. Select openshift-local-storage from the drop-down menu.
6. Adjust the values for Update Channel and Approval Strategy to the values that you want.
7. Click Install.

From the CLI

1. Install the Local Storage Operator from the CLI.

   1. Create an object YAML file to define an Operator group and subscription for the Local Storage Operator, such as openshift-local-storage.yaml:

      Example openshift-local-storage.yaml

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: local-operator-group
        namespace: openshift-local-storage
      spec:
        targetNamespaces:
          - openshift-local-storage
      ---
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: local-storage-operator
        namespace: openshift-local-storage
      spec:
        channel: stable
        installPlanApproval: Automatic 1
        name: local-storage-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

      1

      The user approval policy for an install plan.

2. Create the Local Storage Operator object by entering the following command:

   $ oc apply -f openshift-local-storage.yaml

   At this point, the Operator Lifecycle Manager (OLM) is now aware of the Local Storage Operator. A ClusterServiceVersion (CSV) for the Operator should appear in the target namespace, and APIs provided by the Operator should be available for creation.

3. Verify local storage installation by checking that all pods and the Local Storage Operator have been created:

   1. Check that all the required pods have been created:

      $ oc -n openshift-local-storage get pods

      Example output

      NAME                                      READY   STATUS    RESTARTS   AGE
      local-storage-operator-746bf599c9-vlt5t   1/1     Running   0          19m

   2. Check the ClusterServiceVersion (CSV) YAML manifest to see that the Local Storage Operator is available in the openshift-local-storage project:

      $ oc get csvs -n openshift-local-storage

      Example output

      NAME                                         DISPLAY         VERSION               REPLACES   PHASE
      local-storage-operator.4.2.26-202003230335   Local Storage   4.2.26-202003230335              Succeeded

Procedure

1. Create the local volume resource. This resource must define the nodes and paths to the local volumes.

   Note

   Do not use different storage class names for the same device. Doing so will create multiple persistent volumes (PVs).

   Example: Filesystem

   apiVersion: "local.storage.openshift.io/v1"
   kind: "LocalVolume"
   metadata:
     name: "local-disks"
     namespace: "openshift-local-storage" 1
   spec:
     nodeSelector: 2
       nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - ip-10-0-140-183
             - ip-10-0-158-139
             - ip-10-0-164-33
     storageClassDevices:
       - storageClassName: "local-sc" 3
         volumeMode: Filesystem 4
         fsType: xfs 5
         devicePaths: 6
           - /path/to/device 7

   1

   The namespace where the Local Storage Operator is installed.

   2

   Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.

   3

   The name of the storage class to use when creating persistent volume objects. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.

   4

   The volume mode, either Filesystem or Block, that defines the type of local volumes.

   Note

   A raw block volume (volumeMode: Block) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices.

   5

   The file system that is created when the local volume is mounted for the first time.

   6

   The path containing a list of local storage devices to choose from.

   7

   Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as /dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

   Note

   If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the virsh edit <VM> command to add the <serial>mydisk</serial> definition.

   Example: Block

   apiVersion: "local.storage.openshift.io/v1"
   kind: "LocalVolume"
   metadata:
     name: "local-disks"
     namespace: "openshift-local-storage" 1
   spec:
     nodeSelector: 2
       nodeSelectorTerms:
       - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - ip-10-0-136-143
             - ip-10-0-140-255
             - ip-10-0-144-180
     storageClassDevices:
       - storageClassName: "localblock-sc" 3
         volumeMode: Block 4
         devicePaths: 5
           - /path/to/device 6

   1

   The namespace where the Local Storage Operator is installed.

   2

   Optional: A node selector containing a list of nodes where the local storage volumes are attached. This example uses the node hostnames, obtained from oc get node. If a value is not defined, then the Local Storage Operator will attempt to find matching disks on all available nodes.

   3

   The name of the storage class to use when creating persistent volume objects.

   4

   The volume mode, either Filesystem or Block, that defines the type of local volumes.

   5

   The path containing a list of local storage devices to choose from.

   6

   Replace this value with your actual local disks filepath to the LocalVolume resource by-id, such as dev/disk/by-id/wwn. PVs are created for these local disks when the provisioner is deployed successfully.

   Note

   If you are running OpenShift Container Platform with RHEL KVM, you must assign a serial number to your VM disk. Otherwise, the VM disk can not be identified after reboot. You can use the virsh edit <VM> command to add the <serial>mydisk</serial> definition.

2. Create the local volume resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <local-volume>.yaml

3. Verify that the provisioner was created and that the corresponding daemon sets were created:

   $ oc get all -n openshift-local-storage

   Example output

   NAME                                          READY   STATUS    RESTARTS   AGE
   pod/diskmaker-manager-9wzms                   1/1     Running   0          5m43s
   pod/diskmaker-manager-jgvjp                   1/1     Running   0          5m43s
   pod/diskmaker-manager-tbdsj                   1/1     Running   0          5m43s
   pod/local-storage-operator-7db4bd9f79-t6k87   1/1     Running   0          14m
   
   NAME                                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
   service/local-storage-operator-metrics   ClusterIP   172.30.135.36   <none>        8383/TCP,8686/TCP   14m
   
   NAME                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
   daemonset.apps/diskmaker-manager   3         3         3       3            3           <none>          5m43s
   
   NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/local-storage-operator   1/1     1            1           14m
   
   NAME                                                DESIRED   CURRENT   READY   AGE
   replicaset.apps/local-storage-operator-7db4bd9f79   1         1         1       14m

   Note the desired and current number of daemon set processes. A desired count of 0 indicates that the label selectors were invalid.

4. Verify that the persistent volumes were created:

   $ oc get pv

   Example output

   NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
   local-pv-1cec77cf   100Gi      RWO            Delete           Available           local-sc                88m
   local-pv-2ef7cd2a   100Gi      RWO            Delete           Available           local-sc                82m
   local-pv-3fa1c73    100Gi      RWO            Delete           Available           local-sc                48m

Procedure

1. Define the PV. Create a file, such as example-pv-filesystem.yaml or example-pv-block.yaml, with the PersistentVolume object definition. This resource must define the nodes and paths to the local volumes.

   Note

   Do not use different storage class names for the same device. Doing so will create multiple PVs.

   example-pv-filesystem.yaml

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: example-pv-filesystem
   spec:
     capacity:
       storage: 100Gi
     volumeMode: Filesystem 1
     accessModes:
     - ReadWriteOnce
     persistentVolumeReclaimPolicy: Delete
     storageClassName: local-storage 2
     local:
       path: /dev/xvdf 3
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - example-node

   1

   The volume mode, either Filesystem or Block, that defines the type of PVs.

   2

   The name of the storage class to use when creating PV resources. Use a storage class that uniquely identifies this set of PVs.

   3

   The path containing a list of local storage devices to choose from, or a directory. You can only specify a directory with Filesystem volumeMode.

   Note

   A raw block volume (volumeMode: block) is not formatted with a file system. Use this mode only if any application running on the pod can use raw block devices.

   example-pv-block.yaml

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: example-pv-block
   spec:
     capacity:
       storage: 100Gi
     volumeMode: Block 1
     accessModes:
     - ReadWriteOnce
     persistentVolumeReclaimPolicy: Delete
     storageClassName: local-storage 2
     local:
       path: /dev/xvdf 3
     nodeAffinity:
       required:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - example-node

   1

   The volume mode, either Filesystem or Block, that defines the type of PVs.

   2

   The name of the storage class to use when creating PV resources. Be sure to use a storage class that uniquely identifies this set of PVs.

   3

   The path containing a list of local storage devices to choose from.

2. Create the PV resource in your OpenShift Container Platform cluster. Specify the file you just created:

   $ oc create -f <example-pv>.yaml

3. Verify that the local PV was created:

   $ oc get pv

   Example output

   NAME                    CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                STORAGECLASS    REASON   AGE
   example-pv-filesystem   100Gi      RWO            Delete           Available                        local-storage            3m47s
   example-pv1             1Gi        RWO            Delete           Bound       local-storage/pvc1   local-storage            12h
   example-pv2             1Gi        RWO            Delete           Bound       local-storage/pvc2   local-storage            12h
   example-pv3             1Gi        RWO            Delete           Bound       local-storage/pvc3   local-storage            12h

Procedure

1. Create the PVC using the corresponding storage class:

   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: local-pvc-name 1
   spec:
     accessModes:
     - ReadWriteOnce
     volumeMode: Filesystem 2
     resources:
       requests:
         storage: 100Gi 3
     storageClassName: local-sc 4

   1

   Name of the PVC.

   2

   The type of the PVC. Defaults to Filesystem.

   3

   The amount of storage available to the PVC.

   4

   Name of the storage class required by the claim.

2. Create the PVC in the OpenShift Container Platform cluster, specifying the file you just created:

   $ oc create -f <local-pvc>.yaml

Procedure

1. Include the defined claim in the resource spec. The following example declares the persistent volume claim inside a pod:

   apiVersion: v1
   kind: Pod
   spec:
   # ...
     containers:
       volumeMounts:
       - name: local-disks 1
         mountPath: /data 2
     volumes:
     - name: local-disks
       persistentVolumeClaim:
         claimName: local-pvc-name 3
   # ...

   1

   The name of the volume to mount.

   2

   The path inside the pod where the volume is mounted. Do not mount to the container root, /, or any path that is the same in the host and the container. This can corrupt your host system if the container is sufficiently privileged, such as the host /dev/pts files. It is safe to mount the host by using /host.

   3

   The name of the existing persistent volume claim to use.

2. Create the resource in the OpenShift Container Platform cluster, specifying the file you just created:

   $ oc create -f <local-pod>.yaml

Procedure

1. To enable automatic discovery of local devices from the web console:

   1. In the Administrator perspective, navigate to Operators → Installed Operators and click on the Local Volume Discovery tab.
   2. Click Create Local Volume Discovery.

   3. Select either All nodes or Select nodes, depending on whether you want to discover available disks on all or specific nodes.

      Note

      Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes.

   4. Click Create.

1. To display a continuous list of available devices on a node:

   1. Log in to the OpenShift Container Platform web console.
   2. Navigate to Compute → Nodes.
   3. Click the node name that you want to open. The "Node Details" page is displayed.

   4. Select the Disks tab to display the list of the selected devices.

      The device list updates continuously as local disks are added or removed. You can filter the devices by name, status, type, model, capacity, and mode.

2. To automatically provision local volumes for the discovered devices from the web console:

   1. Navigate to Operators → Installed Operators and select Local Storage from the list of Operators.
   2. Select Local Volume Set → Create Local Volume Set.
   3. Enter a volume set name and a storage class name.

   4. Choose All nodes or Select nodes to apply filters accordingly.

      Note

      Only worker nodes are available, regardless of whether you filter using All nodes or Select nodes.

   5. Select the disk type, mode, size, and limit you want to apply to the local volume set, and click Create.

      A message displays after several minutes, indicating that the "Operator reconciled successfully."

3. Alternatively, to provision local volumes for the discovered devices from the CLI:

   1. Create an object YAML file to define the local volume set, such as local-volume-set.yaml, as shown in the following example:

      apiVersion: local.storage.openshift.io/v1alpha1
      kind: LocalVolumeSet
      metadata:
        name: example-autodetect
      spec:
        nodeSelector:
          nodeSelectorTerms:
            - matchExpressions:
                - key: kubernetes.io/hostname
                  operator: In
                  values:
                    - worker-0
                    - worker-1
        storageClassName: example-storageclass 1
        volumeMode: Filesystem
        fsType: ext4
        maxDeviceCount: 10
        deviceInclusionSpec:
          deviceTypes: 2
            - disk
            - part
          deviceMechanicalProperties:
            - NonRotational
          minSize: 10G
          maxSize: 100G
          models:
            - SAMSUNG
            - Crucial_CT525MX3
          vendors:
            - ATA
            - ST2000LM

      1

      Determines the storage class that is created for persistent volumes that are provisioned from discovered devices. The Local Storage Operator automatically creates the storage class if it does not exist. Be sure to use a storage class that uniquely identifies this set of local volumes.

      2

      When using the local volume set feature, the Local Storage Operator does not support the use of logical volume management (LVM) devices.

   2. Create the local volume set object:

      $ oc apply -f local-volume-set.yaml

   3. Verify that the local persistent volumes were dynamically provisioned based on the storage class:

      $ oc get pv

      Example output

      NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS           REASON   AGE
      local-pv-1cec77cf   100Gi      RWO            Delete           Available           example-storageclass            88m
      local-pv-2ef7cd2a   100Gi      RWO            Delete           Available           example-storageclass            82m
      local-pv-3fa1c73    100Gi      RWO            Delete           Available           example-storageclass            48m

1. Modify the YAML file that defines the Pod and add the LocalVolume spec, as shown in the following example:

     apiVersion: "local.storage.openshift.io/v1"
     kind: "LocalVolume"
     metadata:
       name: "local-disks"
       namespace: "openshift-local-storage"
     spec:
       tolerations:
         - key: localstorage 1
           operator: Equal 2
           value: "localstorage" 3
       storageClassDevices:
           - storageClassName: "localblock-sc"
             volumeMode: Block 4
             devicePaths: 5
               - /dev/xvdg

   1

   Specify the key that you added to the node.

   2

   Specify the Equal operator to require the key/value parameters to match. If operator is Exists, the system checks that the key exists and ignores the value. If operator is Equal, then the key and value must match.

   3

   Specify the value local of the tainted node.

   4

   The volume mode, either Filesystem or Block, defining the type of the local volumes.

   5

   The path containing a list of local storage devices to choose from.

2. Optional: To create local persistent volumes on only tainted nodes, modify the YAML file and add the LocalVolume spec, as shown in the following example:

   spec:
     tolerations:
       - key: node-role.kubernetes.io/master
         operator: Exists

Procedure

1. Define the persistent volume (PV). Create a file, pv.yaml, with the PersistentVolume object definition:

     apiVersion: v1
     kind: PersistentVolume
     metadata:
       name: task-pv-volume 1
       labels:
         type: local
     spec:
       storageClassName: manual 2
       capacity:
         storage: 5Gi
       accessModes:
         - ReadWriteOnce 3
       persistentVolumeReclaimPolicy: Retain
       hostPath:
         path: "/mnt/data" 4

   1

   The name of the volume. This name is how it is identified by persistent volume claims or pods.

   2

   Used to bind persistent volume claim requests to this persistent volume.

   3

   The volume can be mounted as read-write by a single node.

   4

   The configuration file specifies that the volume is at /mnt/data on the cluster’s node. Do not mount to the container root, /, or any path that is the same in the host and the container. This can corrupt your host system. It is safe to mount the host by using /host.

2. Create the PV from the file:

   $ oc create -f pv.yaml

3. Define the persistent volume claim (PVC). Create a file, pvc.yaml, with the PersistentVolumeClaim object definition:

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: task-pvc-volume
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 1Gi
     storageClassName: manual

4. Create the PVC from the file:

   $ oc create -f pvc.yaml

1. Theoretical size.
2. Tested size.

Procedure

1. Create the LVMCluster custom resource (CR).

   Important

   You can only create a single instance of the LVMCluster custom resource (CR) on an OpenShift Container Platform cluster.

   1. Save the following YAML in the lvmcluster.yaml file:

      apiVersion: lvm.topolvm.io/v1alpha1
      kind: LVMCluster
      metadata:
        name: lvmcluster
      spec:
        storage:
          deviceClasses:  1
            - name: vg1
              default: true 2
              deviceSelector:
                paths:
                - /dev/disk/by-path/pci-0000:87:00.0-nvme-1
                - /dev/disk/by-path/pci-0000:88:00.0-nvme-1
              thinPoolConfig:
                name: thin-pool-1
                sizePercent: 90
                overprovisionRatio: 10
              nodeSelector: 3
                nodeSelectorTerms:
                  - matchExpressions:
                    - key: app
                      operator: In
                      values:
                      - test1

      1

      To create multiple device storage classes in the cluster, create a YAML array under deviceClasses for each required storage class. If you add or remove a deviceClass, then the update reflects in the cluster only after deleting and recreating the topolvm-node pod. Configure the local device paths of the disks as an array of values in the deviceSelector field. When configuring multiple device classes, you must specify the device path for each device.

      2

      Mandatory: The LVMCluster resource must contain a single default storage class. Set default: false for secondary device storage classes. If you are upgrading the LVMCluster resource from a previous version, you must specify a single default device class.

      3

      Optional: To control what worker nodes the LVMCluster CR is applied to, specify a set of node selector labels. The specified labels must be present on the node in order for the LVMCluster to be scheduled on that node.

   2. Create the LVMCluster CR:

      $ oc create -f lvmcluster.yaml

      Example output

      lvmcluster/lvmcluster created

      The LVMCluster resource creates the following system-managed CRs:

      LVMVolumeGroup

      Tracks individual volume groups across multiple nodes.

      LVMVolumeGroupNodeStatus

      Tracks the status of the volume groups on a node.

1. Check that the LVMCluster CR is in a ready state by running the following command:

   $ oc get lvmclusters.lvm.topolvm.io -o jsonpath='{.items[*].status.deviceClassStatuses[*]}'

   Example output

   {
       "name": "vg1",
       "nodeStatus": [
           {
               "devices": [
                   "/dev/nvme0n1",
                   "/dev/nvme1n1",
                   "/dev/nvme2n1"
               ],
               "node": "kube-node",
               "status": "Ready"
           }
       ]
   }

2. Check that the storage class is created:

   $ oc get storageclass

   Example output

   NAME          PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   lvms-vg1      topolvm.io           Delete          WaitForFirstConsumer   true                   31m

3. Check that the volume snapshot class is created:

   $ oc get volumesnapshotclass

   Example output

   NAME          DRIVER               DELETIONPOLICY   AGE
   lvms-vg1      topolvm.io           Delete           24h

4. Check that the LVMVolumeGroup resource is created:

   $ oc get lvmvolumegroup vg1 -o yaml

   Example output

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMVolumeGroup
   metadata:
     creationTimestamp: "2022-02-02T05:16:42Z"
     generation: 1
     name: vg1
     namespace: lvm-operator-system
     resourceVersion: "17242461"
     uid: 88e8ad7d-1544-41fb-9a8e-12b1a66ab157
   spec: {}

5. Check that the LVMVolumeGroupNodeStatus resource is created:

   $ oc get lvmvolumegroupnodestatuses.lvm.topolvm.io kube-node -o yaml

   Example output

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMVolumeGroupNodeStatus
   metadata:
     creationTimestamp: "2022-02-02T05:17:59Z"
     generation: 1
     name: kube-node
     namespace: lvm-operator-system
     resourceVersion: "17242882"
     uid: 292de9bb-3a9b-4ee8-946a-9b587986dafd
   spec:
     nodeStatus:
       - devices:
           - /dev/nvme0n1
           - /dev/nvme1n1
           - /dev/nvme2n1
         name: vg1
         status: Ready

Procedure

1. Identify the StorageClass that is created when LVM Storage is deployed.

   The StorageClass name is in the format, lvms-<device-class-name>. The device-class-name is the name of the device class that you provided in the LVMCluster of the Policy YAML. For example, if the deviceClass is called vg1, then the storageClass name is lvms-vg1.

   The volumeBindingMode of the storage class is set to WaitForFirstConsumer.

2. To create a PVC where the application requires storage, save the following YAML to a file with a name such as pvc.yaml.

   Example YAML to create a PVC

   # block pvc
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: lvm-block-1
     namespace: default
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Block
     resources:
       requests:
         storage: 10Gi
     storageClassName: lvms-vg1
   ---
   # file pvc
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: lvm-file-1
     namespace: default
   spec:
     accessModes:
       - ReadWriteOnce
     volumeMode: Filesystem
     resources:
       requests:
         storage: 10Gi
     storageClassName: lvms-vg1

3. Create the PVC by running the following command:

   # oc create -f pvc.yaml -ns <application_namespace>

   The created PVCs remain in pending state until you deploy the pods that use them.

Procedure

1. Update the Subscription resource for the LVM Storage Operator by running the following command:

   $ oc patch subscription lvms-operator -n openshift-storage --type merge --patch '{"spec":{"channel":"<update-channel>"}}' 1

   1

   Replace <update-channel> with the version of the LVM Storage Operator that you want to install, for example stable-4.13.

2. View the upgrade events to check that the installation is complete by running the following command:

   $ oc get events -n openshift-storage

   Example output

   ...
   8m13s       Normal    RequirementsUnknown   clusterserviceversion/lvms-operator.v4.13   requirements not yet checked
   8m11s       Normal    RequirementsNotMet    clusterserviceversion/lvms-operator.v4.13   one or more requirements couldn't be found
   7m50s       Normal    AllRequirementsMet    clusterserviceversion/lvms-operator.v4.13   all requirements found, attempting install
   7m50s       Normal    InstallSucceeded      clusterserviceversion/lvms-operator.v4.13   waiting for install components to report healthy
   7m49s       Normal    InstallWaiting        clusterserviceversion/lvms-operator.v4.13   installing: waiting for deployment lvms-operator to become ready: deployment "lvms-operator" waiting for 1 outdated replica(s) to be terminated
   7m39s       Normal    InstallSucceeded      clusterserviceversion/lvms-operator.v4.13   install strategy completed with no errors
   ...

Procedure

1. Retrieve the list of PVCs by running the following command:

   $ oc get pvc

   Example output

   NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   lvms-test   Pending                                      lvms-vg1       11s

2. Inspect the events associated with a PVC stuck in the Pending state by running the following command:

   $ oc describe pvc <pvc_name> 1

   1

   Replace <pvc_name> with the name of the PVC. For example, lvms-vg1.

   Example output

   Type     Reason              Age               From                         Message
   ----     ------              ----              ----                         -------
   Warning  ProvisioningFailed  4s (x2 over 17s)  persistentvolume-controller  storageclass.storage.k8s.io "lvms-vg1" not found

Procedure

1. Verify the presence of the LVMCluster resource by running the following command:

   $ oc get lvmcluster -n openshift-storage

   Example output

   NAME            AGE
   my-lvmcluster   65m

2. If the cluster doesn’t have an LVMCluster resource, create one by running the following command:

   $ oc create -n openshift-storage -f <custom_resource> 1

   1

   Replace <custom_resource> with a custom resource URL or file tailored to your requirements.

   Example custom resource

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMCluster
   metadata:
     name: my-lvmcluster
   spec:
     storage:
       deviceClasses:
       - name: vg1
         default: true
         thinPoolConfig:
           name: thin-pool-1
           sizePercent: 90
           overprovisionRatio: 10

3. Check that all the pods from LVMS are in the Running state in the openshift-storage namespace by running the following command:

   $ oc get pods -n openshift-storage

   Example output

   NAME                                  READY   STATUS    RESTARTS      AGE
   lvms-operator-7b9fb858cb-6nsml        3/3     Running   0             70m
   topolvm-controller-5dd9cf78b5-7wwr2   5/5     Running   0             66m
   topolvm-node-dr26h                    4/4     Running   0             66m
   vg-manager-r6zdv                      1/1     Running   0             66m

   The expected output is one running instance of lvms-operator and vg-manager. One instance of topolvm-controller and topolvm-node is expected for each node.

   If topolvm-node is stuck in the Init state, there is a failure to locate an available disk for LVMS to use. To retrieve the information necessary to troubleshoot, review the logs of the vg-manager pod by running the following command:

   $ oc logs -l app.kubernetes.io/component=vg-manager -n openshift-storage

Procedure

1. Inspect the events associated with a PVC by running the following command:

   $ oc describe pvc <pvc_name> 1

   1

   Replace <pvc_name> with the name of the PVC. Here are some examples of disk or volume failure error messages and their causes:

   • Failed to check volume existence: Indicates a problem in verifying whether the volume already exists. Volume verification failure can be caused by network connectivity problems or other failures.
   • Failed to bind volume: Failure to bind a volume can happen if the persistent volume (PV) that is available does not match the requirements of the PVC.
   • FailedMount or FailedUnMount: This error indicates problems when trying to mount the volume to a node or unmount a volume from a node. If the disk has failed, this error might appear when a pod tries to use the PVC.
   • Volume is already exclusively attached to one node and can’t be attached to another: This error can appear with storage solutions that do not support ReadWriteMany access modes.

2. Establish a direct connection to the host where the problem is occurring.
3. Resolve the disk issue.

Prerequisites

1. All of the persistent volume claims (PVCs) created using the logical volume manager storage (LVMS) driver have been removed.
2. The pods using those PVCs have been stopped.

Procedure

1. Switch to the openshift-storage namespace by running the following command:

   $ oc project openshift-storage

2. Ensure there is no Logical Volume custom resource (CR) remaining by running the following command:

   $ oc get logicalvolume

   Example output

   No resources found

   1. If there are any LogicalVolume CRs remaining, remove their finalizers by running the following command:

      $ oc patch logicalvolume <name> -p '{"metadata":{"finalizers":[]}}' --type=merge 1

      1

      Replace <name> with the name of the CR.

   2. After removing their finalizers, delete the CRs by running the following command:

      $ oc delete logicalvolume <name> 1

      1

      Replace <name> with the name of the CR.

3. Make sure there are no LVMVolumeGroup CRs left by running the following command:

   $ oc get lvmvolumegroup

   Example output

   No resources found

   1. If there are any LVMVolumeGroup CRs left, remove their finalizers by running the following command:

      $ oc patch lvmvolumegroup <name> -p '{"metadata":{"finalizers":[]}}' --type=merge 1

      1

      Replace <name> with the name of the CR.

   2. After removing their finalizers, delete the CRs by running the following command:

      $ oc delete lvmvolumegroup <name> 1

      1

      Replace <name> with the name of the CR.

4. Remove any LVMVolumeGroupNodeStatus CRs by running the following command:

   $ oc delete lvmvolumegroupnodestatus --all

5. Remove the LVMCluster CR by running the following command:

   $ oc delete lvmcluster --all