Chapter 8. Troubleshooting CephFS PVC creation in external mode

If you have updated the Red Hat Ceph Storage cluster to version 4.1.1 or later from a previous release and is not a freshly deployed cluster, you must manually set the application type for CephFS pool on the Red Hat Ceph Storage cluster to enable CephFS PVC creation in external mode.

  1. Check for CephFS pvc stuck in Pending status.

    $ oc get pvc

    Example output :

    NAME                      STATUS    VOLUME
    CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
    ngx-fs-pxknkcix20-pod     Pending
                              ocs-external-storagecluster-cephfs  28h
    [...]
  2. Check the describe output to see the events for respective pvc.

    Expected error message is cephfs_metadata/csi.volumes.default/csi.volume.pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: (1) Operation not permitted)

    # oc describe pvc ngx-fs-pxknkcix20-pod -n nginx-file

    Example output:

    Name:          ngx-fs-pxknkcix20-pod
    Namespace:     nginx-file
    StorageClass:  ocs-external-storagecluster-cephfs
    Status:        Pending
    Volume:
    Labels:        <none>
    Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
    Finalizers:    [kubernetes.io/pvc-protection]
    Capacity:
    Access Modes:
    VolumeMode:    Filesystem
    Mounted By:    ngx-fs-oyoe047v2bn2ka42jfgg-pod-hqhzf
    Events:
     Type     Reason              Age                   From                                                                                                                      Message
     ----     ------              ----                  ----                                                                                                                      -------
     Warning  ProvisioningFailed  107m (x245 over 22h)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-5f8b66cc96-hvcqp_6b7044af-c904-4795-9ce5-bf0cf63cc4a4
     (combined from similar events): failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Internal desc = error (an error (exit status 1) occurred while
     running rados args: [-m 192.168.13.212:6789,192.168.13.211:6789,192.168.13.213:6789 --id csi-cephfs-provisioner --keyfile=stripped -c /etc/ceph/ceph.conf -p cephfs_metadata getomapval
     csi.volumes.default csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47 /tmp/omap-get-186436239 --namespace=csi]) occurred, command output streams is ( error getting omap value
     cephfs_metadata/csi.volumes.default/csi.volume.pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47: (1) Operation not permitted)
  3. Check the settings for the <cephfs metadata pool name> (here cephfs_metadata ) and <cephfs data pool name> (here cephfs_data). For running the command, you will need jq preinstalled in the Red Hat Ceph Storage client node.

    # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
    {
      "cephfs": {}
    }
    "cephfs_metadata"
    {
       "cephfs": {}
    }
  4. Set the application type for CephFS pool.

    • Run the following commands on the Red Hat Ceph Storage client node :

      # ceph osd pool application set <cephfs metadata pool name> cephfs metadata cephfs
      # ceph osd pool application set <cephfs data pool name> cephfs data cephfs
  5. Verify if the settings are applied.

     # ceph osd pool ls detail --format=json | jq '.[] | select(.pool_name| startswith("cephfs")) | .pool_name, .application_metadata' "cephfs_data"
      {
        "cephfs": {
          "data": "cephfs"
       }
      }
      "cephfs_metadata"
      {
        "cephfs": {
          "metadata": "cephfs"
        }
      }
  6. Check the CephFS PVC status again. The PVC should now be in Bound state.

      # oc get pvc

    Example output :

    NAME                      STATUS    VOLUME
    CAPACITY  ACCESS MODES    STORAGECLASS                        AGE
    ngx-fs-pxknkcix20-pod     Bound     pvc-1ac0c6e6-9428-445d-bbd6-1284d54ddb47
    1Mi       RWO             ocs-external-storagecluster-cephfs  29h
    [...]