OCS/ODF Persistent Volume mount is failing with `permission denied` error

Solution Verified - Updated -

Issue

  • Error mounting CephFS volume in a pod, csi-cephfsplugin on the respective node reports following error.

    Warning  FailedMount  24s   kubelet, node1.example.com  MountVolume.MountDevice failed for volume "pvc-5a793162-33c1-4d0a-a0af-669048eed825" : stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5a793162-33c1-4d0a-a0af-669048eed825/globalmount: permission denied
    
  • On recent OCP versions, you can also see errors like this on the OCP worker node where that pod is trying to start:

    Jan 17 09:04:23 worker1 hyperkube[2450]: E0117 09:04:23.212120    2450 remote_runtime.go:228] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to resolve symlink \"/var/lib/kubelet/pods/4f03a733-e9d0-4124-8cd5-f02382eb235f/volumes/kubernetes.io~csi/pvc-23b3c1b0-631a-4e74-b488-8329a7eb9669/mount\": lstat /var/lib/kubelet/pods/4f03a733-e9d0-4124-8cd5-f02382eb235f/volumes/kubernetes.io~csi/pvc-23b3c1b0-631a-4e74-b488-8329a7eb9669/mount: permission denied" podSandboxID="a2e73fc0656c253dffa2839bd00b8781af5315ce5285f388de04feb0a5765fa3"
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.4 or higher
  • Red Hat OpenShift Container Storage (RHOCS) 4.x / Red Hat OpenShift Data Foundation (RHODF) 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content