OCS/ODF Persistent Volume mount is failing with `permission denied` error
Issue
-
Error mounting CephFS volume in a pod,
csi-cephfspluginon the respective node reports following error.Warning FailedMount 24s kubelet, node1.example.com MountVolume.MountDevice failed for volume "pvc-5a793162-33c1-4d0a-a0af-669048eed825" : stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-5a793162-33c1-4d0a-a0af-669048eed825/globalmount: permission denied -
On recent OCP versions, you can also see errors like this on the OCP worker node where that pod is trying to start:
Jan 17 09:04:23 worker1 hyperkube[2450]: E0117 09:04:23.212120 2450 remote_runtime.go:228] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to resolve symlink \"/var/lib/kubelet/pods/4f03a733-e9d0-4124-8cd5-f02382eb235f/volumes/kubernetes.io~csi/pvc-23b3c1b0-631a-4e74-b488-8329a7eb9669/mount\": lstat /var/lib/kubelet/pods/4f03a733-e9d0-4124-8cd5-f02382eb235f/volumes/kubernetes.io~csi/pvc-23b3c1b0-631a-4e74-b488-8329a7eb9669/mount: permission denied" podSandboxID="a2e73fc0656c253dffa2839bd00b8781af5315ce5285f388de04feb0a5765fa3"
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4.4 or higher
- Red Hat OpenShift Container Storage (RHOCS) 4.x / Red Hat OpenShift Data Foundation (RHODF) 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.