azure-disk is failing to mount with error "special device does not exist" on OpenShift Container Platform 4
Issue
-
In the OpenShift cluster there is AMQ Streams running and since some days, some pods have problems mounting the volume correctly. The following events are reported related to the problem.
Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: Mounting command: mount Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1234567890 /var/lib/kubelet/pods/0c16d5dc-7c5a-4a22-98dd-72419e5af4d0/volumes/kubernetes.io~azure-disk/pvc-24999e90-858c-43b4-b275-1d2d922b2f91 Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: Output: mount: /var/lib/kubelet/pods/0c16d5dc-7c5a-4a22-98dd-72419e5af4d0/volumes/kubernetes.io~azure-disk/pvc-24999e90-858c-43b4-b275-1d2d922b2f91: special device /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1234567890 does not exist. Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: I1110 11:56:48.241468 2755673 azure_mounter.go:137] azureDisk - SetupAt:Mount disk:ab-cdefg-dynamic-pvc-24999e90-858c-43b4-b275-1d2d922b2f91 at dir:/var/lib/kubelet/pods/0c16d5dc-7c5a-4a22-98dd-72419e5af4d0/volumes/kubernetes.io~azure-disk/pvc-24999e90-858c-43b4-b275-1d2d922b2f91 failed during mounting with error:mount failed: exit status 32 -
With
statefulset, pod intermittently failing to mount volume on the node with following error:special device /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/some-id does not exist
Environment
- Red Hat OpenShift Container Platform (RHOCP) 4.8 to 4.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.