azure-disk is failing to mount with error "special device does not exist" on OpenShift Container Platform 4

Solution Verified - Updated -

Issue

  • In the OpenShift cluster there is AMQ Streams running and since some days, some pods have problems mounting the volume correctly. The following events are reported related to the problem.

    Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: Mounting command: mount
    Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: Mounting arguments:  -o bind /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1234567890 /var/lib/kubelet/pods/0c16d5dc-7c5a-4a22-98dd-72419e5af4d0/volumes/kubernetes.io~azure-disk/pvc-24999e90-858c-43b4-b275-1d2d922b2f91
    Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: Output: mount: /var/lib/kubelet/pods/0c16d5dc-7c5a-4a22-98dd-72419e5af4d0/volumes/kubernetes.io~azure-disk/pvc-24999e90-858c-43b4-b275-1d2d922b2f91: special device /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1234567890 does not exist.
    Nov 10 11:56:48 worker-1.example.com hyperkube[2755673]: I1110 11:56:48.241468 2755673 azure_mounter.go:137] azureDisk - SetupAt:Mount disk:ab-cdefg-dynamic-pvc-24999e90-858c-43b4-b275-1d2d922b2f91 at dir:/var/lib/kubelet/pods/0c16d5dc-7c5a-4a22-98dd-72419e5af4d0/volumes/kubernetes.io~azure-disk/pvc-24999e90-858c-43b4-b275-1d2d922b2f91 failed during mounting with error:mount failed: exit status 32
    
  • With statefulset, pod intermittently failing to mount volume on the node with following error:

    special device /var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/some-id does not exist
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4.8 to 4.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content