Volume attach failed on vSphere "Failed to add disk 'scsi0:3'"

Solution Verified - Updated -

Issue

PV is not getting attached to the pod after the pod shifting to a different node.
The following errors may appear in the events and logs.

  • Failed to add disk 'scsi0:3' error message is seen in the events and logs.
E1222 09:55:43.005315       1 operation_generator.go:246] BulkVerifyVolume.BulkVerifyVolumes Error checking volumes are attached with No VM found
I1222 09:55:50.193662       1 reconciler.go:287] attacherDetacher.AttachVolume started for volume "pvc-04780e3e-0c4d-11ea-b62a-005056a0b778" (UniqueName: "kubernetes.io/vsphere-volume/[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk") from node "compute-16-example.com" 
E1222 09:55:51.949238       1 virtualmachine.go:137] Failed to attach the disk with storagePolicy: "" on VM: "". err - Failed to add disk 'scsi0:3'.
E1222 09:55:51.949302       1 vsphere.go:821] Failed to attach disk: [xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk for node: compute-16-example.com. err: +Failed to add disk 'scsi0:3'.
E1222 09:55:51.949333       1 attacher.go:80] Error attaching volume "[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk" to node "compute-16-example.com": Failed to add disk 'scsi0:3'.
E1222 09:55:51.949661       1 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/vsphere-volume/[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk\"" failed. No retries permitted until 2020-12-22 09:57:53.949549525 +0100 CET m=+1576147.971897808 (durationBeforeRetry 2m2s). Error: "AttachVolume.Attach failed for volume \"pvc-04780e3e-0c4d-11ea-b62a-005056a0b778\" (UniqueName: \"kubernetes.io/vsphere-volume/[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk\") from node \"compute-16-example.com\" : Failed to add disk 'scsi0:3'."
I1222 09:55:51.951480       1 event.go:221] Event(v1.ObjectReference{Kind:"Pod", Namespace:"foobar", Name:"cassandra-1", UID:"ffb66123-4058-11eb-8e78-005056a0a8f0", APIVersion:"v1", ResourceVersion:"175761739", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' AttachVolume.Attach failed for volume "pvc-04780e3e-0c4d-11ea-b62a-005056a0b778" : Failed to add disk 'scsi0:3'.
  • SCSI Controller Limit of 4 has been reached, cannot create another SCSI controller error message is seen in the events and logs.
2021-05-25T07:38:48.724491344Z I0525 07:38:48.724476       1 actual_state_of_world.go:322] Volume "kubernetes.io/vsphere-volume/[xxx] kubevols/<pvc-disk-name>.vmdk" is already added to attachedVolume list to node "<node3-name>", update device path "/dev/disk/by-id/###"
2021-05-25T07:38:49.036191450Z W0525 07:38:49.036174       1 reconciler.go:356] Multi-Attach error for volume "<pvc-id>" (UniqueName: "kubernetes.io/vsphere-volume/[xxx] kubevols/<pvc-disk-name>.vmdk") from node "<node1-name>" Volume is already used by pods <project>/<pod-name> on node <node3-name>
...
2021-05-26T16:57:46.567161151Z I0526 16:57:46.567132       1 event.go:291] "Event occurred" object="<project>/<pod-name>" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"<pvc-name>\" : SCSI Controller Limit of 4 has been reached, cannot create another SCSI controller"

Environment

  • OpenShift Container Platform 3.x
  • OpenShift Container Platform 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content