Volume attach failed on vSphere "Failed to add disk 'scsi0:3'"

Issue

A PersistentVolume is not getting attached to the pod after the pod shifting to a different node.
The following errors may appear in the events and logs.

• Failed to add disk 'scsi0:3' error message is seen in the events and logs.

E1222 09:55:43.005315       1 operation_generator.go:246] BulkVerifyVolume.BulkVerifyVolumes Error checking volumes are attached with No VM found
I1222 09:55:50.193662       1 reconciler.go:287] attacherDetacher.AttachVolume started for volume "pvc-04780e3e-0c4d-11ea-b62a-005056a0b778" (UniqueName: "kubernetes.io/vsphere-volume/[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk") from node "compute-16-example.com" 
E1222 09:55:51.949238       1 virtualmachine.go:137] Failed to attach the disk with storagePolicy: "" on VM: "". err - Failed to add disk 'scsi0:3'.
E1222 09:55:51.949302       1 vsphere.go:821] Failed to attach disk: [xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk for node: compute-16-example.com. err: +Failed to add disk 'scsi0:3'.
E1222 09:55:51.949333       1 attacher.go:80] Error attaching volume "[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk" to node "compute-16-example.com": Failed to add disk 'scsi0:3'.
E1222 09:55:51.949661       1 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/vsphere-volume/[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk\"" failed. No retries permitted until 2020-12-22 09:57:53.949549525 +0100 CET m=+1576147.971897808 (durationBeforeRetry 2m2s). Error: "AttachVolume.Attach failed for volume \"pvc-04780e3e-0c4d-11ea-b62a-005056a0b778\" (UniqueName: \"kubernetes.io/vsphere-volume/[xxxx] 779e9b5d-8a5b-0a41-dc72-5480285758f8/kubernetes-dynamic-pvc-04780e3e-0c4d-11ea-b62a-005056a0b778.vmdk\") from node \"compute-16-example.com\" : Failed to add disk 'scsi0:3'."
I1222 09:55:51.951480       1 event.go:221] Event(v1.ObjectReference{Kind:"Pod", Namespace:"foobar", Name:"cassandra-1", UID:"ffb66123-4058-11eb-8e78-005056a0a8f0", APIVersion:"v1", ResourceVersion:"175761739", FieldPath:""}): type: 'Warning' reason: 'FailedAttachVolume' AttachVolume.Attach failed for volume "pvc-04780e3e-0c4d-11ea-b62a-005056a0b778" : Failed to add disk 'scsi0:3'.

• SCSI Controller Limit of 4 has been reached, cannot create another SCSI controller error message is seen in the events and logs.

2021-05-25T07:38:48.724491344Z I0525 07:38:48.724476       1 actual_state_of_world.go:322] Volume "kubernetes.io/vsphere-volume/[xxx] kubevols/<pvc-disk-name>.vmdk" is already added to attachedVolume list to node "<node3-name>", update device path "/dev/disk/by-id/###"
2021-05-25T07:38:49.036191450Z W0525 07:38:49.036174       1 reconciler.go:356] Multi-Attach error for volume "<pvc-id>" (UniqueName: "kubernetes.io/vsphere-volume/[xxx] kubevols/<pvc-disk-name>.vmdk") from node "<node1-name>" Volume is already used by pods <project>/<pod-name> on node <node3-name>
...
2021-05-26T16:57:46.567161151Z I0526 16:57:46.567132       1 event.go:291] "Event occurred" object="<project>/<pod-name>" kind="Pod" apiVersion="v1" type="Warning" reason="FailedAttachVolume" message="AttachVolume.Attach failed for volume \"<pvc-name>\" : SCSI Controller Limit of 4 has been reached, cannot create another SCSI controller"