alertmanager-main-0 failed to run

Issue

• PV PVC is in bound state and created by vsphere dynamic volume provisioner
• Events show below error all the time

Unable to mount volumes for pod "alertmanager-main-0_openshift-monitoring(42b93943-asd)": timeout expired waiting for volumes to attach or mount for pod "openshift-monitoring"/"alertmanager-main-0". list of unmounted volumes=[alertmanager-main-db]. list of unattached volumes=[alertmanager-main-db config-volume secret-alertmanager-main-tls secret-alertmanager-main-proxy

• Pod gets stuck in container creating state

alertmanager-main-0                            0/3       ContainerCreating   0          2h        <none>           node12

• Node logs shows

Oct 08 16:46:46 node122 atomic-openshift-node[11166]: E1008 16:46:46.598652   11166 desired_state_of_world_populator.go:298] Error processing volume "alertmanager-main-db" for pod "alertmanager-main-0_openshift-monitoring(32760cbb-cs) error processing PVC "openshift-monitoring"/"alertmanager-main-db-alertmanager-main-0": can't start pod because PVC openshift-monitoring/alertmanager-main-db-alertmanager-main-0 is being deleted

controller logs:
E1008 18:41:41.359091       1 vsphere.go:892] Failed to detach disk: [asx_OpenShift] OpenShift_PS/alerts_1.vmdk for node: node12. err: +Invalid configuration for device '0'.
E1008 18:42:25.933820       1 vsphere.go:892] Failed to detach disk: [asx_ OpenShift_PS/alerts_2.vmdk for node: node13. err: +Invalid configuration for device '0'.