Volume provisioning failed with error datacenter not found in RHOCP4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.z
  • VMware vSphere

Issue

  • Failed to attach volume with the error datacenter 'XYZ' not found.

    $ oc get events -n openshift-monitoring
2m44s       Warning   FailedAttachVolume   pod/prometheus-k8s-0      AttachVolume.Attach failed for volume 
"pvc-xxxxx-xxxx-xxxxx-xxxx-xxxxxxxx" : datacenter 'XYZ' not found

Resolution

  • Check for the cloud-provider-config file as per the below command:

    $ oc get cm cloud-provider-config -o yaml -n openshift-config

  • Take the backup of config map cloud-provider-config, and change the DATACENTER name to correct datacenter name in the config map cloud-provider-config.

    $ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config.yaml

    $ vi cloud-provider-config.yaml
apiVersion: v1
data:
  config: '[Global]

secret-name = "vsphere-creds"
secret-namespace = "kube-system"
insecure-flag = "1"
[Workspace]
server = xxxxxxx
datacenter = XYZ  <----- update the correct datacenter name
default-datastore = xxxxxxxxxxx
folder = xxxxxxxxxxxxxx
[VirtualCenter "xxxxxxxxx"]
datacenters = XYZ  <----- update the correct datacenter name
'

$ oc apply -f cloud-provider-config.yaml

  • After that delete the notReady pods from the project, and new pods will be recreated.

    $ oc delete pod <pod-name>

Root Cause

Due to wrong configuration changes, volumes may fail to attach.

Diagnostic Steps

  • Unable to attach volume due to the below errors:

    $ oc get events -n openshift-monitoring
2m44s       Warning   FailedAttachVolume   pod/prometheus-k8s-0      AttachVolume.Attach failed for volume 
"pvc-xxxxx-xxxx-xxxxx-xxxx-xxxxxxxx" : datacenter 'XYZ' not found
3m18s       Warning   FailedMount          pod/alertmanager-main-0   MountVolume.WaitForAttach failed for 
volume "pvc-xxxxx-xxxx-xxxxx-xxxx-xxxxxxxx" : WaitForAttach failed for VMDK "[VxRail-Virtual-SAN-Datastore- 
xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx] 00000000-0000-0000-0000-000000000000/cibbbb-h4444-dynamic-pvc-xxxxx-xxxx- 
xxxxx-xxxx-xxxxxxxx.vmdk": devicePath is empty

  • Pod are in the pending state:

    $ oc get pods -n openshift-monitoring      
NAME                                          READY  STATUS   RESTARTS  AGE
alertmanager-main-0                           0/5    Pending  0         22d
prometheus-k8s-0                              0/7    Pending  0         22d
prometheus-k8s-1                              0/7    Pending  0         22d
  • Refer to KCS (Diagnostic steps) to check concerned to datacenter.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments