Unable to mount volumes from Azure Files on Azure Red Hat OpenShift clusters

Solution Verified - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4.x
  • Azure Files

Issue

  • Unable to mount storage from an Azure Storage Account.
  • I have created my storage class following the official Microsoft documentation, but my deployments are not able to mount the volumes.
  • Why my deployments are stuck and the pods cannot be deployed?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Root Cause

Although the procedures may be similar, please notice that the documents below must not be used to configure Azure Files on ARO clusters:

Diagnostic Steps

  1. Check for the affected pods:

    $ oc get pods | grep -vi running
NAME                                           READY   STATUS     RESTARTS   AGE
podname1-xxxxxxxxxx-xxxxx                      0/1     Init:0/1   0          7h
podname2-xxxxxxxxxx-xxxxx                      0/1     Init:0/1   0          3h

  2. Both Persistent Volume (PV) and Persistent Volume Claim (PVC) are presented as Bound:

    $ oc get pvc -n <namespace>
NAME            STATUS     VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS           AGE
<pvcname>       Bound      <volumename>          50Gi       RWX            <storageclassname>     10h

$ oc get pv
NAME           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS       REASON   AGE
<volumename>   50Gi       RWX            Retain           Bound    <namespace>/pvcname>  <storageclassname>          7h

  3. Looking at the pod logs, the errors below may be expected:

    $ oc logs <podname> -n <namespace>
...
<timestamp> | podname1-xxxxxxxxxx-xxxxx | FailedMount | <workerNodeName> | kubelet | Unable to attach or mount volumes: unmounted volumes=[<volumename>], unattached volumes=[<volumenames>]: timed out waiting for the condition
...
<timestamp> | podname1-xxxxxxxxxx-xxxxx | FailedMount | <workerNodeName> | kubelet | MountVolume.SetUp failed for volume "<volumename>" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 192.0.2.X:/tmp /var/lib/kubelet/pods/ff9945c0-ff9b-4688-a834-xxxxxxxxxxxx/volumes/kubernetes.io~nfs/<volumename>
Output: mount.nfs: Connection timed out
...
<timestamp> | podname1-xxxxxxxxxx-xxxxx | FailedMount | <workerNodeName> | kubelet | MountVolume.SetUp failed for volume "<volumename>" : rpc error: code = InvalidArgument desc = GetAccountInfo(csi-8142cd1890873e3f4f7d668c550beaxxxxxxxxxx9b837888141130518bee92a4) failed with error: could not get secret(<secretname>): secrets "<secretname>" is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:azure-file-csi-driver-node-sa" cannot get resource "secrets" in API group "" in the namespace "<namespace>"

  4. For any additional help, please contact our Support team. The information below may be required for further troubleshooting:

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments