Azurefile-csi StorageClass Mount Error(13): Permission Denied

Solution In Progress - Updated -

Environment

  • Azure Red Hat OpenShift (ARO)
    • 4
  • Azure File CSI driver

Issue

  • Using the "azurefile-csi" storage class in an ARO cluster may encounter the following error:
MountVolume.MountDevice failed for volume "pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" : rpc error: code = Internal desc = volume(mrg-mp-its-aro-cus-935#clusterxxxxx#pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx#) mount "//clusterxxxxx.file.core.windows.net/pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx" on "/var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/6c5964b151f7118742c57facbe5bb12a3b1a0daee523cc1ef8897c719e5c6355/globalmount" failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t cifs -o mfsymlinks,cache=strict,nosharesock,actimeo=30,gid=10001,file_mode=0777,dir_mode=0777,<masked> //clusterxxxxx.file.core.windows.net/pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx /var/lib/kubelet/plugins/kubernetes.io/csi/file.csi.azure.com/6c5964b151f7118742c57facbe5bb12a3b1a0daee523cc1ef8897c719e5c6355/globalmount Output: mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Currently, there are two primary workarounds available to address this issue:

  1. Using In-tree Azure-File Storage Class (Not Recommended: Starting with Kubernetes 1.26, support for Azure Disk and Azure File in-tree drivers are discontinued.):
    • Follow the provided documentation to create an intree azure-file storage class.

Creating a Storage Class in Azure
Using Azure Files with Linux

  1. Creating a New CSI Storage Class with Extra Parameter Configurations:
    • Create a new storage class with the following parameters:
parameters:
  resourceGroup: [YOUR_RESOURCE_GROUP]
  storageAccount: [YOUR_STORAGE_ACCOUNT_NAME]

For Example:

$ oc get sc azurefile-csi-new -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azurefile-csi-new
mountOptions:
- mfsymlinks
- cache=strict
- nosharesock
- actimeo=30
parameters:
  resourceGroup: rg-judzhu
  skuName: Standard_LRS
  storageAccount: azurefilecsisa
provisioner: file.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Attention:

  1. Ensure to create a new storage account with proper Networking settings:

    • Enabled from all networks.
    • OR, Enabled from selected virtual networks and IP addresses with ARO Cluster subnets Including Master and Worker subnets.

  2. Provide the cluster Service Principal access to the Resource Group where the new storage account is located.
    e.g.

$ ARO_SERVICE_PRINCIPAL_ID=$(az aro show -g $ARO_RESOURCE_GROUP -n $CLUSTER --query servicePrincipalProfile.clientId -o tsv)

$ az role assignment create --role Contributor --scope /subscriptions/$SUBSCRIPTION_ID/resourceGroups/$ARO_RESOURCE_GROUP --assignee $ARO_SERVICE_PRINCIPAL_ID

Root Cause

The issue is suspected to be related to the storage lockdown feature and the behavior of the Azure File CSI operator. The Red Hat ARO team is actively investigating it.

Diagnostic Steps

To diagnose the mount permission error and verify the naming convention of the storage account, follow these steps:

  1. Check Mount Permission Error:

    • Review the pod's events and logs where the volume is being mounted. Look for any errors related to mounting or permissions.

    • Use the command:

      kubectl describe pod [POD_NAME] -n [NAMESPACE]

      Replace [POD_NAME] with the name of your pod and [NAMESPACE] with the namespace where the pod resides.

  2. Check Storage Account Azure File CSI Driver using

- Look for storage account names that start with clusterxxxxxfrom the error logs.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments