Error mounting a volume using the AWS EFS CSI Driver Operator on ROSA with STS

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Service on AWS (ROSA)
    • 4.10
  • AWS security token service (STS)

Issue

  • When installing the AWS EFS CSI Operator on ROSA cluster with STS enabled, the aws-efs-csi-driver-controller pods are not starting due to missing aws-credentials secret. It is still looking for IAM user credentials which are not available on an STS cluster since everything uses IAM Roles.

  • The events shows error messages similar to the following ones:

    MountVolume.SetUp failed for volume "aws-credentials" : secret "aws-efs-cloud-credentials" not found
    
    Warning  FailedMount  48s (x10 over 5m16s)  kubelet            MountVolume.SetUp failed for volume "pvc-XXXXXX" : rpc error: code = Internal desc = Could not mount "fs-XXXX:/" at "/var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount": mount failed: exit status 32
    Mounting command: mount
    Mounting arguments: -t efs -o accesspoint=fsap-xxxxx,tls fs-XXXX:/ /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount
    Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
    b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
    Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
    

Resolution

Note: The EFS CSI Driver is a TechPreview feature in OCP 4.9 and older as explained in Support for AWS EFS on OCP. It's GA starting with OCP 4.10.

This issue might be caused by not configuring the secret or a misconfiguration of IAM roles.

The configuration of AWS EFS CSI Driver Operator with Secure Token Service is already in the OpenShift documentation.

For ROSA, please refer to AWS EFS CSI Driver Operator installation guide (it includes the configuration steps for STS clusters).

Root Cause

The configuration for installing the AWS EFS CSI Driver Operator is already in the documentation.

Diagnostic Steps

After install the AWS EFS CSI operator on a ROSA cluster with STS enabled, check the aws-efs-csi-driver-controller pods, which are not starting due to missing aws-credentials secret of failing by a missing IAM permissions:

$ oc get pods -n openshift-cluster-csi-drivers
[...]

$ oc describe pod [aws-efs-csi-driver-controller_pod_name] -n openshift-cluster-csi-drivers
[...]
  Warning  FailedMount  2m7s (x21 over 30m)  kubelet            MountVolume.SetUp failed for volume "aws-credentials" : secret "aws-efs-cloud-credentials" not found
[...]

$ oc logs pod [aws-efs-csi-driver-controller_pod_name] -n openshift-cluster-csi-drivers
[...]
E0101 00:00:00.143225       1 mount_linux.go:184] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0cdcec49dfea966e0,tls fs-XXXX:/ /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
E0101 00:00:00.143420       1 driver.go:103] GRPC error: rpc error: code = Internal desc = Could not mount "fs-XXXX:/" at "/var/lib/kubelet/pods/XXXXXX/volumes/kubernetes.io~csi/pvc-xxxx/mount": mount failed: exit status 32
[...]

Check also the events of the failing pod:

$ oc describe pod [failing_pod_name] -n [namespace_name]
[...]
Warning  FailedMount  48s (x10 over 5m16s)  kubelet            MountVolume.SetUp failed for volume "pvc-XXXXXX" : rpc error: code = Internal desc = Could not mount "fs-XXXX:/" at "/var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0cdcec49dfea966e0,tls fs-XXXX:/ /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
[...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments