Error mounting a volume using the AWS EFS CSI Driver Operator on ROSA with STS
Environment
- Red Hat OpenShift Service on AWS (ROSA)
- 4.10
- AWS security token service (STS)
Issue
-
When installing the AWS EFS CSI Operator on ROSA cluster with STS enabled, the
aws-efs-csi-driver-controller
pods
are not starting due to missingaws-credentials
secret
. It is still looking for IAM user credentials which are not available on an STS cluster since everything uses IAM Roles. -
The events shows error messages similar to the following ones:
MountVolume.SetUp failed for volume "aws-credentials" : secret "aws-efs-cloud-credentials" not found
Warning FailedMount 48s (x10 over 5m16s) kubelet MountVolume.SetUp failed for volume "pvc-XXXXXX" : rpc error: code = Internal desc = Could not mount "fs-XXXX:/" at "/var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount": mount failed: exit status 32 Mounting command: mount Mounting arguments: -t efs -o accesspoint=fsap-xxxxx,tls fs-XXXX:/ /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri" b'mount.nfs4: access denied by server while mounting 127.0.0.1:/' Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
Resolution
Note: The EFS CSI Driver is a TechPreview feature in OCP 4.9 and older as explained in Support for AWS EFS on OCP. It's GA starting with OCP 4.10.
This issue might be caused by not configuring the secret
or a misconfiguration of IAM roles.
The configuration of AWS EFS CSI Driver Operator with Secure Token Service is already in the OpenShift documentation.
For ROSA, please refer to AWS EFS CSI Driver Operator installation guide (it includes the configuration steps for STS clusters).
Root Cause
The configuration for installing the AWS EFS CSI Driver Operator is already in the documentation.
Diagnostic Steps
After install the AWS EFS CSI operator on a ROSA cluster with STS enabled, check the aws-efs-csi-driver-controller
pods
, which are not starting due to missing aws-credentials
secret
of failing by a missing IAM permissions:
$ oc get pods -n openshift-cluster-csi-drivers
[...]
$ oc describe pod [aws-efs-csi-driver-controller_pod_name] -n openshift-cluster-csi-drivers
[...]
Warning FailedMount 2m7s (x21 over 30m) kubelet MountVolume.SetUp failed for volume "aws-credentials" : secret "aws-efs-cloud-credentials" not found
[...]
$ oc logs pod [aws-efs-csi-driver-controller_pod_name] -n openshift-cluster-csi-drivers
[...]
E0101 00:00:00.143225 1 mount_linux.go:184] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0cdcec49dfea966e0,tls fs-XXXX:/ /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
E0101 00:00:00.143420 1 driver.go:103] GRPC error: rpc error: code = Internal desc = Could not mount "fs-XXXX:/" at "/var/lib/kubelet/pods/XXXXXX/volumes/kubernetes.io~csi/pvc-xxxx/mount": mount failed: exit status 32
[...]
Check also the events of the failing pod:
$ oc describe pod [failing_pod_name] -n [namespace_name]
[...]
Warning FailedMount 48s (x10 over 5m16s) kubelet MountVolume.SetUp failed for volume "pvc-XXXXXX" : rpc error: code = Internal desc = Could not mount "fs-XXXX:/" at "/var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-0cdcec49dfea966e0,tls fs-XXXX:/ /var/lib/kubelet/pods/XXXXX/volumes/kubernetes.io~csi/pvc-XXXXXX/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'
Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True].
[...]
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments