Pods in OpenShift Cannot Mount EFS (NFS) File Systems and Mount Command Hangs
Issue
After updating the nfs-utils package on the nodes of an OpenShift cluster to version 1.3.0-0.48 or 1.3.0-0.48.el7_4, persistent volumes that use Amazon's Elastic File System (an implementation of NFS) get stuck while mounting.
When this issue occurs, you will find one or more mount commands are hanging as in the following ps -ef | grep nfs output.
root 6387 5903 0 13:36 ? 00:00:00 mount -t nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/1234/volumes/kubernetes.io~nfs/metrics-volume
root 6389 6387 0 13:36 ? 00:00:00 /sbin/mount.nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/1234/volumes/kubernetes.io~nfs/metrics-volume -o rw
root 6398 5903 0 13:36 ? 00:00:00 mount -t nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/5678/volumes/kubernetes.io~nfs/registry-volume
root 6400 6398 0 13:36 ? 00:00:00 /sbin/mount.nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/5678/volumes/kubernetes.io~nfs/registry-volume -o rw
The mount command appears to be hanging and not completing, rather than just taking a long time.
This behavior is more common right after a node reboots and multiple pods are starting up and trying to mount volume from EFS.
Environment
-
Red Hat Enterprise Linux 7.4
- nfs-utils-1.3.0-0.48
-
OpenShift Container Platform
- 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.