Pods in OpenShift Cannot Mount EFS (NFS) File Systems and Mount Command Hangs

Solution In Progress - Updated -

Issue

After updating the nfs-utils package on the nodes of an OpenShift cluster to version 1.3.0-0.48 or 1.3.0-0.48.el7_4, persistent volumes that use Amazon's Elastic File System (an implementation of NFS) get stuck while mounting.

When this issue occurs, you will find one or more mount commands are hanging as in the following ps -ef | grep nfs output.

root       6387   5903  0 13:36 ?        00:00:00 mount -t nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/1234/volumes/kubernetes.io~nfs/metrics-volume
root       6389   6387  0 13:36 ?        00:00:00 /sbin/mount.nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/1234/volumes/kubernetes.io~nfs/metrics-volume -o rw
root       6398   5903  0 13:36 ?        00:00:00 mount -t nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/5678/volumes/kubernetes.io~nfs/registry-volume
root       6400   6398  0 13:36 ?        00:00:00 /sbin/mount.nfs <hostname>:/ /var/lib/origin/openshift.local.volumes/pods/5678/volumes/kubernetes.io~nfs/registry-volume -o rw

The mount command appears to be hanging and not completing, rather than just taking a long time.

This behavior is more common right after a node reboots and multiple pods are starting up and trying to mount volume from EFS.

Environment

  • Red Hat Enterprise Linux 7.4

    • nfs-utils-1.3.0-0.48
  • OpenShift Container Platform

    • 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In