Mounting NFS fails with mount system call failed error on RHEL 9.6

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 9.6
  • CoreOS based on 9.6 kernel
  • NFS 4.1 and NFS 4.2
  • NFS Server: nfs-ganesha below v4.0.5
  • VmWare vSAN Files Service prior to ESXi 8.0 P05/ ESXi 8.0 Update 3e with latest vSAN File Service Update
  • VmWare vSAN 7.x
  • NFS Server: AWS STORAGE GATEWAY (S3 Bucket) (EFS)

Issue

When using a RHEL 9.6 kernel, the following error is reported while trying to mount a previously working NFS share:

mount.nfs: mount system call failed

When using a RHEL 9.5 kernel or older everything works as expected.

Resolution

For NFSv4.1, upgrade the kernel to at least kernel-5.14.0-570.25.1.el9_6.

For NFSv4.2, the workaround is to upgrade the kernel to at least kernel-5.14.0-570.25.1.el9_6 and use NFSv4.0, NFSv4.1, or NFSv3 to mount the NFS Share. Or if NFSv4.2 is required then you will need to boot into the RHEL 9.5 kernel.

If you are using VmWare vSan 7.x or 8.x as NFS server please refer to the Broadcom VmWare knowledge base

For any other NFS server vendor, please open a ticket with your NFS server vendor.

Root Cause

NFSv4.1
This is a bug in RHEL9.6 because FATTR4_OPEN_ARGUMENTS is an NFSv4.2 recommended attribute, RHEL9.6 clients are incorrectly using this feature in NFSv4.1.

A Bug has been opened and recently fixed in kernel release 'kernel-5.14.0-570.25.1.el9_6'.

NFSv4.2
NFS Client running RHEL 9.6 sends FATTR4_OPEN_ARGUMENTS which the NFS server does not seem to recognize and instead of ignoring unknown flag it reply with NFS4ERR_BADXDR which means it could not parse the GETATTR request.

This was added as part of backport of 707f13b3d081 and by cb78f9b7d0c0c9

NFS version 4.2 defines Extensions for Opening and Delegating Files in NFSv4.2 and is described in rfc-9754 and by rfc-5661.

As defined in RFC8881 server must ignore unknown
attribute flags in GETATTR attribute and must not return value.

Diagnostic Steps

The mount attempts fails with -EIO which is translated into mount system call failed.

Running strace of the mount command:

[root@rhel9.6-uefi-0 ~]# strace -fvttTy -s 0 -e mount mount -t nfs  nfs-server:/mnt /mnt/
strace: Process 6271 attached
[pid  6271] 14:27:04.994200 mount("nfs-server:/mnt", "/mnt", ""..., 0, ""...) = -1 EIO (Input/output error) <0.051964>
[pid  6271] 14:27:05.051153 +++ exited with 0 +++
14:27:05.051498 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=6271, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
14:27:05.052750 +++ exited with 0 +++

and tcpdump analysis will reveal:

user@client:~/PWD$ tshark -r client.pcap -T fields -E header=y -e frame.number -e frame.time -e ip.src -e ip.dst -e _ws.col.Info -e nfs.attr  -Y "nfs && nfs.attr==86 || nfs.nfsstat4 == 10036"
frame.number    frame.time    ip.src    ip.dst    _ws.col.info    nfs.attr
112    May 04, 2025 11:06:44.388130000 CEST    172.168.0.22    172.168.0.23    V4 Call GETATTR FH: 0x6234d1b0    0,2,5,6,13,16,17,75,86
                    ^^ - FATTR4_OPEN_ARGUMENTS 
113    May 04, 2025 11:06:44.388302000 CEST    172.168.0.23    172.168.0.22    V4 Reply (Call In 112) GETATTR Status: NFS4ERR_BADXDR

Affected RHEL 9.6 kernels:

kernel-5.14.0-570.12.1.el9_6.x86_64
kernel-5.14.0-570.16.1.el9_6.x86_64
kernel-5.14.0-570.17.1.el9_6.x86_64

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments