Unreclaimable Slab Memory Leak with NFSv4 Mounts

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8.6

    • Reported on kernel-4.18.0-372.26.1.el8_6.x86_64

    • NFSv4 client with NFSv4.0 server

Issue

Unreclaimable slab memory growth is seen eventually causing an oom-killer.

Resolution

This bug is being investigated in a private Bugzilla.

Presently two workarounds exist, one NFS client-side and one NFS server-side. Note that both of these workarounds DO NOT need to be implemented, only one or the other:

Client Side Workaround: On the client side ensure that NFS mounts are being done explicitly with NFS v4.0 such as:

    mount -o vers=4.0 192.168.122.219:/root/nfs/export1 mount1

Server-side Workaround: On the server side edit the /etc/nfs.conf file to ensure higher versions of NFS v4 are being served by setting the specific versions of NFSv4 equal to y:

    [nfsd]
    # debug=0
    # threads=8
    # host=
    # port=0
    # grace-time=90
    # lease-time=90
    # tcp=y
     vers2=n
     vers3=n
     vers4=y    <<----
     vers4.0=y  <<----
     vers4.1=y  <<----
     vers4.2=y  <<----
    # rdma=n
    # rdma-port=20049

Root Cause

This issue occurs when the NFS server is serving NFSv4.0 and the client attempts to mount with only vers=4 specified as the mount option. The mount.nfs binary will attempt to negotiate with the highest version of NFS 4 by first trying to mount with v4.2, then v4.1, then v4.0 until it is successful.

Each time the mount is attempted the sunrpc module creates svc_rqst data structures in unreclaimable slab memory for the v4.2, v4.1, and v4.0 attempts, but the structures are not released until umounting. As a result the failed v4.2 and v4.1 attempts never release their svc_rqst structures because they are created even though the mount failed, and we have nothing to umount for the v4.2 and v4.1 attempts.

Diagnostic Steps

  1. Check for high unreclaimable slab memory reported in /proc/meminfo:

    # grep Slab -A2 /proc/meminfo
Slab:             167864 kB
SReclaimable:      99724 kB
SUnreclaim:       2568140 kB

  2. Check if NFS mounts are being done with the vers=4 option. Here we see the mount is being done with vers=4 and can use a verbose mount with -v to see the failing v4.2 and v4.1 negotiations:

    # mount -o vers=4 192.168.122.219:/root/nfs/export1 mount1 -v
mount.nfs: timeout set for Thu Mar 23 14:14:38 2023
mount.nfs: trying text-based options 
'vers=4.2,addr=192.168.122.219,clientaddr=192.168.122.35'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 
'vers=4,minorversion=1,addr=192.168.122.219,clientaddr=192.168.122.35'
mount.nfs: mount(2): Protocol not supported
mount.nfs: trying text-based options 
'vers=4,addr=192.168.122.219,clientaddr=192.168.122.35'
mount.nfs: mount(2): Device or resource busy

  3. Check what version of NFS is being served on the NFS server:

    # cat /etc/nfs.conf
[...cut...]
[nfsd]
# debug=0
# threads=8
# host=
# port=0
# grace-time=90
# lease-time=90
# udp=y
# tcp=y
vers2=n
vers3=n
vers4=y
vers4.0=y
vers4.1=n
vers4.2=n

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments