Why is disk space not being freed up after deleting files from an NFS mount point?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL)
  • Third party NFS Server

Issue

  • Space not getting freed up even after deleting files from NFS mount point. df -h continues to show same disk usage as before and there is no PID holding the deleted file.
  • Space not getting freed up after deleting files from nfs mount share

Resolution

  • The df command will just process the values reported by NFS server vendor and display them. It is only relying the information, it has received. You would have to contact the NFS server vendor to know the actual volume usage from the NAS perspective.

Root Cause

  • In this scenario, storage level snapshot occupied the disk space, from the NAS end.

Diagnostic Steps

  • No PID holding the files from /nfs_mount_point.

    # lsof /nfs_mount_point | grep -i deleted

  • Check the usage in df command:

    # df -h /nfs_mount_point
Filesystem                         Size      Used         Avail        Use%        Mounted on
192.168.0.3:/share                12TB      11.17TB      849GB       93%       /nfs_mount_point

  • du shows only 6.4 Tb data in /nfs_mount_point

    # du -sxkh /nfs_mount_point/
6.4T    /nfs_mount_point/

  • Capture tcpdump while running df and analyze tcpdump.pcap to get the FSSTAT.

    $ tshark -n -tad -r tcpdump.pcap -Y 'nfs.procedure_v3 == FSSTAT'
93 2019-04-11 09:46:39 192.168.0.2 -> 192.168.0.3 NFS 208 V3 FSSTAT Call, FH: 0xd2a692d9
94 2019-04-11 09:46:39 192.168.0.3 -> 192.168.0.2 NFS 156 V3 FSSTAT Reply (Call In 93)

  • Check for the reply sent by NFS Server for the FSSTAT Call made by client and filtering it to get total size in bytes.

    $ tshark -n -tad -r tcpdump.pcap -Y 'frame.number == 94' -T fields -e nfs.fsstat3_resok.tbytes
13194139533312

  • Calculate total Size of the filesystem in TiB

    Size in TiB = 13194139533312/1024/1024/1024/1024 = 12 TiB

  • Free space in the filesystem in bytes as per response from nfs server.

    $ tshark -n -tad -r tcpdump.pcap -Y 'frame.number == 94' -T fields -e nfs.fsstat3_resok.fbytes
911985344512

  • Available size of the filesystem in GiB.

    911985344512 /1024/1024/1024 = 849.352539063 GiB

  • Calculate used percentage.

    used space = Total space - Free Space = 13194139533312 - 911985344512 = 1.228215419×10¹³ Bytes.

Converting used space into TiB = 1.228215419×10¹³/1024/1024/1024/1024 = 11.17 TiB

Used Percentage based on the bytes or data provided by the NFS Server:-

Used Space/Total Space * 100

11.17/12 * 100 = 93.08 %

  • This is the used space based on the data provided by the NFS Server on the basis of reply for FSSTAT call created by the NFS client for df. Hence it needs to be checked at nfs server end.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments