Why there are a lot of kernel: nfs_statfs: statfs error = 116 error message in my log?
Environment
Red Hat Enterprise Linux 4.6
Issue
There are a lot of nfs_statfs: statfs error = 116 error message in /var/log/message:
automount[6206]: lookup(file): key "paieas_vol1" not found in map.
automount[6206]: failed to mount /nfsc/paieas_vol1
kernel: nfs_statfs: statfs error = 116
Resolution
Possible reasons for getting this error are;
1. Network connectivity issues from NFS client to NFS server at this time. Check "ifconfig" command output for TX/RX errors or dropped packet count.
2. The file/directory is deleted manually or by any other application on the NFS server. Check if the file/directory still exist on the NFS server
3. Client side mount problems though it is rare. If the client use automount with short timeout period (default 10 minutes) then the "automountd" umount the share automatically when it gets timed out. Increase the timeout period ("--timeout=" option in the autofs configuration) to mitigate the issue.
#vi /etc/auto.master
.... --timeout 1200
4. NFS server is under heavy load and fails to respond to the NFS call. Check this from "top" output.
5. Heavy NFS requests and the running nfsd fails to respond to all the requests on time. Increase the number of running nfsd by modifying /etc/sysconfig/nfs and set RPCNFSDCOUNT=16 (default is 8)
#vi /etc/sysconfig/nfs
RPCNFSDCOUNT=16
Root Cause
when the application (or a shell script) on NFS client opens a file residing on a NFS mounted directory and the file becomes unavailable (for some reasons) so the application creates a pseudo file and reports stale NFS handle.
In the source code, the "nfs_statfs: statfs error = 116 "means the file which the nfs client wants to access in nfs server has been stale.
#define ESTALE 116 /* Stale NFS file handle */
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments