Unable to lock files on NFS filesystem on Red Hat Enterprise Linux

Issue

• While trying to write a lock file to an NFS filesystem it fails on RHEL 5.5 Server

• The NFS filesystem is sitting on a NetApp vfiler share. An application is trying to create the lock file when it starts. The lock file cannot be created. An strace against the application will show output similar to:

  open("/dev903/mylockfile", O_RDWR|O_CREAT, 0644) = 3
  fcntl(3, F_SETLKW, {type=F_WRLCK, whence=SEEK_SET, start=0, len=0}) = -1 ENOLCK (No locks available)
  

• While trying to lock a file on the NFS filesystem concurrently from different Java process on 2 different systems, both these processes hang.

• Lock operation is unusually slow.
• The NFS client syslog contains "lockd: server <nfs-server-ip> not responding, timed out".
• nfslock is not running on either NFS server or NFS client.