RHEL6.7: NFS client single NFSv4 READ of an executable file hangs on a read only mount, infinite READ loop with NFS4ERR_OLD_STATEID (10024)

Issue

multiple similarly configured Linux client hosts (under VMWare) with 2.6.32-573.8.1.el6.x86_64 kernel, and RHEL6-based NFS server (plus HPE-provided ServiceGuard). Customer runs both executables and scripts from an NFS-share (mounted read-only) and sometimes a process doing this becomes blocked and unkillable. The only way to fix the issue is to reboot. There are no errors on NFS-server reported at the moment when client hangs.

The problems started to appear randomly on all client hosts after upgrading the NFS server:
Both NFS server cluster nodes got updated on 12th of March 2016 from
2.6.32-431.5.1.el6.x86_64 to
2.6.32-573.1.1.el6.x86_64.

Service Guard was updated from
11.20.22 to
12.00.40.

The hangs occur about once per two weeks unpredictably, the customer has already seen them three or four times (different hosts, different files).

it is always executable files that lead to hang! The customer told us explicitly that it is always just a single file that is in consistently hung state until reboot.

We use a hardlink structure to ensure consistent multiple versions of our ERP system application binaries. Details
* The binary files of the application are modified on NFS server only while all the clients mount this nfs share read-only.
* A symbolic link on a RW nfs share points to the active version fs tree on the read only share.
* With each change we make a new “hardlink copy” of the application filesystem tree structure and just replace the files related to an update.
* The clients are updated by just changing the symbolic link to the new version.
* This ensures consistent versions and avoids unwanted modification of business software.