RHEL7: Periodically open(..., O_WRONLY|O_CREAT|O_TRUNC, 0666) calls suffer from latency spikes and takes many seconds when a second process is actively listing a directory containing a larger number of files

Issue

• We have periodic slow (4 second and above) open(..., O_WRONLY|O_CREAT|O_TRUNC, 0666) calls over NFS.
• We have seen this in the wild on SAP prod systems but have reproduced this as follows.
  • An NFS directory contains tens or hundreds of thousands of files
  • One process is repeatedly listing the directory with 'ls -l'
  • A second process is doing repeated open(..., O_WRONLY|O_CREAT|O_TRUNC, 0666), read(), close() on separate files in the same directory
  • Sporadically, the 'open' system call of the second process takes many seconds
• Details of reproducer
  • process1: cd /mnt/nfs; while true; do ls -l > /dev/null; sleep 1; done &
  • process2: cd /mnt/nfs; strace -Tttvfs 1024 -o /tmp/nfs-open-test.strace ./nfs-open-test &

# cat nfs-open-test
#!/usr/bin/perl -w

open(my $urfh, "<", "/dev/urandom")
        or die "cannot open /dev/urandom: $!";

for(my $i = 0; $i < 700000; $i++) {
        open(my $wfh, ">", "file.$i");
        read($urfh, my $rnd_block, 4096);
        print $wfh  $rnd_block;
        close($wfh);
}