NFSv4.1 pNFS client nfs_file_direct_write() does not return even if the reply from the NFS server was received

Issue

KVM hosts hit a problem that qemu-kvm threads hang on waiting NFSv4.1 direct I/Os, even though it has received the reply from the server. We got vmcore, packet captures, and rpcdebug log.

The vmcore shows qemu-kvm:21135 does not return from a write I/O to fd:14 as follows:

  PID: 21135  TASK: ffff883f92afbec0  CPU: 14  COMMAND: "qemu-kvm"
   #0 [ffff883ff96efb10] __schedule at ffffffff8168c0a5
   #1 [ffff883ff96efb78] schedule at ffffffff8168c6f9
   #2 [ffff883ff96efb88] schedule_timeout at ffffffff8168a139
   #3 [ffff883ff96efc30] wait_for_completion_killable at ffffffff8168d616
   #4 [ffff883ff96efc98] nfs_file_direct_write at ffffffffa080aeb6 [nfs]
   #5 [ffff883ff96efd10] nfs_file_write at ffffffffa08001d9 [nfs]
   #6 [ffff883ff96efd58] do_sync_readv_writev at ffffffff811fe249
   #7 [ffff883ff96efe30] do_readv_writev at ffffffff811ff8ee
   #8 [ffff883ff96eff20] vfs_writev at ffffffff811ffb15
   #9 [ffff883ff96eff30] sys_pwritev at ffffffff811fff12
  #10 [ffff883ff96eff80] system_call_fastpath at ffffffff81697709
      RIP: 00007fca7dd1d7ab  RSP: 00007fc96e1f9c30  RFLAGS: 00000206
      RAX: 0000000000000128  RBX: ffffffff81697709  RCX: ffffffffffffffff
      RDX: 0000000000000002  RSI: 00007fca8b6be028  RDI: 000000000000000e
      RBP: 00007fca8b556f10   R8: 0000000000000000   R9: 000000000000528f
      R10: 00000000e1ad4000  R11: 0000000000000293  R12: 00007fca8b556f00
      R13: 00007fca8b556f38  R14: 00007fca8b556f98  R15: 00007fca8b5601e0
      ORIG_RAX: 0000000000000128  CS: 0033  SS: 002b

   FD       FILE            DENTRY           INODE       TYPE PATH
   14 ffff881feca7ed00 ffff883fd4e34f00 ffff883fd4de01c8 REG  /var/path

Other threads, mainly qemu-kvm, are waiting in inode_dio_wait() for i_dio_count becoming zero, that was incremented through nfs_file_direct_write() by qemu-kvm:21135.

  crash> struct inode.i_dio_count ffff883fd4de01c8
    i_dio_count = {
      counter = 1
    }
  crash>

This i_dio_count should have been decremented by the waker of qemu-kvm:21135 through nfs_direct_complete().