System hang due to blocked tasks in fanotify code

Issue

Periodic system hang with multiple blocked task warnings on the console:

kernel: INFO: task fsnotify_mark:159 blocked for more than 120 seconds.
kernel: INFO: task tail:52498 blocked for more than 120 seconds.
kernel: INFO: task systemd:1 blocked for more than 120 seconds.

The system may partially or fully become unresponsive. Any userspace task may become blocked. More specific to this issue we commonly see systemd or a fsnotify_mark process stuck in D-state. Most of the blocked tasks have a stack trace similar to below. Note each contains some fsnotify or fanotify function.

PID: 57101  TASK: ffff8800897a3980  CPU: 0   COMMAND: "java"
 #0 [ffff880425c37ab8] __schedule at ffffffff8163b5bd
 #1 [ffff880425c37b20] schedule at ffffffff8163bc59
 #2 [ffff880425c37b30] fanotify_handle_event at ffffffff81226808
 #3 [ffff880425c37b98] fsnotify at ffffffff81222510
 #4 [ffff880425c37c88] security_file_open at ffffffff812868f6
 #5 [ffff880425c37ca8] do_dentry_open at ffffffff811dc61e
 #6 [ffff880425c37cf0] vfs_open at ffffffff811dc939
 #7 [ffff880425c37d00] do_last at ffffffff811ebf2d
 #8 [ffff880425c37db0] path_openat at ffffffff811eec92
 #9 [ffff880425c37e48] do_filp_open at ffffffff811f045b
#10 [ffff880425c37f18] do_sys_open at ffffffff811dde03
#11 [ffff880425c37f70] sys_open at ffffffff811ddf1e
#12 [ffff880425c37f80] system_call_fastpath at ffffffff81646c49
    RIP: 00007f76bdfea9cd  RSP: 00007f76990d8230  RFLAGS: 00010202
    RAX: 0000000000000002  RBX: ffffffff81646c49  RCX: 0000000000000001
    RDX: 00000000000001b6  RSI: 0000000000000042  RDI: 00007f76381f6730
    RBP: 00007f76990d8270   R8: 00007f76381f6730   R9: 0000000000000035
    R10: 0000000000000004  R11: 0000000000000293  R12: ffffffff811ddf1e
    R13: ffff880425c37f78  R14: 0000000000000042  R15: 00000000000001b6
    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b

PID: 1      TASK: ffff880428590000  CPU: 3   COMMAND: "systemd"
 #0 [ffff88042859bc70] __schedule at ffffffff8163b5bd
 #1 [ffff88042859bcd8] schedule at ffffffff8163bc59
 #2 [ffff88042859bce8] schedule_timeout at ffffffff81639949
 #3 [ffff88042859bd98] wait_for_completion at ffffffff8163c026
 #4 [ffff88042859bdf8] __synchronize_srcu at ffffffff810ab6c6
 #5 [ffff88042859be78] fsnotify_destroy_group at ffffffff8122345e
 #6 [ffff88042859be90] inotify_release at ffffffff812256e2
 #7 [ffff88042859bea8] __fput at ffffffff811e0949
 #8 [ffff88042859bef0] ____fput at ffffffff811e0c0e
 #9 [ffff88042859bf00] task_work_run at ffffffff810a2317
#10 [ffff88042859bf30] do_notify_resume at ffffffff81014b12
#11 [ffff88042859bf50] int_signal at ffffffff81646efd
    RIP: 00007f09a90e228d  RSP: 00007fff140709d0  RFLAGS: 00000293
    RAX: 0000000000000000  RBX: 000000000000001b  RCX: ffffffffffffffff
    RDX: 00007f09a90cd7b8  RSI: 0000000000000000  RDI: 000000000000001b
    RBP: 00007f09aa5cd710   R8: 0000000000000001   R9: 00007f09aa6d5945
    R10: 0000000000000000  R11: 0000000000000293  R12: 0000000000000000
    R13: 00007f09ac3dd870  R14: 00007f09ac53e680  R15: 0000000000000000
    ORIG_RAX: 0000000000000003  CS: 0033  SS: 002b