System hang due to blocked tasks in fanotify code

Solution Verified - Updated -

Issue

Periodic system hang with multiple blocked task warnings on the console:

kernel: INFO: task fsnotify_mark:159 blocked for more than 120 seconds.
kernel: INFO: task tail:52498 blocked for more than 120 seconds.
kernel: INFO: task systemd:1 blocked for more than 120 seconds.

The system may partially or fully become unresponsive. Any userspace task may become blocked. More specific to this issue we commonly see systemd or a fsnotify_mark process stuck in D-state. Most of the blocked tasks have a stack trace similar to below. Note each contains some fsnotify or fanotify function.

PID: 57101  TASK: ffff8800897a3980  CPU: 0   COMMAND: "java"
 #0 [ffff880425c37ab8] __schedule at ffffffff8163b5bd
 #1 [ffff880425c37b20] schedule at ffffffff8163bc59
 #2 [ffff880425c37b30] fanotify_handle_event at ffffffff81226808
 #3 [ffff880425c37b98] fsnotify at ffffffff81222510
 #4 [ffff880425c37c88] security_file_open at ffffffff812868f6
 #5 [ffff880425c37ca8] do_dentry_open at ffffffff811dc61e
 #6 [ffff880425c37cf0] vfs_open at ffffffff811dc939
 #7 [ffff880425c37d00] do_last at ffffffff811ebf2d
 #8 [ffff880425c37db0] path_openat at ffffffff811eec92
 #9 [ffff880425c37e48] do_filp_open at ffffffff811f045b
#10 [ffff880425c37f18] do_sys_open at ffffffff811dde03
#11 [ffff880425c37f70] sys_open at ffffffff811ddf1e
#12 [ffff880425c37f80] system_call_fastpath at ffffffff81646c49
    RIP: 00007f76bdfea9cd  RSP: 00007f76990d8230  RFLAGS: 00010202
    RAX: 0000000000000002  RBX: ffffffff81646c49  RCX: 0000000000000001
    RDX: 00000000000001b6  RSI: 0000000000000042  RDI: 00007f76381f6730
    RBP: 00007f76990d8270   R8: 00007f76381f6730   R9: 0000000000000035
    R10: 0000000000000004  R11: 0000000000000293  R12: ffffffff811ddf1e
    R13: ffff880425c37f78  R14: 0000000000000042  R15: 00000000000001b6
    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b

PID: 1      TASK: ffff880428590000  CPU: 3   COMMAND: "systemd"
 #0 [ffff88042859bc70] __schedule at ffffffff8163b5bd
 #1 [ffff88042859bcd8] schedule at ffffffff8163bc59
 #2 [ffff88042859bce8] schedule_timeout at ffffffff81639949
 #3 [ffff88042859bd98] wait_for_completion at ffffffff8163c026
 #4 [ffff88042859bdf8] __synchronize_srcu at ffffffff810ab6c6
 #5 [ffff88042859be78] fsnotify_destroy_group at ffffffff8122345e
 #6 [ffff88042859be90] inotify_release at ffffffff812256e2
 #7 [ffff88042859bea8] __fput at ffffffff811e0949
 #8 [ffff88042859bef0] ____fput at ffffffff811e0c0e
 #9 [ffff88042859bf00] task_work_run at ffffffff810a2317
#10 [ffff88042859bf30] do_notify_resume at ffffffff81014b12
#11 [ffff88042859bf50] int_signal at ffffffff81646efd
    RIP: 00007f09a90e228d  RSP: 00007fff140709d0  RFLAGS: 00000293
    RAX: 0000000000000000  RBX: 000000000000001b  RCX: ffffffffffffffff
    RDX: 00007f09a90cd7b8  RSI: 0000000000000000  RDI: 000000000000001b
    RBP: 00007f09aa5cd710   R8: 0000000000000001   R9: 00007f09aa6d5945
    R10: 0000000000000000  R11: 0000000000000293  R12: 0000000000000000
    R13: 00007f09ac3dd870  R14: 00007f09ac53e680  R15: 0000000000000000
    ORIG_RAX: 0000000000000003  CS: 0033  SS: 002b

Environment

  • Red Hat Enterprise Linux Server 7
  • Red Hat Enterprise Linux Server 8
  • Red Hat Enterprise Linux Server 9
  • Systems with software utilizing the fanotify facility including, but not limited to:
    • McAfee VSELinux
    • McAfee Endpoint Security for Linux
    • VMware's vsep thin client
    • TrendMicro's Deep Security Anti-malware engine
    • Microsoft Defender Advanced Threat Protection
    • Sophos Anti virus
    • CheckPoint Linux Agent
    • Clam AV.
    • Traps Agent

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content