[RHEL 8] Deadlock between auditd and kauditd when audit_backlog_limit has been reached

Solution Verified - Updated -

Issue

  • System hangs with many tasks stuck waiting for an audit buffer with kernel stacks similar to:
 #0 [ffffb4a6983ffcb0] __schedule at ffffffffa554a1b4
 #1 [ffffb4a6983ffd48] schedule at ffffffffa554a628
 #2 [ffffb4a6983ffd58] schedule_timeout at ffffffffa554dcb3
 #3 [ffffb4a6983ffdf0] audit_log_start at ffffffffa4daade3
 #4 [ffffb4a6983ffe70] audit_log_exit at ffffffffa4db0542
 #5 [ffffb4a6983ffed8] __audit_syscall_exit at ffffffffa4db2d48
 #6 [ffffb4a6983fff10] syscall_slow_exit_work at ffffffffa4c038d1
 #7 [ffffb4a6983fff38] do_syscall_64 at ffffffffa4c04320
 #8 [ffffb4a6983fff50] entry_SYSCALL_64_after_hwframe at ffffffffa56000ad
  • Message buffer shows messages like these preceding the hang:
[834605.997585] audit: audit_backlog=16397 > audit_backlog_limit=16384
[834605.997588] audit: audit_lost=1197277094 audit_rate_limit=10000 audit_backlog_limit=16384
[834605.997643] audit: audit_backlog=16397 > audit_backlog_limit=16384
  • In another time, system hangs with the following call traces of the task/tasks
 #0 [ffffbe9943143c10] __schedule at ffffffff82fa0731
 #1 [ffffbe9943143ca0] schedule at ffffffff82fa0cc5
 #2 [ffffbe9943143cb0] schedule_preempt_disabled at ffffffff82fa0fea
 #3 [ffffbe9943143cb8] __mutex_lock at ffffffff82fa2c10
 #4 [ffffbe9943143d20] audit_receive at ffffffff827c2a75
 #5 [ffffbe9943143d40] netlink_unicast at ffffffff82e53ff6
 #6 [ffffbe9943143d80] netlink_sendmsg at ffffffff82e54294
 #7 [ffffbe9943143df0] sock_sendmsg at ffffffff82db753c
       ...

Environment

  • Red Hat Enterprise Linux 8.x
  • kernel 4.18.0-305.10.2.el8_4.x86_64
  • kernel 4.18.0-372.9.1.el8.x86_64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content