[RHEL 8] Deadlock between auditd and kauditd when audit_backlog_limit has been reached

Issue

• System hangs with many tasks stuck waiting for an audit buffer with kernel stacks similar to:

   #0 [ffffb4a6983ffcb0] __schedule at ffffffffa554a1b4
   #1 [ffffb4a6983ffd48] schedule at ffffffffa554a628
   #2 [ffffb4a6983ffd58] schedule_timeout at ffffffffa554dcb3
   #3 [ffffb4a6983ffdf0] audit_log_start at ffffffffa4daade3
   #4 [ffffb4a6983ffe70] audit_log_exit at ffffffffa4db0542
   #5 [ffffb4a6983ffed8] __audit_syscall_exit at ffffffffa4db2d48
   #6 [ffffb4a6983fff10] syscall_slow_exit_work at ffffffffa4c038d1
   #7 [ffffb4a6983fff38] do_syscall_64 at ffffffffa4c04320
   #8 [ffffb4a6983fff50] entry_SYSCALL_64_after_hwframe at ffffffffa56000ad
  

• Message buffer shows messages like these preceding the hang:

  [834605.997585] audit: audit_backlog=16397 > audit_backlog_limit=16384
  [834605.997588] audit: audit_lost=1197277094 audit_rate_limit=10000 audit_backlog_limit=16384
  [834605.997643] audit: audit_backlog=16397 > audit_backlog_limit=16384
  

• In another time, system hangs with the following call traces of the task/tasks:

  #0 [ffffbe9943143c10] __schedule at ffffffff82fa0731
  #1 [ffffbe9943143ca0] schedule at ffffffff82fa0cc5
  #2 [ffffbe9943143cb0] schedule_preempt_disabled at ffffffff82fa0fea
  #3 [ffffbe9943143cb8] __mutex_lock at ffffffff82fa2c10
  #4 [ffffbe9943143d20] audit_receive at ffffffff827c2a75
  #5 [ffffbe9943143d40] netlink_unicast at ffffffff82e53ff6
  #6 [ffffbe9943143d80] netlink_sendmsg at ffffffff82e54294
  #7 [ffffbe9943143df0] sock_sendmsg at ffffffff82db753c
  ...