RHEL7: auditing of a specific system call causes non-audited syscall code paths to contend on shared directory's dentry.d_lockref spinlock causing high CPU usage

Solution In Progress - Updated -

Issue

  • I've enabled auditing of certain system calls but I'm seeing audit code path related overhead for system calls that are not audited
  • High CPU usage when audit is enabled on certain system calls and a shared directory is used by many users

Environment

  • Red Hat Enterprise Linux 7
    • seen on 3.10.0-514.el7
    • kernel.org kernels up to at least 4.10.1 are believed to be affected as well
  • Auditing enabled for on systemcall exit for at least one system call
# cat /etc/audit/rules.d/audit.rules 
-D
-b 320

-a exit,always -F arch=b32 -S execve
-a exit,always -F arch=b64 -S execve
  • Workload general description
    • database / file workload application and is shared by thousands of people
  • Workload specifics
  1. One system call that can log cwd audit records enabled for auditing (for example, execve)
  2. A large number of programs running in a shared working directory
  3. Those programs need to make a large number of other system calls that can also log cwd audit records but aren't enabled (e.g. open, access, stat, statfs - what is passed in does not matter and it should not matter if those system calls fail or work).
  4. One or more programs in the same shared working directory making a large number of calls to the system call that is being audited causing cwd entries to be created (in the reproduction example the programs were exec'ing themselves over and over in a loop in the shared directory).
    • NOTE: Without step 4 the issue never happens. With a few copies of the program running doing the execs that was being audited the issue was immediately reproducible.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content