RHEL7: auditing of a specific system call causes non-audited syscall code paths to contend on shared directory's dentry.d_lockref spinlock causing high CPU usage
Issue
- I've enabled auditing of certain system calls but I'm seeing audit code path related overhead for system calls that are not audited
- High CPU usage when audit is enabled on certain system calls and a shared directory is used by many users
Environment
- Red Hat Enterprise Linux 7
- seen on 3.10.0-514.el7
- kernel.org kernels up to at least 4.10.1 are believed to be affected as well
- Auditing enabled for on systemcall exit for at least one system call
# cat /etc/audit/rules.d/audit.rules
-D
-b 320
-a exit,always -F arch=b32 -S execve
-a exit,always -F arch=b64 -S execve
- Workload general description
- database / file workload application and is shared by thousands of people
- Workload specifics
- One system call that can log cwd audit records enabled for auditing (for example, execve)
- A large number of programs running in a shared working directory
- Those programs need to make a large number of other system calls that can also log cwd audit records but aren't enabled (e.g. open, access, stat, statfs - what is passed in does not matter and it should not matter if those system calls fail or work).
- One or more programs in the same shared working directory making a large number of calls to the system call that is being audited causing cwd entries to be created (in the reproduction example the programs were exec'ing themselves over and over in a loop in the shared directory).
- NOTE: Without step 4 the issue never happens. With a few copies of the program running doing the execs that was being audited the issue was immediately reproducible.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.