Is there any system performance penalty to enable auditing ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5

Issue

  • Is there any system performance penalty to enable auditing
  • Is there any alternatives to audit to trace a killer, that have less impact on system performance to trace killer?

Resolution

  • In order to trace the event before happening, the only way is setting the audit rule.
  • If using 64 bit softwares in the system, and kill is the only syscall that is killing your process then the correct rule should be:
# /sbin/auditctl -a exit,always -F arch=b64 -S kill -S tkill -S tgkill -F a1=10 -k signal10
  • Adding a rule for both 32 bit and 64 bit is likely to add overhead without any benefit considering performance of the system.
  • If system is running both 32 bit and 64 bit compiled processes, add audit rule for both.
  • There is also performance impact to capture more rules for syscalls at once as below.
-a exit,always -F arch=b64 -S execve -S exit -S exit_group -S fork -S clone -F auid!=4294967295 -k MSexecve64
-a exit,always -F arch=b32 -S execve -S exit -S exit_group -S fork -S clone -F auid!=4294967295 -k MSexecve32

Root Cause

  • The auditd is the userspace component to the Linux Auditing System. It's responsible for writing audit records to the disk. while a number of audit rules are configured, high %sys of CPUs are utilised accordingly. As long as lots of syscalls are captured, lots of logs such as flooding are also written to the disk. If the disk is type of spinning HDD it takes longer to write it down, SSD is much faster.

Diagnostic Steps

  • audit performance impact test while running test script.
  - P:rhel 7.7 / Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz - SSD
    elapsed 0.6  user 0.4  sys 0.2  CPU 104.33 - no audit rule
    elapsed 4.4  user 0.5  sys 0.4  CPU 20.90  - audit rule

  - P:rhel 7.7  / Intel(R) Core(TM) i7-3740QM CPU @ 2.70GHz - Spinning disk, HDD
    elapsed 0.7  user 0.4  sys 0.2  CPU 100.65
    elapsed 39.5  user 0.5  sys 0.4  CPU 2.66 - audit rule

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments