Multiple services are killed with SIGKILL (signal 9) at once on multiple servers

Issue

• Why multiple servers in production environment are killing all the services at same time with kill signal 9/KILL.

  Jul 26 21:07:25 localhost systemd[1]: sshd.service: Main process exited, code=killed, status=9/KILL
  Jul 26 21:07:25 localhost systemd[1]: smb.service: Main process exited, code=killed, status=9/KILL
  Jul 26 21:07:25 localhost systemd[1]: rpcbind.service: Main process exited, code=killed, status=9/KILL
  

  Jul 25 14:07:40 node1 systemd[1]: lvm2-lvmetad.service: main process exited, code=killed, status=9/KILL
  Jul 25 14:07:40 node1 systemd[1]: auditd.service: main process exited, code=killed, status=9/KILL
  Jul 25 14:07:40 node1 systemd[1]: systemd-udevd.service: main process exited, code=killed, status=9/KILL
  Jul 25 14:07:40 node1 systemd[1]: irqbalance.service: main process exited, code=killed, status=9/KILL
  Jul 25 14:07:40 node1 systemd[1]: systemd-logind.service: main process exited, code=killed, status=9/KILL
  Jul 25 14:07:40 node1 systemd[1]: systemd-journald.service: main process exited, code=killed, status=9/KILL
  

• How to find out what is sending kill signal SIGKILL/9 to multiple services including systemd on RHEL systems ?

• Multiple important services were killed and are in failed state across multiple servers including VM, physical host and running different RHEL versions and causing high impact on server availability.

• The systemctl --failed list includes services which failed to start after the suspected issue. Other services like systemd, systemd-journald, sshd were able to start and are running.

  # systemctl --failed 
   UNIT                  LOAD   ACTIVE SUB    DESCRIPTION
  ● acfssihamount.service loaded failed failed (null)
  ● atd.service           loaded failed failed Job spooling tools
  ● auditd.service        loaded failed failed Security Auditing Service
  ● chronyd.service       loaded failed failed NTP client/server
  ● firewalld.service     loaded failed failed firewalld - dynamic firewall daemon
  ● gssproxy.service      loaded failed failed GSSAPI Proxy Daemon
  ● irqbalance.service    loaded failed failed irqbalance daemon
  ● multipathd.service    loaded failed failed Device-Mapper Multipath Device Controller
  ● postfix.service       loaded failed failed Postfix Mail Transport Agent
  ● puppet.service        loaded failed failed Puppet agent
  ● rpc-gssd.service      loaded failed failed RPC security service for NFS client and server
  ● rpc-statd.service     loaded failed failed NFS status monitor for NFSv2/3 locking.
  ● rpcbind.service       loaded failed failed RPC bind service
  ● snmpd.service         loaded failed failed Simple Network Management Protocol (SNMP) Daemon.
  ● tuned.service         loaded failed failed Dynamic System Tuning Daemon