Server reboot due to Ansible play or Ad hoc command

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 7
  • Red Hat Enterprise Linux (RHEL) 8
  • Red Hat Enterprise Linux (RHEL) 9

Issue

  • Server rebooted unexpectedly
  • Seeing _raw_params=shutdown -r or patterns=['shutdown'] messages seen in the systemd journal or in the /var/log/messages prior to reboot.
  • Server reboots after patching is performed
  • Server reboots on the same hour every day/week/month

Resolution

  • Verify with your administration team if anyone manually issued a reboot of the server via Ansible, or if there are any scheduled Ansible plays that could had issued the reboot.

  • If Ansible is not involved in the reboot event, please engage with your Red Hat support representative for further analysis.

Root Cause

The _raw_params=shutdown -r or patterns=['shutdown'] messages may be logged in the systemd journal or in the /var/log/messages after a system reboot is issue via an Ansible play or an Ad-hoc command.

Diagnostic Steps

Using journalctl

If persistent storage is configured for log files produced by journald, you may use the journalctl and grep commands to search for the ansible _raw_params=shutdown -r or patterns=['shutdown'] messages recorded in the journal after the system has rebooted.

Example A:

# journalctl | grep 'patterns=..shutdown..'

Jan 22 09:15:57 rhel-host python3[1964]: ansible-ansible.legacy.find Invoked with   paths=['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/sbin'] patterns=['shutdown'] file_type=any read_whole_file=False age_stamp=mtime recurse=False hidden=False follow=False get_checksum=False use_regex=False excludes=None contains=None age=None size=None depth=None
Example B:

# journalctl | grep 'raw_params=shutdown -r'

Jan 22 09:21:14 rhel-host python3[1694]: ansible-ansible.legacy.command Invoked with _raw_params=shutdown -r _uses_shell=False warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None

NOTE: By default, log entries for the systemd journald service, do not persist across reboots. If persistent storage was not configured already, there will be no historical data for analysis inside the journal.

Using /var/log/messages

If persistent storage is not configured for the systemd journal, or if the reboot occured in a RHEL 6 system (which does not use systemd) you may then search for the Ansible shutdown messages in the /var/log/messages files.

Example A:

# grep 'patterns=..shutdown..' /var/log/messages

Jan 22 09:15:57 rhel-host python3[1964]: ansible-ansible.legacy.find Invoked with paths=['/sbin', '/bin', '/usr/sbin', '/usr/bin', '/usr/local/sbin'] patterns=['shutdown'] file_type=any read_whole_file=False age_stamp=mtime recurse=False hidden=False follow=False get_checksum=False use_regex=False excludes=None contains=None age=None size=None depth=None
Example B:

# grep 'raw_params=shutdown -r'` /var/log/messages

Jan 22 09:21:14 rhel-host python3[1694]: ansible-ansible.legacy.command Invoked with _raw_params=shutdown -r _uses_shell=False warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments