Timestamp out of order in /var/log/messages file

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7

Issue

Server crashed. When the messages files was viewed for events during that time, the timestamp is out of order.

Resolution

The hardware clock can be configured to synch with NTP. This process is documented in:

https://access.redhat.com/knowledge/solutions/2649

Root Cause

The system clock is owned by the Operating System, and the hardware clock is owned by CMOS/BIOS. While the Server is running and operational, it will use the the system clock as the clock resource. The system clock is configured to receive time updates via NTP. The hardware clock's time has drifted slightly and was not up to date.

Diagnostic Steps

Reviewing /var/log/messages reveals the following behaviour:

Apr 12 14:45:14 <SERVER> shutdown[15050]: shutting down for system reboot
Apr 12 14:45:15 <SERVER> init: Switching to runlevel: 6

Apr 12 14:45:26 <SERVER> kernel: SysRq : Resetting
Apr 12 13:49:48 <SERVER> syslogd 1.4.1: restart.
Apr 12 13:49:48 <SERVER> kernel: klogd 1.4.1, log source = /proc/kmsg started.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments