Chapter 10. Troubleshooting problems using log files

Log files contain messages about the system, including the kernel, services, and applications running on it. These contain information that helps troubleshoot issues or monitor system functions. The logging system in Red Hat Enterprise Linux is based on the built-in syslog protocol. Particular programs use this system to record events and organize them into log files, which are useful when auditing the operating system and troubleshooting various problems.

10.1. Services handling syslog messages

The following two services handle syslog messages:

  • The systemd-journald daemon
  • The Rsyslog service

The systemd-journald daemon collects messages from various sources and forwards them to Rsyslog for further processing. The systemd-journald daemon collects messages from the following sources:

  • Kernel
  • Early stages of the boot process
  • Standard and error output of daemons as they start up and run
  • Syslog

The Rsyslog service sorts the syslog messages by type and priority and writes them to the files in the /var/log directory. The /var/log directory persistently stores the log messages.

10.2. Subdirectories storing syslog messages

The following subdirectories under the /var/log directory store syslog messages.

  • /var/log/messages - all syslog messages except the following
  • /var/log/secure - security and authentication-related messages and errors
  • /var/log/maillog - mail server-related messages and errors
  • /var/log/cron - log files related to periodically executed tasks
  • /var/log/boot.log - log files related to system startup

10.3. Inspecting log files using the web console

Follow the steps in this procedure to inspect the log files using the web console.

Procedure

  1. Log into the Red Hat Enterprise Linux web console.
  2. Click Logs.

Figure 10.1. Inspecting the log files in the RHEL 8 web console

cs viewing logs web console

Additional resources

10.4. Viewing logs using the command line

The Journal is a component of systemd that helps to view and manage log files. It addresses problems connected with traditional logging, closely integrated with the rest of the system, and supports various logging technologies and access management for the log files.

You can use the journalctl command to view messages in the system journal using the command line, for example:

$ journalctl -b | grep kvm
May 15 11:31:41 localhost.localdomain kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00
May 15 11:31:41 localhost.localdomain kernel: kvm-clock: cpu 0, msr 76401001, primary cpu clock
...

Table 10.1. Viewing system information

CommandDescription

journalctl

Shows all collected journal entries.

journalctl FILEPATH

Shows logs related to a specific file. For example, the journalctl /dev/sda command displays logs related to the /dev/sda file system.

journalctl -b

Shows logs for the current boot.

journalctl -k -b -1

Shows kernel logs for the current boot.

Table 10.2. Viewing information on specific services

CommandDescription

journalctl -b _SYSTEMD_UNIT=foo

Filters log to see ones matching the "foo" systemd service.

journalctl -b _SYSTEMD_UNIT=foo _PID=number

Combines matches. For example, this command shows logs for systemd-units that match foo and the PID number.

journalctl -b _SYSTEMD_UNIT=foo _PID=number + _SYSTEMD_UNIT=foo1

The separator “+” combines two expressions in a logical OR. For example, this command shows all messages from the foo service process with the PID plus all messages from the foo1 service (from any of its processes).

journalctl -b _SYSTEMD_UNIT=foo _SYSTEMD_UNIT=foo1

This command shows all entries matching either expression, referring to the same field. Here, this command shows logs matching a systemd-unit foo or a systemd-unit foo1.

Table 10.3. Viewing logs related to specific boots

CommandDescription

journalctl --list-boots

Shows a tabular list of boot numbers, their IDs, and the timestamps of the first and last message pertaining to the boot. You can use the ID in the next command to view detailed information.

journalctl --boot=ID _SYSTEMD_UNIT=foo

Shows information about the specified boot ID.

10.5. Additional resources