Chapter 8. Troubleshooting problems by using log files

Log files contain messages about the system, including the kernel, services, and applications running on it. These contain information that helps troubleshoot issues or monitor system functions. The logging system in Red Hat Enterprise Linux is based on the built-in syslog protocol. Particular programs use this system to record events and organize them into log files, which are useful when auditing the operating system and troubleshooting various problems.

8.1. Services handling syslog messages

The following two services handle syslog messages:

  • The systemd-journald daemon
  • The Rsyslog service

The systemd-journald daemon collects messages from various sources and forwards them to Rsyslog for further processing. The systemd-journald daemon collects messages from the following sources:

  • Kernel
  • Early stages of the boot process
  • Standard and error output of daemons as they start up and run
  • Syslog

The Rsyslog service sorts the syslog messages by type and priority and writes them to the files in the /var/log directory. The /var/log directory persistently stores the log messages.

8.2. Subdirectories storing syslog messages

The following subdirectories under the /var/log directory store syslog messages.

  • /var/log/messages - all syslog messages except the following
  • /var/log/secure - security and authentication-related messages and errors
  • /var/log/maillog - mail server-related messages and errors
  • /var/log/cron - log files related to periodically executed tasks
  • /var/log/boot.log - log files related to system startup

8.3. Inspecting log files using the web console

Follow the steps in this procedure to inspect the log files using the RHEL web console.

Procedure

  1. Log into the RHEL web console. For details see Logging in to the web console.
  2. Click Logs.

Figure 8.1. Inspecting the log files in the RHEL 9 web console

viewing logs

8.4. Viewing logs using the command line

The Journal is a component of systemd that helps to view and manage log files. It addresses problems connected with traditional logging, closely integrated with the rest of the system, and supports various logging technologies and access management for the log files.

You can use the journalctl command to view messages in the system journal using the command line, for example:

$ journalctl -b | grep kvm
May 15 11:31:41 localhost.localdomain kernel: kvm-clock: Using msrs 4b564d01 and 4b564d00
May 15 11:31:41 localhost.localdomain kernel: kvm-clock: cpu 0, msr 76401001, primary cpu clock
...

Table 8.1. Viewing system information

CommandDescription

journalctl

Shows all collected journal entries.

journalctl FILEPATH

Shows logs related to a specific file. For example, the journalctl /dev/sda command displays logs related to the /dev/sda file system.

journalctl -b

Shows logs for the current boot.

journalctl -k -b -1

Shows kernel logs for the current boot.

Table 8.2. Viewing information about specific services

CommandDescription

journalctl -b _SYSTEMD_UNIT=<name.service>

Filters log to show entries matching the systemd service.

journalctl -b _SYSTEMD_UNIT=<name.service> _PID=<number>

Combines matches. For example, this command shows logs for systemd-units that match <name.service> and the PID <number>.

journalctl -b _SYSTEMD_UNIT=<name.service> _PID=<number> + _SYSTEMD_UNIT=<name2.service>

The plus sign (+) separator combines two expressions in a logical OR. For example, this command shows all messages from the <name.service> service process with the PID plus all messages from the <name2.service> service (from any of its processes).

journalctl -b _SYSTEMD_UNIT=<name.service> _SYSTEMD_UNIT=<name2.service>

This command shows all entries matching either expression, referring to the same field. Here, this command shows logs matching a systemd-unit <name.service> or a systemd-unit <name2.service>.

Table 8.3. Viewing logs related to specific boots

CommandDescription

journalctl --list-boots

Shows a tabular list of boot numbers, their IDs, and the timestamps of the first and last message pertaining to the boot. You can use the ID in the next command to view detailed information.

journalctl --boot=ID _SYSTEMD_UNIT=<name.service>

Shows information about the specified boot ID.

8.5. Additional resources