Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 13. Getting started with kernel logging

Log files are files that contain messages about the system, including the kernel, services, and applications running on it. The logging system in Red Hat Enterprise Linux is based on the built-in syslog protocol. Various utilities use this system to record events and organize them into log files. These files are useful when auditing the operating system or troubleshooting problems.

13.1. What is the kernel ring buffer

During the boot process, the console provides a lot of important information about the initial phase of the system startup. To avoid loss of the early messages the kernel utilizes what is called a ring buffer. This buffer stores all messages, including boot messages, generated by the printk() function within the kernel code. The messages from the kernel ring buffer are then read and stored in log files on permanent storage, for example, by the syslog service.

The buffer mentioned above is a cyclic data structure which has a fixed size, and is hard-coded into the kernel. Users can display data stored in the kernel ring buffer through the dmesg command or the /var/log/boot.log file. When the ring buffer is full, the new data overwrites the old.

Additional resources

  • syslog(2) and dmesg(1) manual page

13.2. Role of printk on log-levels and kernel logging

Each message the kernel reports has a log-level associated with it that defines the importance of the message. The kernel ring buffer, as described in What is the kernel ring buffer, collects kernel messages of all log-levels. It is the kernel.printk parameter that defines what messages from the buffer are printed to the console.

The log-level values break down in this order:

0
Kernel emergency. The system is unusable.
1
Kernel alert. Action must be taken immediately.
2
Condition of the kernel is considered critical.
3
General kernel error condition.
4
General kernel warning condition.
5
Kernel notice of a normal but significant condition.
6
Kernel informational message.
7
Kernel debug-level messages.

By default, kernel.printk in RHEL 8 contains the following four values: 

# sysctl kernel.printk
kernel.printk = 7	4	1	7

The four values define the following, in order:

  1. Console log-level, defines the lowest priority of messages printed to the console.
  2. Default log-level for messages without an explicit log-level attached to them.
  3. Sets the lowest possible log-level configuration for the console log-level.

  4. Sets default value for the console log-level at boot time.

    Each of these values above defines a different rule for handling error messages.

Important

The default 7 4 1 7 printk value allows for better debugging of kernel activity. However, when coupled with a serial console, this printk setting might cause intense I/O bursts that might lead to a RHEL system becoming temporarily unresponsive. To avoid these situations, setting a printk value of 4 4 1 7 typically works, but at the expense of losing the extra debugging information.

Also note that certain kernel command line parameters, such as quiet or debug, change the default kernel.printk values.

Additional resources

  • syslog(2) manual page