Chapter 5. Configuring kernel parameters at runtime

As a system administrator, you can modify many facets of the Red Hat Enterprise Linux kernel’s behavior at runtime. This section describes how to configure kernel parameters at runtime by using the sysctl command and by modifying the configuration files in the /etc/sysctl.d/ and /proc/sys/ directories.

5.1. What are kernel parameters

Kernel parameters are tunable values which you can adjust while the system is running. There is no requirement to reboot or recompile the kernel for changes to take effect.

It is possible to address the kernel parameters through:

  • The sysctl command
  • The virtual file system mounted at the /proc/sys/ directory
  • The configuration files in the /etc/sysctl.d/ directory

Tunables are divided into classes by the kernel subsystem. Red Hat Enterprise Linux has the following tunable classes:

Table 5.1. Table of sysctl classes

Tunable classSubsystem

abi

Execution domains and personalities

crypto

Cryptographic interfaces

debug

Kernel debugging interfaces

dev

Device-specific information

fs

Global and specific file system tunables

kernel

Global kernel tunables

net

Network tunables

sunrpc

Sun Remote Procedure Call (NFS)

user

User Namespace limits

vm

Tuning and management of memory, buffers, and cache

Additional resources

  • For more information about sysctl, see sysctl(8) manual pages.
  • For more information about /etc/sysctl.d/ see, sysctl.d(5) manual pages.

5.2. Setting kernel parameters at runtime

Important

Configuring kernel parameters on a production system requires careful planning. Unplanned changes may render the kernel unstable, requiring a system reboot. Verify that you are using valid options before changing any kernel values.

5.2.1. Configuring kernel parameters temporarily with sysctl

The following procedure describes how to use the sysctl command to temporarily set kernel parameters at runtime. The command is also useful for listing and filtering tunables.

Prerequisites

Procedure

  1. To list all parameters and their values, use the following: 

    # sysctl -a
    Note

    The # sysctl -a command displays kernel parameters, which can be adjusted at runtime and at boot time.

  2. To configure a parameter temporarily, use the command as in the following example: 

    # sysctl <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

    The sample command above changes the parameter value while the system is running. The changes take effect immediately, without a need for restart.

    Note

    The changes return back to default after your system reboots.

Additional resources

  • For more information about sysctl, see the sysctl(8) manual page.
  • To permanently modify kernel parameters, either use the sysctl command to write the values to the /etc/sysctl.conf file or make manual changes to the configuration files in the /etc/sysctl.d/ directory.

5.2.2. Configuring kernel parameters permanently with sysctl

The following procedure describes how to use the sysctl command to permanently set kernel parameters.

Prerequisites

Procedure

  1. To list all parameters, use the following: 

    # sysctl -a

    The command displays all kernel parameters that can be configured at runtime.

  2. To configure a parameter permanently: 

    # sysctl -w <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE> >> /etc/sysctl.conf

    The sample command changes the tunable value and writes it to the /etc/sysctl.conf file, which overrides the default values of kernel parameters. The changes take effect immediately and persistently, without a need for restart.

Note

To permanently modify kernel parameters you can also make manual changes to the configuration files in the /etc/sysctl.d/ directory.

Additional resources

5.2.3. Using configuration files in /etc/sysctl.d/ to adjust kernel parameters

The following procedure describes how to manually modify configuration files in the /etc/sysctl.d/ directory to permanently set kernel parameters.

Prerequisites

Procedure

  1. Create a new configuration file in /etc/sysctl.d/: 

    # vim /etc/sysctl.d/<some_file.conf>

  2. Include kernel parameters, one per line, as follows: 

    <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
<TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
  3. Save the configuration file.

  4. Reboot the machine for the changes to take effect.

    • Alternatively, to apply changes without rebooting, execute: 

      # sysctl -p /etc/sysctl.d/<some_file.conf>

      The command enables you to read values from the configuration file, which you created earlier.

Additional resources

  • For more information about sysctl, see the sysctl(8) manual page.
  • For more information about /etc/sysctl.d/, see the sysctl.d(5) manual page.

5.2.4. Configuring kernel parameters temporarily through /proc/sys/

The following procedure describes how to set kernel parameters temporarily through the files in the virtual file system /proc/sys/ directory.

Prerequisites

Procedure

  1. Identify a kernel parameter you want to configure: 

    # ls -l /proc/sys/<TUNABLE_CLASS>/

    The writable files returned by the command can be used to configure the kernel. The files with read-only permissions provide feedback on the current settings.

  2. Assign a target value to the kernel parameter: 

    # echo <TARGET_VALUE> > /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

    The command makes configuration changes that will disappear once the system is restarted.

  3. Optionally, verify the value of the newly set kernel parameter: 

    # cat /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

Additional resources

5.3. Keeping kernel panic parameters disabled in virtualized environments

When configuring a virtualized environment in Red Hat Enterprise Linux 8 (RHEL 8), you should not enable the softlockup_panic and nmi_watchdog kernel parameters, as the virtualized environment may trigger a spurious soft lockup that should not require a system panic.

The following sections explain the reasons behind this advice by summarizing:

  • What causes a soft lockup.
  • Describing the kernel parameters that control a system’s behavior on a soft lockup.
  • Explaining how soft lockups may be triggered in a virtualized environment.

5.3.1. What is a soft lockup

A soft lockup is a situation usually caused by a bug, when a task is executing in kernel space on a CPU without rescheduling. The task also does not allow any other task to execute on that particular CPU. As a result, a warning is displayed to a user through the system console. This problem is also referred to as the soft lockup firing.

Additional resources

  • For a technical reason behind a soft lockup, example log messages, and other details, see the following Knowledge Article.

5.3.2. Parameters controlling kernel panic

The following kernel parameters can be set to control a system’s behavior when a soft lockup is detected.

softlockup_panic

Controls whether or not the kernel will panic when a soft lockup is detected.

TypeValueEffect

Integer

0

kernel does not panic on soft lockup

Integer

1

kernel panics on soft lockup

By default, on RHEL8 this value is 0.

In order to panic, the system needs to detect a hard lockup first. The detection is controlled by the nmi_watchdog parameter.

nmi_watchdog

Controls whether lockup detection mechanisms (watchdogs) are active or not. This parameter is of integer type.

ValueEffect

0

disables lockup detector

1

enables lockup detector

The hard lockup detector monitors each CPU for its ability to respond to interrupts.

watchdog_thresh

Controls frequency of watchdog hrtimer, NMI events, and soft/hard lockup thresholds.

Default thresholdSoft lockup threshold

10 seconds

2 * watchdog_thresh

Setting this parameter to zero disables lockup detection altogether.

Additional resources

5.3.3. Spurious soft lockups in virtualized environments

The soft lockup firing on physical hosts, as described in Section 5.3.1, “What is a soft lockup”, usually represents a kernel or hardware bug. The same phenomenon happening on guest operating systems in virtualized environments may represent a false warning.

Heavy work-load on a host or high contention over some specific resource such as memory, usually causes a spurious soft lockup firing. This is because the host may schedule out the guest CPU for a period longer than 20 seconds. Then when the guest CPU is again scheduled to run on the host, it experiences a time jump which triggers due timers. The timers include also watchdog hrtimer, which can consequently report a soft lockup on the guest CPU.

Because a soft lockup in a virtualized environment may be spurious, you should not enable the kernel parameters that would cause a system panic when a soft lockup is reported on a guest CPU.

Important

To understand soft lockups in guests, it is essential to know that the host schedules the guest as a task, and the guest then schedules its own tasks.

Additional resources

5.4. Adjusting kernel parameters for database servers

There are different sets of kernel parameters which can affect performance of specific database applications. The following sections explain what kernel parameters to configure to secure efficient operation of database servers and databases.

5.4.1. Introduction to database servers

A database server is a hardware device which has a certain amount of main memory, and a database (DB) application installed. This DB application provides services as a means of writing the cached data from the main memory, which is usually small and expensive, to DB files (database). These services are provided to multiple clients on a network. There can be as many DB servers as a machine’s main memory and storage allows.

Red Hat Enterprise Linux 8 provides the following database applications:

  • MariaDB 10.3
  • MySQL 8.0
  • PostgreSQL 10
  • PostgreSQL 9.6
  • PostgreSQL 12 - available since RHEL 8.1.1

5.4.2. Parameters affecting performance of database applications

The following kernel parameters affect performance of database applications.

fs.aio-max-nr

Defines the maximum number of asynchronous I/O operations the system can handle on the server.

Note

Raising the fs.aio-max-nr parameter produces no additional changes beyond increasing the aio limit.

fs.file-max

Defines the maximum number of file handles (temporary file names or IDs assigned to open files) the system supports at any instance.

The kernel dynamically allocates file handles whenever a file handle is requested by an application. The kernel however does not free these file handles when they are released by the application. The kernel recycles these file handles instead. This means that over time the total number of allocated file handles will increase even though the number of currently used file handles may be low.

kernel.shmall
Defines the total number of shared memory pages that can be used system-wide. To use the entire main memory, the value of the kernel.shmall parameter should be ≤ total main memory size.
kernel.shmmax
Defines the maximum size in bytes of a single shared memory segment that a Linux process can allocate in its virtual address space.
kernel.shmmni
Defines the maximum number of shared memory segments the database server is able to handle.
net.ipv4.ip_local_port_range
Defines the port range the system can use for programs which want to connect to a database server without a specific port number.
net.core.rmem_default
Defines the default receive socket memory through Transmission Control Protocol (TCP).
net.core.rmem_max
Defines the maximum receive socket memory through Transmission Control Protocol (TCP).
net.core.wmem_default
Defines the default send socket memory through Transmission Control Protocol (TCP).
net.core.wmem_max
Defines the maximum send socket memory through Transmission Control Protocol (TCP).
vm.dirty_bytes / vm.dirty_ratio
Defines a threshold in bytes / in percentage of dirty-able memory at which a process generating dirty data is started in the write() function.
Note

Either vm.dirty_bytes or vm.dirty_ratio can be specified at a time.

vm.dirty_background_bytes / vm.dirty_background_ratio
Defines a threshold in bytes / in percentage of dirty-able memory at which the kernel tries to actively write dirty data to hard-disk.
Note

Either vm.dirty_background_bytes or vm.dirty_background_ratio can be specified at a time.

vm.dirty_writeback_centisecs

Defines a time interval between periodic wake-ups of the kernel threads responsible for writing dirty data to hard-disk.

This kernel parameters measures in 100th’s of a second.

vm.dirty_expire_centisecs

Defines the time after which dirty data is old enough to be written to hard-disk.

This kernel parameters measures in 100th’s of a second.

Additional resources