Procedure

1. To update the kernel, use the following:

   # dnf update kernel

   This command updates the kernel along with all dependencies to the latest available version.

2. Reboot your system for the changes to take effect.

Procedure

1. Select a kernel module you want to load.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Load the relevant kernel module:

   # modprobe <MODULE_NAME>

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

3. Optionally, verify the relevant module was loaded:

   $ lsmod | grep <MODULE_NAME>

   If the module was loaded correctly, this command displays the relevant kernel module. For example:

   $ lsmod | grep serio_raw
   serio_raw              16384  0

Procedure

1. Execute the lsmod command and select a kernel module you want to unload.

   If a kernel module has dependencies, unload those prior to unloading the kernel module. For details on identifying modules with dependencies, see Listing currently loaded kernel modules and Kernel module dependencies.

2. Unload the relevant kernel module:

   # modprobe -r <MODULE_NAME>

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

   Warning

   Do not unload kernel modules when they are used by the running system. Doing so can lead to an unstable or non-operational system.

3. Optionally, verify the relevant module was unloaded:

   $ lsmod | grep <MODULE_NAME>

   If the module was unloaded successfully, this command does not display any output.

Procedure

1. Select a kernel module you want to load during the boot process.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Create a configuration file for the module:

   # echo <MODULE_NAME> > /etc/modules-load.d/<MODULE_NAME>.conf

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

3. Optionally, after reboot, verify the relevant module was loaded:

   $ lsmod | grep <MODULE_NAME>

   The example command above should succeed and display the relevant kernel module.

Procedure

1. Select a kernel module that you want to put in a denylist:

   $ lsmod
   
   Module                  Size  Used by
   fuse                  126976  3
   xt_CHECKSUM            16384  1
   ipt_MASQUERADE         16384  1
   uinput                 20480  1
   xt_conntrack           16384  1
   …​

   The lsmod command displays a list of modules loaded to the currently running kernel.

   • Alternatively, identify an unloaded kernel module you want to prevent from potentially loading.

     All kernel modules are located in the /lib/modules/<KERNEL_VERSION>/kernel/<SUBSYSTEM>/ directory.

2. Create a configuration file for a denylist:

   # vim /etc/modprobe.d/blacklist.conf
   
   	# Blacklists <KERNEL_MODULE_1>
   	blacklist <MODULE_NAME_1>
   	install <MODULE_NAME_1> /bin/false
   
   	# Blacklists <KERNEL_MODULE_2>
   	blacklist <MODULE_NAME_2>
   	install <MODULE_NAME_2> /bin/false
   
   	# Blacklists <KERNEL_MODULE_n>
   	blacklist <MODULE_NAME_n>
   	install <MODULE_NAME_n> /bin/false
   	…​

   The example shows the contents of the blacklist.conf file, edited by the vim editor. The blacklist line ensures that the relevant kernel module will not be automatically loaded during the boot process. The blacklist command, however, does not prevent the module from being loaded as a dependency for another kernel module that is not in a denylist. Therefore the install line causes the /bin/false to run instead of installing a module.

   The lines starting with a hash sign are comments to make the file more readable.

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

3. Create a backup copy of the current initial ramdisk image before rebuilding:

   # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img

   The command above creates a backup initramfs image in case the new version has an unexpected problem.

   • Alternatively, create a backup copy of other initial ramdisk image which corresponds to the kernel version for which you want to put kernel modules in a denylist:

     # cp /boot/initramfs-<SOME_VERSION>.img /boot/initramfs-<SOME_VERSION>.img.bak.$(date +%m-%d-%H%M%S)

4. Generate a new initial ramdisk image to reflect the changes:

   # dracut -f -v

   • If you are building an initial ramdisk image for a different kernel version than you are currently booted into, specify both target initramfs and kernel version:

     # dracut -f -v /boot/initramfs-<TARGET_VERSION>.img <CORRESPONDING_TARGET_KERNEL_VERSION>

5. Reboot the system:

   $ reboot

Procedure

1. Create the /root/testmodule/test.c file with the following content.

   #include <linux/module.h>
   #include <linux/kernel.h>
   
   int init_module(void)
       { printk("Hello World\n This is a test\n"); return 0; }
   
   void cleanup_module(void)
       { printk("Good Bye World"); }
   
   MODULE_LICENSE("GPL");

   The test.c file is a source file that provides the main functionality to the kernel module. The file has been created in a dedicated /root/testmodule/ directory for organizational purposes. After the module compilation, the /root/testmodule/ directory will contain multiple files.

   The test.c file includes from the system libraries:

   • The linux/kernel.h header file is necessary for the printk() function in the example code.

   • The linux/module.h file contains function declarations and macro definitions to be shared between several source files written in C programming language.

     Next follow the init_module() and cleanup_module() functions to start and end the kernel logging function printk(), which prints text.

2. Create the /root/testmodule/Makefile file with the following content.

   obj-m := test.o

   The Makefile contains instructions that the compiler has to produce an object file specifically named test.o. The obj-m directive specifies that the resulting test.ko file is going to be compiled as a loadable kernel module. Alternatively, the obj-y directive would instruct to build test.ko as a built-in kernel module.

3. Compile the kernel module.

   # make -C /lib/modules/$(uname -r)/build M=/root/testmodule modules
   make: Entering directory '/usr/src/kernels/5.14.0-70.17.1.el9_0.x86_64'
     CC [M]  /root/testmodule/test.o
     MODPOST /root/testmodule/Module.symvers
     CC [M]  /root/testmodule/test.mod.o
     LD [M]  /root/testmodule/test.ko
     BTF [M] /root/testmodule/test.ko
   Skipping BTF generation for /root/testmodule/test.ko due to unavailability of vmlinux
   make: Leaving directory '/usr/src/kernels/5.14.0-70.17.1.el9_0.x86_64'

   The compiler creates an object file (test.o) for each source file (test.c) as an intermediate step before linking them together into the final kernel module (test.ko).

   After a successful compilation, /root/testmodule/ contains additional files that relate to the compiled custom kernel module. The compiled module itself is represented by the test.ko file.

Verification

1. Optional: check the contents of the /root/testmodule/ directory:

   # ls -l /root/testmodule/
   total 152
   -rw-r—​r--. 1 root root    16 Jul 26 08:19 Makefile
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 modules.order
   -rw-r—​r--. 1 root root     0 Jul 26 08:20 Module.symvers
   -rw-r—​r--. 1 root root   224 Jul 26 08:18 test.c
   -rw-r—​r--. 1 root root 62176 Jul 26 08:20 test.ko
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 test.mod
   -rw-r—​r--. 1 root root   849 Jul 26 08:20 test.mod.c
   -rw-r—​r--. 1 root root 50936 Jul 26 08:20 test.mod.o
   -rw-r—​r--. 1 root root 12912 Jul 26 08:20 test.o

2. Copy the kernel module to the /lib/modules/$(uname -r)/ directory:

   # cp /root/testmodule/test.ko /lib/modules/$(uname -r)/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v test
   insmod /lib/modules/5.14.0-1.el9.x86_64/test.ko

5. Verify that the kernel module was successfully loaded:

   # lsmod | grep test
   test                   16384  0

6. Read the latest messages from the kernel ring buffer:

   # dmesg
   [74422.545004] Hello World
                   This is a test

Procedure

1. Create a configuration file with parameters for the key pair generation:

   # cat << EOF > configuration_file.config
   [ req ]
   default_bits = 4096
   distinguished_name = req_distinguished_name
   prompt = no
   string_mask = utf8only
   x509_extensions = myexts
   
   [ req_distinguished_name ]
   O = Organization
   CN = Organization signing key
   emailAddress = E-mail address
   
   [ myexts ]
   basicConstraints=critical,CA:FALSE
   keyUsage=digitalSignature
   subjectKeyIdentifier=hash
   authorityKeyIdentifier=keyid
   EOF

2. Create an X.509 public and private key pair as shown in the following example:

   # openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 \
   -batch -config configuration_file.config -outform DER \
   -out my_signing_key_pub.der \
   -keyout my_signing_key.priv

   The public key will be written to the my_signing_key_pub.der file and the private key will be written to the my_signing_key.priv file.

   Important

   In RHEL 9, the validity dates of the key pair do not matter. The key does not expire, but it is recommended for good security practices that the kernel module be signed within the validity period of its signing key. However, the sign-file utility will not warn you and the key will be usable regardless of the validity dates.

3. Optionally, you can review the validity dates of your public keys like in the example below:

   # openssl x509 -inform der -text -noout -in my_signing_key_pub.der
   
   Validity
               Not Before: Feb 14 16:34:37 2019 GMT
               Not After : Feb 11 16:34:37 2029 GMT

4. Enroll your public key on all systems where you want to authenticate and load your kernel module.

Procedure

1. Request the addition of your public key to the MOK list:

   # mokutil --import my_signing_key_pub.der

   You will be asked to enter and confirm a password for this MOK enrollment request.

2. Reboot the machine.

   The pending MOK key enrollment request will be noticed by shim.efi and it will launch MokManager.efi to allow you to complete the enrollment from the UEFI console.

3. Choose "Enroll MOK" and enter the password you previously associated with this request when prompted and confirm the enrollment.

   Your public key is added to the MOK list, which is persistent.

Procedure

1. Verify that your public keys are on the system keyring:

   # keyctl list %:.platform

2. Install the kernel-modules-extra package which will create the /lib/modules/$(uname -r)/extra/ directory:

   # dnf -y install kernel-modules-extra

3. Copy the kernel module into the /extra/ directory of the kernel you want:

   # cp my_module.ko /lib/modules/$(uname -r)/extra/

4. Update the modular dependency list:

   # depmod -a

5. Load the kernel module and verify that it was successfully loaded:

   # modprobe -v my_module
   # lsmod | grep my_module

   1. Optionally, to load the module on boot, add it to the /etc/modules-loaded.d/my_module.conf file:

      # echo "my_module" > /etc/modules-load.d/my_module.conf

Procedure

1. Select the kernel you want to start when the GRUB 2 boot menu appears and press the e key to edit the kernel parameters.
2. Find the kernel command line by moving the cursor down. The kernel command line starts with linux on 64-Bit IBM Power Series and x86-64 BIOS-based systems, or linuxefi on UEFI systems.

3. Move the cursor to the end of the line.

   Note

   Press Ctrl+a to jump to the start of the line and Ctrl+e to jump to the end of the line. On some systems, Home and End keys might also work.

4. Edit the kernel parameters as required. For example, to run the system in emergency mode, add the emergency parameter at the end of the linux line:

1. Press Ctrl+x to boot with the selected kernel and the modified command line parameters.

Procedure

1. Add the following two lines to the /etc/default/grub file:

   GRUB_TERMINAL="serial"
   GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

   The first line disables the graphical terminal. The GRUB_TERMINAL key overrides values of GRUB_TERMINAL_INPUT and GRUB_TERMINAL_OUTPUT keys.

   The second line adjusts the baud rate (--speed), parity and other values to fit your environment and hardware. Note that a much higher baud rate, for example 115200, is preferable for tasks such as following log files.

2. Update the GRUB 2 configuration file.

   • On BIOS-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

3. Reboot the system for the changes to take effect.

Procedure

1. To list all parameters and their values, use the following:

   # sysctl -a

   Note

   The # sysctl -a command displays kernel parameters, which can be adjusted at runtime and at boot time.

2. To configure a parameter temporarily, use the command as in the following example:

   # sysctl <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

   The sample command above changes the parameter value while the system is running. The changes take effect immediately, without a need for restart.

   Note

   The changes return back to default after your system reboots.

Procedure

1. To list all parameters, use the following:

   # sysctl -a

   The command displays all kernel parameters that can be configured at runtime.

2. To configure a parameter permanently:

   # sysctl -w <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE> >> /etc/sysctl.conf

   The sample command changes the tunable value and writes it to the /etc/sysctl.conf file, which overrides the default values of kernel parameters. The changes take effect immediately and persistently, without a need for restart.

Procedure

1. Create a new configuration file in /etc/sysctl.d/:

   # vim /etc/sysctl.d/<some_file.conf>

2. Include kernel parameters, one per line, as follows:

   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

3. Save the configuration file.

4. Reboot the machine for the changes to take effect.

   • Alternatively, to apply changes without rebooting, execute:

     # sysctl -p /etc/sysctl.d/<some_file.conf>

     The command enables you to read values from the configuration file, which you created earlier.

Procedure

1. Identify a kernel parameter you want to configure:

   # ls -l /proc/sys/<TUNABLE_CLASS>/

   The writable files returned by the command can be used to configure the kernel. The files with read-only permissions provide feedback on the current settings.

2. Assign a target value to the kernel parameter:

   # echo <TARGET_VALUE> > /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

   The command makes configuration changes that will disappear once the system is restarted.

3. Optionally, verify the value of the newly set kernel parameter:

   # cat /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

1. value. Console log-level, defines the lowest priority of messages printed to the console.
2. value. Default log-level for messages without an explicit log-level attached to them.
3. value. Sets the lowest possible log-level configuration for the console log-level.

4. value. Sets default value for the console log-level at boot time.

   Each of these values above defines a different rule for handling error messages.

Procedure

1. Check the status of kdump installation on your system:

   # rpm -q kexec-tools

   Output if the package is installed:

   # kexec-tools-2.0.22-13.el9.x86_64

   Output if the package is not installed:

   # package kexec-tools is not installed

2. Install kdump and other necessary packages:

   # yum install kexec-tools

Procedure

1. Configure the default value for crashkernel:

   # kdumpctl reset-crashkernel --kernel=ALL

2. (Optional) To use a custom crashkernel value:

   1. Configure the required memory reserve:

      crashkernel=192M

      The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4 GB. If the total amount of memory is more than 4 GB, 256 MB is reserved for kdump instead.

      • (Optional) Offset the reserved memory:

        Some systems require to reserve memory with a certain fixed offset since crash kernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

        crashkernel=192M@16M

        The example above reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If the offset parameter is set to 0 or omitted entirely, kdump offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example, crashkernel=1G-4G:192M,2G-64G:256M@16M.

   2. Update the bootloader configuration:

      # grubby --update-kernel ALL --args "crashkernel=<CUSTOM-VALUE>”

      On IBM Z systems that use the zIPL bootloader, the command adds a new kernel parameter to each /boot/loader/entries/<ENTRY>.conf file.

      • On IBM Z systems, to update the boot menu, execute the zipl command with no options specified:

        # zipl

3. Reboot for changes to take effect:

   # reboot

Verification

1. Activate the sysrq key to boot into the kdump kernel:

   # echo 1 > /proc/sys/kernel/sysrq
   # echo c > /proc/sysrq-trigger

   This forces the Linux kernel to crash and copy the address-YYYY-MM-DD-HH:MM:SS/vmcore file to the target location specified in the configuration file.

2. Verify that the vmcore file is dumped in the target as specified in the /etc/kdump.conf file:

   $ ls /var/crash/127.0.0.1-2022-01-18-0
   
   /var/crash/127.0.0.1-2022-01-18-05:23:10':
   kexec-dmesg.log  vmcore  vmcore-dmesg.txt

   In this example, the kernel saves the vmcore in the default target directory, /var/crash/.

Procedure

1. As root user, edit the /etc/kdump.conf configuration file to remove the hash sign ("#") from the beginning of the following command:

   core_collector makedumpfile -z -d 31 --message-level 1

2. To enable dump file compression, specify one of the makedumpfile options:

   core_collector makedumpfile -z -d 31 --message-level 1

   where,

   • -z specifies the dump compressed file format.
   • -d specifies dump level as 31.
   • --message-level specifies message level as 1.

Procedure

1. As root, remove the hash sign ("#") from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

2. Replace the value with a desired action.

   failure_action poweroff

Procedure

1. To enable the kdump service, use the following command:

   # systemctl enable kdump.service

   This enables the service for multi-user.target.

2. To start the service in the current session, use the following command:

   # systemctl start kdump.service

3. To stop the kdump service, type the following command:

   # systemctl stop kdump.service

4. To disable the kdump service, execute the following command:

   # systemctl disable kdump.service

Procedure

1. Reboot the system with kdump enabled.

2. Make sure that kdump is running:

   # systemctl is-active kdump
   active

3. Force the Linux kernel to crash:

   # echo 1 > /proc/sys/kernel/sysrq
   # echo c > /proc/sysrq-trigger

   Warning

   The command above crashes the kernel, and a reboot is required.

   Once booted again, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file (by default to /var/crash/).

   Note

   This action confirms the validity of the configuration. Also it is possible to use this action to record how long it takes for a crash dump to complete with a representative work-load.

Procedure

1. Select a kernel module that you intend to block from loading.

   $ lsmod
   
   Module                  Size  Used by
   fuse                  126976  3
   xt_CHECKSUM            16384  1
   ipt_MASQUERADE         16384  1
   uinput                 20480  1
   xt_conntrack           16384  1

   The lsmod command displays a list of modules that are loaded to the currently running kernel.

2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file.

   # KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

   Also,consider the following example using the modprobe.blacklist=<modules> configuration option.

   # KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"

3. Restart the kdump service.

   # systemctl restart kdump

Procedure

1. Print the estimate crashkernel value:

   # kdumpctl estimate
   
   Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement
      Reserved crashkernel:    256M
      Recommended crashkernel: 652M
      Kernel image size:   47M
      Kernel modules size: 8M
      Initramfs size:      20M
      Runtime reservation: 64M
      LUKS required size:  512M
      Large modules: none
      WARNING: Current crashkernel size is lower than recommended size 652M.

2. Configure the amount of required memory by increasing crashkernel to the desired value.

   # grubby –args=”crashkernel=652M” --update-kernel=ALL

3. Reboot for changes to take effect.

   # reboot

Procedure

1. To configure final_action, edit the /etc/kdump.conf file and add one of the following options:

   # final_action <reboot | halt | poweroff>

2. Restart the kdump service for the changes to take effect:

   # kdumpctl restart

Procedure:

1. To configure an action to take if the dump fails, edit the /etc/kdump.conf file and specify one of the failure_action options:

   # failure_action <reboot | halt | poweroff | shell | dump_to_rootfs>

2. Restart the kdump service for the changes to take effect:

   # kdumpctl restart

Procedure

1. Install the kexec-tools package.

2. Configure the default value for crashkernel.

   # kdumpctl reset-crashkernel –fadump=on --kernel=ALL

3. (Optional) Reserve boot memory instead of the default value.

   # grubby --update-kernel ALL --args=”fadump=on crashkernel=xxM"

   where, xx is the required memory size in megabytes.

   Note

   When specifying boot configuration options, test the configuration by rebooting the kernel with kdump enabled. If the kdump kernel fails to boot, increase the crashkernel value gradually to set an appropriate value.

4. Reboot for changes to take effect.

   # reboot

Procedure

1. Add or edit the following lines in the /etc/sysctl.conf file to ensure that kdump starts as expected for sadump:

   # kernel.panic=0
   kernel.unknown_nmi_panic=1

   Warning

   In particular, ensure that after kdump, the system does not reboot. If the system reboots after kdump has fails to save the vmcore file, then it is not possible to invoke the sadump.

2. Set the failure_action parameter in /etc/kdump.conf appropriately as halt or shell.

   # failure_action shell

Procedure

1. Enable the relevant repositories:

   # subscription-manager repos --enable pass:quotes[baseos repository]

   # subscription-manager repos --enable pass:quotes[appstream repository]

   # subscription-manager repos --enable pass:quotes[rhel-9-for-x86_64-baseos-debug-rpms]

2. Install the crash package:

   # yum install crash

3. Install the kernel-debuginfo package:

   # yum install kernel-debuginfo

   The package corresponds to your running kernel and provides the data necessary for the dump analysis.

Procedure

1. To start the crash utility, two necessary parameters need to be passed to the command:

   • The debug-info (a decompressed vmlinuz image), for example /usr/lib/debug/lib/modules/4.18.0-5.el9.x86_64/vmlinux provided through a specific kernel-debuginfo package.

   • The actual vmcore file, for example /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore

     The resulting crash command then looks like this:

     # crash /usr/lib/debug/lib/modules/4.18.0-5.el9.x86_64/vmlinux /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore

     Use the same <kernel> version that was captured by kdump.

     Running the crash utility

     The following example shows analyzing a core dump created on October 6 2018 at 14:05 PM, using the 4.18.0-5.el8.x86_64 kernel.

     WARNING: kernel relocated [202MB]: patching 90160 gdb minimal_symbol values
     
           KERNEL: /usr/lib/debug/lib/modules/4.18.0-5.el8.x86_64/vmlinux
         DUMPFILE: /var/crash/127.0.0.1-2018-10-06-14:05:33/vmcore  [PARTIAL DUMP]
             CPUS: 2
             DATE: Sat Oct  6 14:05:16 2018
           UPTIME: 01:03:57
     LOAD AVERAGE: 0.00, 0.00, 0.00
            TASKS: 586
         NODENAME: localhost.localdomain
          RELEASE: 4.18.0-5.el8.x86_64
          VERSION: #1 SMP Wed Aug 29 11:51:55 UTC 2018
          MACHINE: x86_64  (2904 Mhz)
           MEMORY: 2.9 GB
            PANIC: "sysrq: SysRq : Trigger a crash"
              PID: 10635
          COMMAND: "bash"
             TASK: ffff8d6c84271800  [THREAD_INFO: ffff8d6c84271800]
              CPU: 1
            STATE: TASK_RUNNING (SYSRQ)
     
     crash>

2. To exit the interactive prompt and terminate crash, type exit or q.

   crash> exit
   ~]#

Procedure

1. Access the Kernel Oops Analyzer tool.

2. To diagnose a kernel crash issue, upload a kernel oops log generated in vmcore.

   • Alternatively you can also diagnose a kernel crash issue by providing a text message or a vmcore-dmesg.txt as an input.

     
3. Click DETECT to compare the oops message based on information from the makedumpfile against known solutions.

1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel by systemd on next boot.
2. The kpatch module is loaded into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
3. When the kernel accesses the patched function, it is redirected by the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Procedure

1. Optionally, check your kernel version:

   # uname -r
   5.14.0-1.el9.x86_64

2. Search for a live patching package that corresponds to the version of your kernel:

   # dnf search $(uname -r)

3. Install the live patching package:

   # dnf install "kpatch-patch = $(uname -r)"

   The command above installs and applies the latest cumulative live patches for that specific kernel only.

   If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-5_14_0-1-0-0.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

4. Optionally, verify that the kernel is patched:

   # kpatch list
   Loaded patch modules:
   kpatch_5_14_0_1_0_1 [enabled]
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)
   …​

   The output shows that the kernel patch module has been loaded into the kernel, which is now patched with the latest fixes from the kpatch-patch-5_14_0-1-0-1.el9.x86_64.rpm package.

Procedure

1. Optionally, check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            5.14.0-1.el9              @beaker-BaseOS
   kernel-core.x86_64            5.14.0-2.el9              @@commandline
   ...
   
   # uname -r
   5.14.0-2.el9.x86_64

2. Install the kpatch-dnf plugin:

   # dnf install kpatch-dnf

3. Enable automatic subscription to kernel live patches:

   # dnf kpatch auto
   Updating Subscription Management repositories.
   Last metadata expiration check: 1:38:21 ago on Fri 17 Sep 2021 07:29:53 AM EDT.
   Dependencies resolved.
   ==================================================
    Package                             Architecture
   ==================================================
   Installing:
    kpatch-patch-5_14_0-1               x86_64
    kpatch-patch-5_14_0-2               x86_64
   
   Transaction Summary
   ===================================================
   Install  2 Packages
   …​

   This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.

   In the future, when you update the kernel, live patches will automatically be installed during the new kernel installation process.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-5_14_0-1-0-0.el9.x86_64.rpm .The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optionally, check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            5.14.0-1.el9              @beaker-BaseOS
   kernel-core.x86_64            5.14.0-2.el9              @@commandline
   ...
   
   # uname -r
   5.14.0-2.el9.x86_64

2. Disable automatic subscription to kernel live patches:

   # dnf kpatch manual
   Updating Subscription Management repositories.

Procedure

1. Select the live patching package:

   # dnf list installed | grep kpatch-patch
   kpatch-patch-5_14_0-1.x86_64        0-1.el9        @@commandline
   …​

   The example output above lists live patching packages that you installed.

2. Remove the live patching package:

   # dnf remove kpatch-patch-5_14_0-1.x86_64

   When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.

3. Reboot your system.

4. Verify that the live patching package has been removed:

   # dnf list installed | grep kpatch-patch

   The command displays no output if the package has been successfully removed.

5. Optionally, verify that the kernel live patching solution is disabled:

   # kpatch list
   Loaded patch modules:

   The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.

Procedure

1. Select a kernel patch module:

   # kpatch list
   Loaded patch modules:
   kpatch_5_14_0_1_0_1 [enabled]
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)
   …​

2. Uninstall the selected kernel patch module:

   # kpatch uninstall kpatch_5_14_0_1_0_1
   uninstalling kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

   • Note that the uninstalled kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_5_14_0_1_0_1 [enabled]
     
     Installed patch modules:
     <NO_RESULT>

     When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

3. Reboot your system.

4. Optionally, verify that the kernel patch module has been uninstalled:

   # kpatch list
   Loaded patch modules:
   …​

   The example output above shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Procedure

1. Verify kpatch.service is enabled:

   # systemctl is-enabled kpatch.service
   enabled

2. Disable kpatch.service:

   # systemctl disable kpatch.service
   Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

   • Note that the applied kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_5_14_0_1_0_1 [enabled]
     
     Installed patch modules:
     kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

3. Reboot your system.

4. Optionally, verify the status of kpatch.service:

   # systemctl status kpatch.service
   ● kpatch.service - "Apply kpatch kernel patches"
      Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
      Active: inactive (dead)

   The example output testifies that kpatch.service has been disabled and is not running. Thereby, the kernel live patching solution is not active.

5. Verify that the kernel patch module has been unloaded:

   # kpatch list
   Loaded patch modules:
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

   The example output above shows that a kernel patch module is still installed but the kernel is not patched.

1. Check the assigned values for the service of your choice.

   # systemctl show --property <unit file option> <service name>

2. Set the required value of the CPU time allocation policy option:

   # systemctl set-property <service name> <unit file option>=<value>

Procedure

1. To view which cgroup a process belongs to, run the # cat proc/<PID>/cgroup command:

   # cat /proc/2467/cgroup
   0::/system.slice/example.service

   The example output relates to a process of interest. In this case, it is a process identified by PID 2467, which belongs to the example.service unit. You can determine whether the process was placed in a correct control group as defined by the systemd unit file specifications.

2. To display what controllers the cgroup utilizes and the respective configuration files, check the cgroup directory:

   # cat /sys/fs/cgroup/system.slice/example.service/cgroup.controllers
   memory pids
   
   # ls /sys/fs/cgroup/system.slice/example.service/
   cgroup.controllers
   cgroup.events
   …​
   cpu.pressure
   cpu.stat
   io.pressure
   memory.current
   memory.events
   …​
   pids.current
   pids.events
   pids.max

Procedure

1. To see a dynamic account of currently running cgroups, execute the # systemd-cgtop command:

   # systemd-cgtop
   Control Group                            Tasks   %CPU   Memory  Input/s Output/s
   /                                          607   29.8     1.5G        -        -
   /system.slice                              125      -   428.7M        -        -
   /system.slice/ModemManager.service           3      -     8.6M        -        -
   /system.slice/NetworkManager.service         3      -    12.8M        -        -
   /system.slice/accounts-daemon.service        3      -     1.8M        -        -
   /system.slice/boot.mount                     -      -    48.0K        -        -
   /system.slice/chronyd.service                1      -     2.0M        -        -
   /system.slice/cockpit.socket                 -      -     1.3M        -        -
   /system.slice/colord.service                 3      -     3.5M        -        -
   /system.slice/crond.service                  1      -     1.8M        -        -
   /system.slice/cups.service                   1      -     3.1M        -        -
   /system.slice/dev-hugepages.mount            -      -   244.0K        -        -
   /system.slice/dev-mapper-rhel\x2dswap.swap   -      -   912.0K        -        -
   /system.slice/dev-mqueue.mount               -      -    48.0K        -        -
   /system.slice/example.service                2      -     2.0M        -        -
   /system.slice/firewalld.service              2      -    28.8M        -        -
   ...

   The example output displays currently running cgroups ordered by their resource usage (CPU, memory, disk I/O load). The list refreshes every 1 second by default. Therefore, it offers a dynamic insight into the actual resource usage of each control group.

Procedure

1. Modify the /usr/lib/systemd/system/example.service file to limit the memory usage of a service:

   …​
   [Service]
   MemoryMax=1500K
   …​

   The configuration above places a maximum memory limit, which the processes in a control group cannot exceed. The example.service service is part of such a control group which has imposed limitations. You can use suffixes K, M, G, or T to identify Kilobyte, Megabyte, Gigabyte, or Terabyte as a unit of measurement.

2. Reload all unit configuration files:

   # systemctl daemon-reload

3. Restart the service:

   # systemctl restart example.service

Verification

1. Check that the changes took effect:

   # cat /sys/fs/cgroup/system.slice/example.service/memory.max
   1536000

   The example output shows that the memory consumption was limited at around 1,500 KB.

1. Check the values of the CPUAffinity unit file option in the service of your choice:

   $ systemctl show --property <CPU affinity configuration option> <service name>

2. As a root, set the required value of the CPUAffinity unit file option for the CPU ranges used as the affinity mask:

   # systemctl set-property <service name> CPUAffinity=<value>

3. Restart the service to apply the changes.

   # systemctl restart <service name>

1. Set the CPU numbers for the CPUAffinity= option in the /etc/systemd/system.conf file.

2. Save the edited file and reload the systemd service:

   # systemctl daemon-reload

3. Reboot the server to apply the changes.

1. Check the values of the NUMAPolicy unit file option in the service of your choice:

   $ systemctl show --property <NUMA policy configuration option> <service name>

2. As a root, set the required policy type of the NUMAPolicy unit file option:

   # systemctl set-property <service name> NUMAPolicy=<value>

3. Restart the service to apply the changes.

   # systemctl restart <service name>

1. Search in the /etc/systemd/system.conf file for the NUMAPolicy option.
2. Edit the policy type and save the file.

3. Reload the systemd configuration:

   # systemd daemon-reload

4. Reboot the server.

Procedure

1. Enable zswap permanently:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="zswap.enabled=1"

2. Reboot the system for the changes to take effect.

Procedure

1. Create the /sys/fs/cgroup/Example/ directory:

   # mkdir /sys/fs/cgroup/Example/

   The /sys/fs/cgroup/Example/ directory defines a child group. When you create the /sys/fs/cgroup/Example/ directory, some cgroups-v2 interface files are automatically created in the directory. The /sys/fs/cgroup/Example/ directory contains also controller-specific files for the memory and pids controllers.

2. Optionally, inspect the newly created child control group:

   # ll /sys/fs/cgroup/Example/
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.controllers
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.events
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.freeze
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.procs
   …​
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.subtree_control
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 memory.events.local
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 memory.high
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 memory.low
   …​
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 pids.current
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 pids.events
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 pids.max

   The example output shows general cgroup control interface files such as cgroup.procs or cgroup.controllers. These files are common to all control groups, regardless of enabled controllers.

   The files such as memory.high and pids.max relate to the memory and pids controllers, which are in the root control group (/sys/fs/cgroup/), and are enabled by default by systemd.

   By default, the newly created child group inherits all settings from the parent cgroup. In this case, there are no limits from the root cgroup.

3. Verify that the desired controllers are available in the /sys/fs/cgroup/cgroup.controllers file:

   # cat /sys/fs/cgroup/cgroup.controllers
   cpuset cpu io memory hugetlb pids rdma

4. Enable the desired controllers. In this example it is cpu and cpuset controllers:

   # echo "+cpu" >> /sys/fs/cgroup/cgroup.subtree_control
   # echo "+cpuset" >> /sys/fs/cgroup/cgroup.subtree_control

   These commands enable the cpu and cpuset controllers for the immediate child groups of the /sys/fs/cgroup/ root control group. Including the newly created Example control group. A child group is where you can specify processes and apply control checks to each of the processes based on your criteria.

   Users can read the contents of the cgroup.subtree_control file at any level to get an idea of what controllers are going to be available for enablement in the immediate child group.

   Note

   By default, the /sys/fs/cgroup/cgroup.subtree_control file in the root control group contains memory and pids controllers.

5. Enable the desired controllers for child cgroups of the Example control group:

   # echo "+cpu +cpuset" >> /sys/fs/cgroup/Example/cgroup.subtree_control

   This command ensures that the immediate child control group will only have controllers relevant to regulate the CPU time distribution - not to memory or pids controllers.

6. Create the /sys/fs/cgroup/Example/tasks/ directory:

   # mkdir /sys/fs/cgroup/Example/tasks/

   The /sys/fs/cgroup/Example/tasks/ directory defines a child group with files that relate purely to cpu and cpuset controllers. You can now assign processes to this control group and utilize cpu and cpuset controller options for your processes.

7. Optionally, inspect the child control group:

   # ll /sys/fs/cgroup/Example/tasks
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.controllers
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.events
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.freeze
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.max.depth
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.max.descendants
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.procs
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.stat
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.subtree_control
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.threads
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.type
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.max
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.pressure
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.effective
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.partition
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.mems
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpuset.mems.effective
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpu.stat
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.weight
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.weight.nice
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 io.pressure
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 memory.pressure

Procedure

1. Configure desired CPU weights to achieve resource restrictions within the control groups:

   # echo "150" > /sys/fs/cgroup/Example/g1/cpu.weight
   # echo "100" > /sys/fs/cgroup/Example/g2/cpu.weight
   # echo "50" > /sys/fs/cgroup/Example/g3/cpu.weight

2. Add the applications' PIDs to the g1, g2, and g3 child groups:

   # echo "33373" > /sys/fs/cgroup/Example/g1/cgroup.procs
   # echo "33374" > /sys/fs/cgroup/Example/g2/cgroup.procs
   # echo "33377" > /sys/fs/cgroup/Example/g3/cgroup.procs

   The example commands ensure that desired applications become members of the Example/g*/ child cgroups and will get their CPU time distributed as per the configuration of those cgroups.

   The weights of the children cgroups (g1, g2, g3) that have running processes are summed up at the level of the parent cgroup (Example). The CPU resource is then distributed proportionally based on the respective weights.

   As a result, when all processes run at the same time, the kernel allocates to each of them the proportionate CPU time based on their respective cgroup’s cpu.weight file:

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~50% (150/300)
   g2	100	~33% (100/300)
   g3	50	~16% (50/300)

   The value of the cpu.weight controller file is not a percentage.

   If one process stopped running, leaving cgroup g2 with no running processes, the calculation would omit the cgroup g2 and only account weights of cgroups g1 and g3:

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~75% (150/200)
   g3	50	~25% (50/200)

   Important

   If a child cgroup had multiple running processes, the CPU time allocated to the respective cgroup would be distributed equally to the member processes of that cgroup.

Verification

1. Verify that the applications run in the specified control groups:

   # cat /proc/33373/cgroup /proc/33374/cgroup /proc/33377/cgroup
   0::/Example/g1
   0::/Example/g2
   0::/Example/g3

   The command output shows the processes of the specified applications that run in the Example/g*/ child cgroups.

2. Inspect the current CPU consumption of the throttled applications:

   # top
   top - 05:17:18 up 1 day, 18:25,  1 user,  load average: 3.03, 3.03, 3.00
   Tasks:  95 total,   4 running,  91 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 18.1 us, 81.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
   MiB Mem :   3737.0 total,   3233.7 free,    132.8 used,    370.5 buff/cache
   MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.   3373.1 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     33373 root      20   0   18720   1748   1460 R  49.5   0.0 415:05.87 sha1sum
     33374 root      20   0   18720   1756   1464 R  32.9   0.0 412:58.33 sha1sum
     33377 root      20   0   18720   1860   1568 R  16.3   0.0 411:03.12 sha1sum
       760 root      20   0  416620  28540  15296 S   0.3   0.7   0:10.23 tuned
         1 root      20   0  186328  14108   9484 S   0.0   0.4   0:02.00 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthread
   ...

   Note

   We forced all the example processes to run on a single CPU for clearer illustration. The CPU weight applies the same principles also when used on multiple CPUs.

   Notice that the CPU resource for the PID 33373, PID 33374, and PID 33377 was allocated based on the weights, 150, 100, 50, you assigned to the respective child cgroups. The weights correspond to around 50%, 33%, and 16% allocation of CPU time for each application.

Procedure

1. Configure the system to mount cgroups-v1 by default during system boot by the systemd system and service manager:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller"

   This adds the necessary kernel command-line parameters to the current boot entry.

   To add the same parameters to all kernel boot entries:

   # grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller"

2. Reboot the system for the changes to take effect.

Verification

1. Optionally, verify that the cgroups-v1 filesystem was mounted:

   # mount -l | grep cgroup
   tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755,inode64)
   cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
   cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
   cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
   cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
   cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
   cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
   cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
   cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
   cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
   cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
   cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,misc)
   cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
   cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)

   The cgroups-v1 filesystems that correspond to various cgroup-v1 controllers, were successfully mounted on the /sys/fs/cgroup/ directory.

2. Optionally, inspect the contents of the /sys/fs/cgroup/ directory:

   # ll /sys/fs/cgroup/
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 blkio
   lrwxrwxrwx.  1 root root 11 Mar 16 09:34 cpu → cpu,cpuacct
   lrwxrwxrwx.  1 root root 11 Mar 16 09:34 cpuacct → cpu,cpuacct
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 cpu,cpuacct
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 cpuset
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 devices
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 freezer
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 hugetlb
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 memory
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 misc
   lrwxrwxrwx.  1 root root 16 Mar 16 09:34 net_cls → net_cls,net_prio
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 net_cls,net_prio
   lrwxrwxrwx.  1 root root 16 Mar 16 09:34 net_prio → net_cls,net_prio
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 perf_event
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 pids
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 rdma
   dr-xr-xr-x. 11 root root  0 Mar 16 09:34 systemd

   The /sys/fs/cgroup/ directory, also called the root control group, by default, contains controller-specific directories such as cpuset. In addition, there are some directories related to systemd.

Procedure

1. Identify the process ID (PID) of the application you want to restrict in CPU consumption:

   # top
   top - 11:34:09 up 11 min,  1 user,  load average: 0.51, 0.27, 0.22
   Tasks: 267 total,   3 running, 264 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 49.0 us,  3.3 sy,  0.0 ni, 47.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
   MiB Mem :   1826.8 total,    303.4 free,   1046.8 used,    476.5 buff/cache
   MiB Swap:   1536.0 total,   1396.0 free,    140.0 used.    616.4 avail Mem
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    6955 root      20   0  228440   1752   1472 R  99.3   0.1   0:32.71 sha1sum
    5760 jdoe      20   0 3603868 205188  64196 S   3.7  11.0   0:17.19 gnome-shell
    6448 jdoe      20   0  743648  30640  19488 S   0.7   1.6   0:02.73 gnome-terminal-
       1 root      20   0  245300   6568   4116 S   0.3   0.4   0:01.87 systemd
     505 root      20   0       0      0      0 I   0.3   0.0   0:00.75 kworker/u4:4-events_unbound
   ...

   The example output of the top program reveals that PID 6955 (illustrative application sha1sum) consumes a lot of CPU resources.

2. Create a sub-directory in the cpu resource controller directory:

   # mkdir /sys/fs/cgroup/cpu/Example/

   The directory above represents a control group, where you can place specific processes and apply certain CPU limits to the processes. At the same time, some cgroups-v1 interface files and cpu controller-specific files will be created in the directory.

3. Optionally, inspect the newly created control group:

   # ll /sys/fs/cgroup/cpu/Example/
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cgroup.clone_children
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cgroup.procs
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.stat
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_all
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_sys
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_user
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_sys
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_user
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.cfs_period_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.cfs_quota_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.rt_period_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.rt_runtime_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.shares
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpu.stat
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 notify_on_release
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 tasks

   The example output shows files, such as cpuacct.usage, cpu.cfs._period_us, that represent specific configurations and/or limits, which can be set for processes in the Example control group. Notice that the respective file names are prefixed with the name of the control group controller to which they belong.

   By default, the newly created control group inherits access to the system’s entire CPU resources without a limit.

4. Configure CPU limits for the control group:

   # echo "1000000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us
   # echo "200000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us

   The cpu.cfs_period_us file represents a period of time in microseconds (µs, represented here as "us") for how frequently a control group’s access to CPU resources should be reallocated. The upper limit is 1 second and the lower limit is 1000 microseconds.

   The cpu.cfs_quota_us file represents the total amount of time in microseconds for which all processes collectively in a control group can run during one period (as defined by cpu.cfs_period_us). As soon as processes in a control group, during a single period, use up all the time specified by the quota, they are throttled for the remainder of the period and not allowed to run until the next period. The lower limit is 1000 microseconds.

   The example commands above set the CPU time limits so that all processes collectively in the Example control group will be able to run only for 0.2 seconds (defined by cpu.cfs_quota_us) out of every 1 second (defined by cpu.cfs_period_us).

5. Optionally, verify the limits:

   # cat /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us
   1000000
   200000

6. Add the application’s PID to the Example control group:

   # echo "6955" > /sys/fs/cgroup/cpu/Example/cgroup.procs
   
   or
   
   # echo "6955" > /sys/fs/cgroup/cpu/Example/tasks

   The previous command ensures that a desired application becomes a member of the Example control group and hence does not exceed the CPU limits configured for the Example control group. The PID should represent an existing process in the system. The PID 6955 here was assigned to process sha1sum /dev/zero &, used to illustrate the use-case of the cpu controller.

7. Verify that the application runs in the specified control group:

   # cat /proc/6955/cgroup
   12:cpuset:/
   11:hugetlb:/
   10:net_cls,net_prio:/
   9:memory:/user.slice/user-1000.slice/user@1000.service
   8:devices:/user.slice
   7:blkio:/
   6:freezer:/
   5:rdma:/
   4:pids:/user.slice/user-1000.slice/user@1000.service
   3:perf_event:/
   2:cpu,cpuacct:/Example
   1:name=systemd:/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service

   The example output above shows that the process of the desired application runs in the Example control group, which applies CPU limits to the application’s process.

8. Identify the current CPU consumption of your throttled application:

   # top
   top - 12:28:42 up  1:06,  1 user,  load average: 1.02, 1.02, 1.00
   Tasks: 266 total,   6 running, 260 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 11.0 us,  1.2 sy,  0.0 ni, 87.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.2 st
   MiB Mem :   1826.8 total,    287.1 free,   1054.4 used,    485.3 buff/cache
   MiB Swap:   1536.0 total,   1396.7 free,    139.2 used.    608.3 avail Mem
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    6955 root      20   0  228440   1752   1472 R  20.6   0.1  47:11.43 sha1sum
    5760 jdoe      20   0 3604956 208832  65316 R   2.3  11.2   0:43.50 gnome-shell
    6448 jdoe      20   0  743836  31736  19488 S   0.7   1.7   0:08.25 gnome-terminal-
     505 root      20   0       0      0      0 I   0.3   0.0   0:03.39 kworker/u4:4-events_unbound
    4217 root      20   0   74192   1612   1320 S   0.3   0.1   0:01.19 spice-vdagentd
   ...

   Notice that the CPU consumption of the PID 6955 has decreased from 99% to 20%.

Procedure

1. Install bcc-tools:

   # dnf install bcc-tools

   The BCC tools are installed in the /usr/share/bcc/tools/ directory.

2. Optionally, inspect the tools:

   # ll /usr/share/bcc/tools/
   ...
   -rwxr-xr-x. 1 root root  4198 Dec 14 17:53 dcsnoop
   -rwxr-xr-x. 1 root root  3931 Dec 14 17:53 dcstat
   -rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector
   -rw-r--r--. 1 root root  7105 Dec 14 17:53 deadlock_detector.c
   drwxr-xr-x. 3 root root  8192 Mar 11 10:28 doc
   -rwxr-xr-x. 1 root root  7588 Dec 14 17:53 execsnoop
   -rwxr-xr-x. 1 root root  6373 Dec 14 17:53 ext4dist
   -rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
   ...

   The doc directory in the listing above contains documentation for each tool.

1. Execute the execsnoop program in one terminal:

   # /usr/share/bcc/tools/execsnoop

2. In another terminal execute for example:

   $ ls /usr/share/bcc/tools/doc/

   The above creates a short-lived process of the ls command.

3. The terminal running execsnoop shows the output similar to the following:

   PCOMM	PID    PPID   RET ARGS
   ls   	8382   8287     0 /usr/bin/ls --color=auto /usr/share/bcc/tools/doc/
   ...

   The execsnoop program prints a line of output for each new process, which consumes system resources. It even detects processes of programs that run very shortly, such as ls, and most monitoring tools would not register them.

   The execsnoop output displays the following fields:

   • PCOMM - The parent process name. (ls)
   • PID - The process ID. (8382)
   • PPID - The parent process ID. (8287)
   • RET - The return value of the exec() system call (0), which loads program code into new processes.
   • ARGS - The location of the started program with arguments.

1. Execute the opensnoop program in one terminal:

   # /usr/share/bcc/tools/opensnoop -n uname

   The above prints output for files, which are opened only by the process of the uname command.

2. In another terminal execute:

   $ uname

   The command above opens certain files, which are captured in the next step.

3. The terminal running opensnoop shows the output similar to the following:

   PID    COMM 	FD ERR PATH
   8596   uname 	3  0   /etc/ld.so.cache
   8596   uname 	3  0   /lib64/libc.so.6
   8596   uname 	3  0   /usr/lib/locale/locale-archive
   ...

   The opensnoop program watches the open() system call across the whole system, and prints a line of output for each file that uname tried to open along the way.

   The opensnoop output displays the following fields:

   • PID - The process ID. (8596)
   • COMM - The process name. (uname)
   • FD - The file descriptor - a value that open() returns to refer to the open file. (3)
   • ERR - Any errors.

   • PATH - The location of files that open() tried to open.

     If a command tries to read a non-existent file, then the FD column returns -1 and the ERR column prints a value corresponding to the relevant error. As a result, opensnoop can help you identify an application that does not behave properly.

1. Execute the biotop program in one terminal:

   # /usr/share/bcc/tools/biotop 30

   The command enables you to monitor the top processes, which perform I/O operations on the disk. The argument ensures that the command will produce a 30 second summary.

   Note

   When no argument provided, the output screen by default refreshes every 1 second.

2. In another terminal execute for example :

   # dd if=/dev/vda of=/dev/zero

   The command above reads the content from the local hard disk device and writes the output to the /dev/zero file. This step generates certain I/O traffic to illustrate biotop.

3. The terminal running biotop shows the output similar to the following:

   PID    COMM             D MAJ MIN DISK       I/O  Kbytes     AVGms
   9568   dd               R 252 0   vda      16294 14440636.0  3.69
   48     kswapd0          W 252 0   vda       1763 120696.0    1.65
   7571   gnome-shell      R 252 0   vda        834 83612.0     0.33
   1891   gnome-shell      R 252 0   vda       1379 19792.0     0.15
   7515   Xorg             R 252 0   vda        280  9940.0     0.28
   7579   llvmpipe-1       R 252 0   vda        228  6928.0     0.19
   9515   gnome-control-c  R 252 0   vda         62  6444.0     0.43
   8112   gnome-terminal-  R 252 0   vda         67  2572.0     1.54
   7807   gnome-software   R 252 0   vda         31  2336.0     0.73
   9578   awk              R 252 0   vda         17  2228.0     0.66
   7578   llvmpipe-0       R 252 0   vda        156  2204.0     0.07
   9581   pgrep            R 252 0   vda         58  1748.0     0.42
   7531   InputThread      R 252 0   vda         30  1200.0     0.48
   7504   gdbus            R 252 0   vda          3  1164.0     0.30
   1983   llvmpipe-1       R 252 0   vda         39   724.0     0.08
   1982   llvmpipe-0       R 252 0   vda         36   652.0     0.06
   ...

   The biotop output displays the following fields:

   • PID - The process ID. (9568)
   • COMM - The process name. (dd)
   • DISK - The disk performing the read operations. (vda)
   • I/O - The number of read operations performed. (16294)
   • Kbytes - The amount of Kbytes reached by the read operations. (14,440,636)
   • AVGms - The average I/O time of read operations. (3.69)

1. Execute the xfsslower program in one terminal:

   # /usr/share/bcc/tools/xfsslower 1

   The command above measures the time the XFS file system spends in performing read, write, open or sync (fsync) operations. The 1 argument ensures that the program shows only the operations that are slower than 1 ms.

   Note

   When no arguments provided, xfsslower by default displays operations slower than 10 ms.

2. In another terminal execute, for example, the following:

   $ vim text

   The command above creates a text file in the vim editor to initiate certain interaction with the XFS file system.

3. The terminal running xfsslower shows something similar upon saving the file from the previous step:

   TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
   13:07:14 b'bash'        4754   R 256     0           7.11 b'vim'
   13:07:14 b'vim'         4754   R 832     0           4.03 b'libgpm.so.2.1.0'
   13:07:14 b'vim'         4754   R 32      20          1.04 b'libgpm.so.2.1.0'
   13:07:14 b'vim'         4754   R 1982    0           2.30 b'vimrc'
   13:07:14 b'vim'         4754   R 1393    0           2.52 b'getscriptPlugin.vim'
   13:07:45 b'vim'         4754   S 0       0           6.71 b'text'
   13:07:45 b'pool'        2588   R 16      0           5.58 b'text'
   ...

   Each line above represents an operation in the file system, which took more time than a certain threshold. xfsslower is good at exposing possible file system problems, which can take form of unexpectedly slow operations.

   The xfsslower output displays the following fields:

   • COMM - The process name. (b’bash')

   • T - The operation type. (R)

     • Read
     • Write
     • Sync
   • OFF_KB - The file offset in KB. (0)
   • FILENAME - The file being read, written, or synced.

Procedure

1. To create a trusted key using a TPM, execute:

   # keyctl add trusted <name> "new <key_length> [options]" <key_ring>

   • Based on the syntax, construct an example command as follows:

     # keyctl add trusted kmk "new 32" @u
     642500861

     The command creates a trusted key called kmk with the length of 32 bytes (256 bits) and places it in the user keyring (@u). The keys may have a length of 32 to 128 bytes (256 to 1024 bits).

2. To list the current structure of the kernel keyrings:

   # keyctl show
   Session Keyring
          -3 --alswrv    500   500  keyring: ses 97833714 --alswrv 500 -1 \ keyring: uid.1000 642500861 --alswrv 500 500 \ trusted: kmk

3. To export the key to a user-space blob, execute:

   # keyctl pipe 642500861 > kmk.blob

   The command uses the pipe subcommand and the serial number of kmk.

4. To load the trusted key from the user-space blob, use the add subcommand with the blob as an argument:

   # keyctl add trusted kmk "load `cat kmk.blob`" @u
   268728824

5. Create secure encrypted keys based on the TPM-sealed trusted key:

   # keyctl add encrypted <pass:quotes[name]> "new [format] <pass:quotes[key_type]>:<pass:quotes[primary_key_name]> <pass:quotes[keylength]>" <pass:quotes[key_ring]>

   • Based on the syntax, generate an encrypted key using the already created trusted key:

     # keyctl add encrypted encr-key "new trusted:kmk 32" @u
     159771175

     The command uses the TPM-sealed trusted key (kmk), produced in the previous step, as a primary key for generating encrypted keys.

Procedure

1. Use a random sequence of numbers to generate a user key:

   # keyctl add user kmk-user "$(dd if=/dev/urandom bs=1 count=32 2>/dev/null)" @u
   427069434

   The command generates a user key called kmk-user which acts as a primary key and is used to seal the actual encrypted keys.

2. Generate an encrypted key using the primary key from the previous step:

   # keyctl add encrypted encr-key "new user:kmk-user 32" @u
   1012412758

3. Optionally, list all keys in the specified user keyring:

   # keyctl list @u
   2 keys in keyring:
   427069434: --alswrv  1000  1000 user: kmk-user
   1012412758: --alswrv  1000  1000 encrypted: encr-key

Procedure

1. Add the following kernel command line parameters:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="ima_policy=appraise_tcb ima_appraise=fix evm=fix"

   The command enables IMA and EVM in the fix mode for the current boot entry and allows users to gather and update the IMA measurements.

   The ima_policy=appraise_tcb kernel command line parameter ensures that the kernel uses the default Trusted Computing Base (TCB) measurement policy and the appraisal step. The appraisal part forbids access to files, whose prior and current measures do not match.

2. Reboot to make the changes come into effect.

3. Optionally, verify that the parameters have been added to the kernel command line:

   # cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-1.el9.x86_64 root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet ima_policy=appraise_tcb ima_appraise=fix evm=fix

4. Create a kernel master key to protect the EVM key:

   # keyctl add user kmk "$(dd if=/dev/urandom bs=1 count=32 2> /dev/null)" @u
   748544121

   The kernel master key (kmk) is kept entirely in the kernel space memory. The 32-byte long value of the kernel master key kmk is generated from random bytes from the /dev/urandom file and placed in the user (@u) keyring. The key serial number is on the second line of the previous output.

5. Create an encrypted EVM key based on the kmk key:

   # keyctl add encrypted evm-key "new user:kmk 64" @u
   641780271

   The command uses kmk to generate and encrypt a 64-byte long user key (named evm-key) and places it in the user (@u) keyring. The key serial number is on the second line of the previous output.

   Important

   It is necessary to name the user key as evm-key because that is the name the EVM subsystem is expecting and is working with.

6. Create a directory for exported keys.

   # mkdir -p /etc/keys/

7. Search for the kmk key and export its value into a file:

   # keyctl pipe $(keyctl search @u user kmk) > /etc/keys/kmk

   The command places the unencrypted value of the kernel master key (kmk) into a file of previously defined location (/etc/keys/).

8. Search for the evm-key user key and export its value into a file:

   # keyctl pipe $(keyctl search @u encrypted evm-key) > /etc/keys/evm-key

   The command places the encrypted value of the user evm-key key into a file of arbitrary location. The evm-key has been encrypted by the kernel master key earlier.

9. Optionally, view the newly created keys:

   # keyctl show
   Session Keyring
   974575405   --alswrv     0        0      keyring: ses 299489774 --alswrv 0 65534 \ keyring: uid.0 748544121 --alswrv 0 0 \ user: kmk
   641780271   --alswrv     0        0           \_ encrypted: evm-key
   
   # ls -l /etc/keys/
   total 8
   -rw-r—​r--. 1 root root 246 Jun 24 12:44 evm-key
   -rw-r—​r--. 1 root root  32 Jun 24 12:43 kmk

   You should be able to see a similar output.

10. Activate EVM:

    # echo 1 > /sys/kernel/security/evm

Procedure

1. Create a test file:

   # echo <Test_text> > test_file

   IMA and EVM ensure that the test_file example file has assigned hash values that are stored as its extended attributes.

2. Inspect extended attributes of the file:

   # getfattr -m . -d test_file
   # file: test_file
   security.evm=0sAnDIy4VPA0HArpPO/EqiutnNyBql
   security.ima=0sAQOEDeuUnWzwwKYk+n66h/vby3eD
   security.selinux="unconfined_u:object_r:admin_home_t:s0"

   The example output shows extended attributes related to SELinux and the IMA and EVM hash values. EVM adds a security.evm extended attribute related to the other attributes. You can use the evmctl utility on security.evm to generate either an RSA based digital signature or a Hash-based Message Authentication Code (HMAC-SHA1). To successfully conduct the previous operation you need a valid trusted or encrypted key, which is stored in the kernel keyring. This configuration prevents offline tampering attacks on the extended attributes.

Procedure

1. Optionally, review the inventory file for illustration purposes:

   #  cat /home/jdoe/<ansible_project_name>/inventory
   [testingservers]
   pdoe@192.168.122.98
   fdoe@192.168.122.226
   
   [db-servers]
   db1.example.com
   db2.example.com
   
   [webservers]
   web1.example.com
   web2.example.com
   192.0.2.42

   The file defines the [testingservers] group and other groups. It allows you to run Ansible more effectively against a specific set of systems.

2. Create a configuration file to set defaults and privilege escalation for Ansible operations.

   1. Create a new YAML file and open it in a text editor, for example:

      #  vi /home/jdoe/<ansible_project_name>/ansible.cfg

   2. Insert the following content into the file:

      [defaults]
      inventory = ./inventory
      
      [privilege_escalation]
      become = true
      become_method = sudo
      become_user = root
      become_ask_pass = true

      The [defaults] section specifies a path to the inventory file of managed hosts. The [privilege_escalation] section defines that user privileges be shifted to root on the specified managed hosts. This is necessary for successful configuration of kernel parameters. When Ansible playbook is run, you will be prompted for user password. The user automatically switches to root by means of sudo after connecting to a managed host.

3. Create an Ansible playbook that uses the Kernel Settings role.

   1. Create a new YAML file and open it in a text editor, for example:

      #  vi /home/jdoe/<ansible_project_name>/kernel-roles.yml

      This file represents a playbook and usually contains an ordered list of tasks, also called plays, that are run against specific managed hosts selected from your inventory file.

   2. Insert the following content into the file:

      ---
      -
        hosts: testingservers
        name: "Configure kernel settings"
        roles:
          - rhel-system-roles.kernel_settings
        vars:
          kernel_settings_sysctl:
            - name: fs.file-max
              value: 400000
            - name: kernel.threads-max
              value: 65536
          kernel_settings_sysfs:
            - name: /sys/class/net/lo/mtu
              value: 65000
          kernel_settings_transparent_hugepages: madvise

      The name key is optional. It associates an arbitrary string with the play as a label and identifies what the play is for. The hosts key in the play specifies the hosts against which the play is run. The value or values for this key can be provided as individual names of managed hosts or as groups of hosts as defined in the inventory file.

      The vars section represents a list of variables containing selected kernel parameter names and values to which they have to be set.

      The roles key specifies what system role is going to configure the parameters and values mentioned in the vars section.

      Note

      You can modify the kernel parameters and their values in the playbook to fit your needs.

4. Optionally, verify that the syntax in your play is correct.

   #  ansible-playbook --syntax-check kernel-roles.yml
   
   playbook: kernel-roles.yml

   This example shows the successful verification of a playbook.

5. Execute your playbook.

   #  ansible-playbook kernel-roles.yml
   
   ...
   
   BECOME password:
   
   PLAY [Configure kernel settings] **********************************************************************************
   
   
   
   PLAY RECAP ********************************************************************************************************
   fdoe@192.168.122.226       : ok=10   changed=4    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
   pdoe@192.168.122.98        : ok=10   changed=4    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0

   Before Ansible runs your playbook, you are going to be prompted for your password and so that a user on managed hosts can be switched to root, which is necessary for configuring kernel parameters.

   The recap section shows that the play finished successfully (failed=0) for all managed hosts, and that 4 kernel parameters have been applied (changed=4).

6. Restart your managed hosts and check the affected kernel parameters to verify that the changes have been applied and persist across reboots.