Procedure

1. To update the kernel, enter the following command:

   # dnf update kernel

   This command updates the kernel along with all dependencies to the latest available version.

2. Reboot your system for the changes to take effect.

Procedure

1. Select a kernel module you want to load.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Load the relevant kernel module:

   # modprobe <MODULE_NAME>

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

3. Optionally, verify the relevant module was loaded:

   $ lsmod | grep <MODULE_NAME>

   If the module was loaded correctly, this command displays the relevant kernel module. For example:

   $ lsmod | grep serio_raw
   serio_raw              16384  0

Procedure

1. Enter the lsmod command and select a kernel module you want to unload.

   If a kernel module has dependencies, unload those prior to unloading the kernel module. For details on identifying modules with dependencies, see Listing currently loaded kernel modules and Kernel module dependencies.

2. Unload the relevant kernel module:

   # modprobe -r <MODULE_NAME>

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

   Warning

   Do not unload kernel modules when they are used by the running system. Doing so can lead to an unstable or non-operational system.

3. Optionally, verify the relevant module was unloaded:

   $ lsmod | grep <MODULE_NAME>

   If the module was unloaded successfully, this command does not display any output.

Procedure

1. Select a kernel module you want to load during the boot process.

   The modules are located in the /lib/modules/$(uname -r)/kernel/<SUBSYSTEM>/ directory.

2. Create a configuration file for the module:

   # echo <MODULE_NAME> > /etc/modules-load.d/<MODULE_NAME>.conf

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

3. Optionally, after reboot, verify the relevant module was loaded:

   $ lsmod | grep <MODULE_NAME>

   The example command above should succeed and display the relevant kernel module.

Procedure

1. List modules loaded to the currently running kernel by using the lsmod command:

   $ lsmod
   Module                  Size  Used by
   tls                   131072  0
   uinput                 20480  1
   snd_seq_dummy          16384  0
   snd_hrtimer            16384  1
   …

   In the output, identify the module you want to prevent from being loaded.

   • Alternatively, identify an unloaded kernel module you want to prevent from potentially loading in the /lib/modules/<KERNEL-VERSION>/kernel/<SUBSYSTEM>/ directory, for example:

     $ ls /lib/modules/4.18.0-477.20.1.el8_8.x86_64/kernel/crypto/
     ansi_cprng.ko.xz        chacha20poly1305.ko.xz  md4.ko.xz               serpent_generic.ko.xz
     anubis.ko.xz            cmac.ko.xz…

2. Create a configuration file serving as a denylist:

   # touch /etc/modprobe.d/denylist.conf

3. In a text editor of your choice, combine the names of modules you want to exclude from automatic loading to the kernel with the blacklist configuration command, for example:

   # Prevents <KERNEL-MODULE-1> from being loaded
   blacklist <MODULE-NAME-1>
   install <MODULE-NAME-1> /bin/false
   
   # Prevents <KERNEL-MODULE-2> from being loaded
   blacklist <MODULE-NAME-2>
   install <MODULE-NAME-2> /bin/false
   …

   Because the blacklist command does not prevent the module from being loaded as a dependency for another kernel module that is not in a denylist, you must also define the install line. In this case, the system runs /bin/false instead of installing the module. The lines starting with a hash sign are comments you can use to make the file more readable.

   Note

   When entering the name of a kernel module, do not append the .ko.xz extension to the end of the name. Kernel module names do not have extensions; their corresponding files do.

4. Create a backup copy of the current initial RAM disk image before rebuilding:

   # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img

   • Alternatively, create a backup copy of an initial RAM disk image which corresponds to the kernel version for which you want to prevent kernel modules from automatic loading:

     # cp /boot/initramfs-<VERSION>.img /boot/initramfs-<VERSION>.img.bak.$(date +%m-%d-%H%M%S)

5. Generate a new initial RAM disk image to apply the changes:

   # dracut -f -v

   • If you build an initial RAM disk image for a different kernel version than your system currently uses, specify both target initramfs and kernel version:

     # dracut -f -v /boot/initramfs-<TARGET-VERSION>.img <CORRESPONDING-TARGET-KERNEL-VERSION>

6. Restart the system:

   $ reboot

Procedure

1. Create the /root/testmodule/test.c file with the following content.

   #include <linux/module.h>
   #include <linux/kernel.h>
   
   int init_module(void)
       { printk("Hello World\n This is a test\n"); return 0; }
   
   void cleanup_module(void)
       { printk("Good Bye World"); }
   
   MODULE_LICENSE("GPL");

   The test.c file is a source file that provides the main functionality to the kernel module. The file has been created in a dedicated /root/testmodule/ directory for organizational purposes. After the module compilation, the /root/testmodule/ directory will contain multiple files.

   The test.c file includes from the system libraries:

   • The linux/kernel.h header file is necessary for the printk() function in the example code.

   • The linux/module.h file contains function declarations and macro definitions to be shared between several source files written in C programming language.

     Next follow the init_module() and cleanup_module() functions to start and end the kernel logging function printk(), which prints text.

2. Create the /root/testmodule/Makefile file with the following content.

   obj-m := test.o

   The Makefile contains instructions that the compiler has to produce an object file specifically named test.o. The obj-m directive specifies that the resulting test.ko file is going to be compiled as a loadable kernel module. Alternatively, the obj-y directive would instruct to build test.ko as a built-in kernel module.

3. Compile the kernel module.

   # make -C /lib/modules/$(uname -r)/build M=/root/testmodule modules
   make: Entering directory '/usr/src/kernels/5.14.0-70.17.1.el9_0.x86_64'
     CC [M]  /root/testmodule/test.o
     MODPOST /root/testmodule/Module.symvers
     CC [M]  /root/testmodule/test.mod.o
     LD [M]  /root/testmodule/test.ko
     BTF [M] /root/testmodule/test.ko
   Skipping BTF generation for /root/testmodule/test.ko due to unavailability of vmlinux
   make: Leaving directory '/usr/src/kernels/5.14.0-70.17.1.el9_0.x86_64'

   The compiler creates an object file (test.o) for each source file (test.c) as an intermediate step before linking them together into the final kernel module (test.ko).

   After a successful compilation, /root/testmodule/ contains additional files that relate to the compiled custom kernel module. The compiled module itself is represented by the test.ko file.

Verification

1. Optional: check the contents of the /root/testmodule/ directory:

   # ls -l /root/testmodule/
   total 152
   -rw-r—​r--. 1 root root    16 Jul 26 08:19 Makefile
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 modules.order
   -rw-r—​r--. 1 root root     0 Jul 26 08:20 Module.symvers
   -rw-r—​r--. 1 root root   224 Jul 26 08:18 test.c
   -rw-r—​r--. 1 root root 62176 Jul 26 08:20 test.ko
   -rw-r—​r--. 1 root root    25 Jul 26 08:20 test.mod
   -rw-r—​r--. 1 root root   849 Jul 26 08:20 test.mod.c
   -rw-r—​r--. 1 root root 50936 Jul 26 08:20 test.mod.o
   -rw-r—​r--. 1 root root 12912 Jul 26 08:20 test.o

2. Copy the kernel module to the /lib/modules/$(uname -r)/ directory:

   # cp /root/testmodule/test.ko /lib/modules/$(uname -r)/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v test
   insmod /lib/modules/5.14.0-1.el9.x86_64/test.ko

5. Verify that the kernel module was successfully loaded:

   # lsmod | grep test
   test                   16384  0

6. Read the latest messages from the kernel ring buffer:

   # dmesg
   [74422.545004] Hello World
                   This is a test

Procedure

1. Select the kernel you want to start when the GRUB 2 boot menu appears and press the e key to edit the kernel parameters.
2. Find the kernel command line by moving the cursor down. The kernel command line starts with linux on 64-Bit IBM Power Series and x86-64 BIOS-based systems, or linuxefi on UEFI systems.

3. Move the cursor to the end of the line.

   Note

   Press Ctrl+a to jump to the start of the line and Ctrl+e to jump to the end of the line. On some systems, Home and End keys might also work.

4. Edit the kernel parameters as required. For example, to run the system in emergency mode, add the emergency parameter at the end of the linux line:

1. Press Ctrl+x to boot with the selected kernel and the modified command line parameters.

Procedure

1. Add the following two lines to the /etc/default/grub file:

   GRUB_TERMINAL="serial"
   GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

   The first line disables the graphical terminal. The GRUB_TERMINAL key overrides values of GRUB_TERMINAL_INPUT and GRUB_TERMINAL_OUTPUT keys.

   The second line adjusts the baud rate (--speed), parity and other values to fit your environment and hardware. Note that a much higher baud rate, for example 115200, is preferable for tasks such as following log files.

2. Update the GRUB configuration file.

   • On BIOS-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

   • On UEFI-based machines:

     # grub2-mkconfig -o /boot/grub2/grub.cfg

3. Reboot the system for the changes to take effect.

Procedure

1. List all parameters and their values.

   # sysctl -a

   Note

   The # sysctl -a command displays kernel parameters, which can be adjusted at runtime and at boot time.

2. To configure a parameter temporarily, enter:

   # sysctl <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

   The sample command above changes the parameter value while the system is running. The changes take effect immediately, without a need for restart.

   Note

   The changes return back to default after your system reboots.

Procedure

1. List all parameters.

   # sysctl -a

   The command displays all kernel parameters that can be configured at runtime.

2. Configure a parameter permanently:

   # sysctl -w <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE> >> /etc/sysctl.conf

   The sample command changes the tunable value and writes it to the /etc/sysctl.conf file, which overrides the default values of kernel parameters. The changes take effect immediately and persistently, without a need for restart.

Procedure

1. Create a new configuration file in /etc/sysctl.d/.

   # vim /etc/sysctl.d/<some_file.conf>

2. Include kernel parameters, one per line.

   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>
   <TUNABLE_CLASS>.<PARAMETER>=<TARGET_VALUE>

3. Save the configuration file.

4. Reboot the machine for the changes to take effect.

   • Alternatively, to apply changes without rebooting, enter:

     # sysctl -p /etc/sysctl.d/<some_file.conf>

     The command enables you to read values from the configuration file, which you created earlier.

Procedure

1. Identify a kernel parameter you want to configure.

   # ls -l /proc/sys/<TUNABLE_CLASS>/

   The writable files returned by the command can be used to configure the kernel. The files with read-only permissions provide feedback on the current settings.

2. Assign a target value to the kernel parameter.

   # echo <TARGET_VALUE> > /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

   The command makes configuration changes that will disappear once the system is restarted.

3. Optionally, verify the value of the newly set kernel parameter.

   # cat /proc/sys/<TUNABLE_CLASS>/<PARAMETER>

Procedure

1. Optionally, review the inventory file for illustration purposes:

   #  cat /home/jdoe/<ansible_project_name>/inventory
   [testingservers]
   pdoe@192.168.122.98
   fdoe@192.168.122.226
   
   [db-servers]
   db1.example.com
   db2.example.com
   
   [webservers]
   web1.example.com
   web2.example.com
   192.0.2.42

   The file defines the [testingservers] group and other groups. It allows you to run Ansible more effectively against a specific set of systems.

2. Create a configuration file to set defaults and privilege escalation for Ansible operations.

   1. Create a new YAML file and open it in a text editor, for example:

      #  vi /home/jdoe/<ansible_project_name>/ansible.cfg

   2. Insert the following content into the file:

      [defaults]
      inventory = ./inventory
      
      [privilege_escalation]
      become = true
      become_method = sudo
      become_user = root
      become_ask_pass = true

      The [defaults] section specifies a path to the inventory file of managed hosts. The [privilege_escalation] section defines that user privileges be shifted to root on the specified managed hosts. This is necessary for successful configuration of kernel parameters. When Ansible playbook is run, you will be prompted for user password. The user automatically switches to root by means of sudo after connecting to a managed host.

3. Create an Ansible playbook that uses the kernel_settings role.

   1. Create a new YAML file and open it in a text editor, for example:

      #  vi /home/jdoe/<ansible_project_name>/kernel-roles.yml

      This file represents a playbook and usually contains an ordered list of tasks, also called plays, that are run against specific managed hosts selected from your inventory file.

   2. Insert the following content into the file:

      ---
      -
        hosts: testingservers
        name: "Configure kernel settings"
        roles:
          - rhel-system-roles.kernel_settings
        vars:
          kernel_settings_sysctl:
            - name: fs.file-max
              value: 400000
            - name: kernel.threads-max
              value: 65536
          kernel_settings_sysfs:
            - name: /sys/class/net/lo/mtu
              value: 65000
          kernel_settings_transparent_hugepages: madvise

      The name key is optional. It associates an arbitrary string with the play as a label and identifies what the play is for. The hosts key in the play specifies the hosts against which the play is run. The value or values for this key can be provided as individual names of managed hosts or as groups of hosts as defined in the inventory file.

      The vars section represents a list of variables containing selected kernel parameter names and values to which they have to be set.

      The roles key specifies what system role is going to configure the parameters and values mentioned in the vars section.

      Note

      You can modify the kernel parameters and their values in the playbook to fit your needs.

4. Optionally, verify that the syntax in your play is correct.

   #  ansible-playbook --syntax-check kernel-roles.yml
   
   playbook: kernel-roles.yml

   This example shows the successful verification of a playbook.

5. Execute your playbook.

   #  ansible-playbook kernel-roles.yml
   
   ...
   
   BECOME password:
   
   PLAY [Configure kernel settings] **********************************************************************************
   
   
   
   PLAY RECAP ********************************************************************************************************
   fdoe@192.168.122.226       : ok=10   changed=4    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
   pdoe@192.168.122.98        : ok=10   changed=4    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0

   Before Ansible runs your playbook, you are going to be prompted for your password and so that a user on managed hosts can be switched to root, which is necessary for configuring kernel parameters.

   The recap section shows that the play finished successfully (failed=0) for all managed hosts, and that 4 kernel parameters have been applied (changed=4).

6. Restart your managed hosts and check the affected kernel parameters to verify that the changes have been applied and persist across reboots.

1. The kernel patch module is copied to the /var/lib/kpatch/ directory and registered for re-application to the kernel by systemd on next boot.
2. The kpatch module is loaded into the running kernel and the new functions are registered to the ftrace mechanism with a pointer to the location in memory of the new code.
3. When the kernel accesses the patched function, it is redirected by the ftrace mechanism which bypasses the original functions and redirects the kernel to patched version of the function.

Procedure

1. Optionally, check your kernel version:

   # uname -r
   5.14.0-1.el9.x86_64

2. Search for a live patching package that corresponds to the version of your kernel:

   # dnf search $(uname -r)

3. Install the live patching package:

   # dnf install "kpatch-patch = $(uname -r)"

   The command above installs and applies the latest cumulative live patches for that specific kernel only.

   If the version of a live patching package is 1-1 or higher, the package will contain a patch module. In that case the kernel will be automatically patched during the installation of the live patching package.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during the future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-5_14_0-1-0-0.x86_64.rpm. The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optionally, check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            5.14.0-1.el9              @beaker-BaseOS
   kernel-core.x86_64            5.14.0-2.el9              @@commandline
   ...
   
   # uname -r
   5.14.0-2.el9.x86_64

2. Install the kpatch-dnf plugin:

   # dnf install kpatch-dnf

3. Enable automatic subscription to kernel live patches:

   # dnf kpatch auto
   Updating Subscription Management repositories.
   Last metadata expiration check: 1:38:21 ago on Fri 17 Sep 2021 07:29:53 AM EDT.
   Dependencies resolved.
   ==================================================
    Package                             Architecture
   ==================================================
   Installing:
    kpatch-patch-5_14_0-1               x86_64
    kpatch-patch-5_14_0-2               x86_64
   
   Transaction Summary
   ===================================================
   Install  2 Packages
   …​

   This command subscribes all currently installed kernels to receiving kernel live patches. The command also installs and applies the latest cumulative live patches, if any, for all installed kernels.

   In the future, when you update the kernel, live patches will automatically be installed during the new kernel installation process.

   The kernel patch module is also installed into the /var/lib/kpatch/ directory to be loaded by the systemd system and service manager during future reboots.

   Note

   An empty live patching package will be installed when there are no live patches available for a given kernel. An empty live patching package will have a kpatch_version-kpatch_release of 0-0, for example kpatch-patch-5_14_0-1-0-0.el9.x86_64.rpm .The installation of the empty RPM subscribes the system to all future live patches for the given kernel.

Procedure

1. Optionally, check all installed kernels and the kernel you are currently running:

   # dnf list installed | grep kernel
   Updating Subscription Management repositories.
   Installed Packages
   ...
   kernel-core.x86_64            5.14.0-1.el9              @beaker-BaseOS
   kernel-core.x86_64            5.14.0-2.el9              @@commandline
   ...
   
   # uname -r
   5.14.0-2.el9.x86_64

2. Disable automatic subscription to kernel live patches:

   # dnf kpatch manual
   Updating Subscription Management repositories.

Procedure

1. Select the live patching package.

   # dnf list installed | grep kpatch-patch
   kpatch-patch-5_14_0-1.x86_64        0-1.el9        @@commandline
   …​

   The example output above lists live patching packages that you installed.

2. Remove the live patching package.

   # dnf remove kpatch-patch-5_14_0-1.x86_64

   When a live patching package is removed, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk. On future reboot, the corresponding kernel will no longer be patched.

3. Reboot your system.

4. Verify that the live patching package has been removed.

   # dnf list installed | grep kpatch-patch

   The command displays no output if the package has been successfully removed.

5. Optionally, verify that the kernel live patching solution is disabled.

   # kpatch list
   Loaded patch modules:

   The example output shows that the kernel is not patched and the live patching solution is not active because there are no patch modules that are currently loaded.

Procedure

1. Select a kernel patch module:

   # kpatch list
   Loaded patch modules:
   kpatch_5_14_0_1_0_1 [enabled]
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)
   …​

2. Uninstall the selected kernel patch module.

   # kpatch uninstall kpatch_5_14_0_1_0_1
   uninstalling kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

   • Note that the uninstalled kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_5_14_0_1_0_1 [enabled]
     
     Installed patch modules:
     <NO_RESULT>

     When the selected module is uninstalled, the kernel remains patched until the next reboot, but the kernel patch module is removed from disk.

3. Reboot your system.

4. Optionally, verify that the kernel patch module has been uninstalled.

   # kpatch list
   Loaded patch modules:
   …​

   The example output above shows no loaded or installed kernel patch modules, therefore the kernel is not patched and the kernel live patching solution is not active.

Procedure

1. Verify kpatch.service is enabled.

   # systemctl is-enabled kpatch.service
   enabled

2. Disable kpatch.service:

   # systemctl disable kpatch.service
   Removed /etc/systemd/system/multi-user.target.wants/kpatch.service.

   • Note that the applied kernel patch module is still loaded:

     # kpatch list
     Loaded patch modules:
     kpatch_5_14_0_1_0_1 [enabled]
     
     Installed patch modules:
     kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

3. Reboot your system.

4. Optionally, verify the status of kpatch.service.

   # systemctl status kpatch.service
   ● kpatch.service - "Apply kpatch kernel patches"
      Loaded: loaded (/usr/lib/systemd/system/kpatch.service; disabled; vendor preset: disabled)
      Active: inactive (dead)

   The example output testifies that kpatch.service has been disabled and is not running. Thereby, the kernel live patching solution is not active.

5. Verify that the kernel patch module has been unloaded.

   # kpatch list
   Loaded patch modules:
   
   Installed patch modules:
   kpatch_5_14_0_1_0_1 (5.14.0-1.el9.x86_64)

   The example output above shows that a kernel patch module is still installed but the kernel is not patched.

1. value. Console log-level, defines the lowest priority of messages printed to the console.
2. value. Default log-level for messages without an explicit log-level attached to them.
3. value. Sets the lowest possible log-level configuration for the console log-level.

4. value. Sets default value for the console log-level at boot time.

   Each of these values above defines a different rule for handling error messages.

Procedure

1. Export your public key to the sb_cert.cer file:

   # certutil -d /etc/pki/pesign \
              -n 'Custom Secure Boot key' \
              -Lr \
              > sb_cert.cer

2. Import your public key into the MOK list:

   # mokutil --import sb_cert.cer

3. Enter a new password for this MOK enrollment request.

4. Reboot the machine.

   The shim boot loader notices the pending MOK key enrollment request and it launches MokManager.efi to enable you to complete the enrollment from the UEFI console.

5. Choose Enroll MOK, enter the password you previously associated with this request when prompted, and confirm the enrollment.

   Your public key is added to the MOK list, which is persistent.

   Once a key is on the MOK list, it will be automatically propagated to the .platform keyring on this and subsequent boots when UEFI Secure Boot is enabled.

Procedure

1. Export your public key to the sb_cert.cer file:

   # certutil -d /etc/pki/pesign \
              -n 'Custom Secure Boot key' \
              -Lr \
              > sb_cert.cer

2. Extract the key from the NSS database as a PKCS #12 file:

   # pk12util -o sb_cert.p12 \
              -n 'Custom Secure Boot key' \
              -d /etc/pki/pesign

3. When the previous command prompts you, enter a new password that encrypts the private key.

4. Export the unencrypted private key:

   # openssl pkcs12 \
            -in sb_cert.p12 \
            -out sb_cert.priv \
            -nocerts \
            -noenc

   Important

   Handle the unencrypted private key with care.

5. Sign your kernel module. The following command appends the signature directly to the ELF image in your kernel module file:

   # /usr/src/kernels/$(uname -r)/scripts/sign-file \
             sha256 \
             sb_cert.priv \
             sb_cert.cer \
             my_module.ko

Verification

1. Display information about the kernel module’s signature:

   # modinfo my_module.ko | grep signer
     signer:         Your Name Key

   Check that the signature lists your name as entered during generation.

   Note

   The appended signature is not contained in an ELF image section and is not a formal part of the ELF image. Therefore, utilities such as readelf cannot display the signature on your kernel module.

2. Load the module:

   # insmod my_module.ko

3. Remove (unload) the module:

   # modprobe -r my_module.ko

Procedure

1. Verify that your public keys are on the system keyring:

   # keyctl list %:.platform

2. Copy the kernel module into the extra/ directory of the kernel that you want:

   # cp my_module.ko /lib/modules/$(uname -r)/extra/

3. Update the modular dependency list:

   # depmod -a

4. Load the kernel module:

   # modprobe -v my_module

5. Optionally, to load the module on boot, add it to the /etc/modules-loaded.d/my_module.conf file:

   # echo "my_module" > /etc/modules-load.d/my_module.conf

Procedure

1. Determine the current version of the Revocation List:

   # fwupdmgr get-devices

   See the Current version field under UEFI dbx.

2. Enable the LVFS Revocation List repository:

   # fwupdmgr enable-remote lvfs

3. Refresh the repository metadata:

   # fwupdmgr refresh

4. Apply the Revocation List update:

   • On the command line:

     # fwupdmgr update

   • In the graphical interface:

     1. Open the Software application
     2. Navigate to the Updates tab.
     3. Find the Secure Boot dbx Configuration Update entry.
     4. Click Update.
5. At the end of the update, fwupdmgr or Software asks you to reboot the system. Confirm the reboot.

Procedure

1. Determine the current version of the Revocation List:

   # fwupdmgr get-devices

   See the Current version field under UEFI dbx.

2. List the updates available from RHEL:

   # ls /usr/share/dbxtool/

3. Select the most recent update file for your architecture. The file names use the following format:

   DBXUpdate-date-architecture.cab

4. Install the selected update file:

   # fwupdmgr install /usr/share/dbxtool/DBXUpdate-date-architecture.cab

5. At the end of the update, fwupdmgr asks you to reboot the system. Confirm the reboot.

Procedure

1. Create a 2048-bit RSA key with an SHA-256 primary storage key with a persistent handle of, for example, 81000001, by using one of the following utilities:

   1. By using the tss2 package:

      # TPM_DEVICE=/dev/tpm0 tsscreateprimary -hi o -st
      Handle 80000000
      # TPM_DEVICE=/dev/tpm0 tssevictcontrol -hi o -ho 80000000 -hp 81000001

   2. By using the tpm2-tools package:

      # tpm2_createprimary --key-algorithm=rsa2048 --key-context=key.ctxt
      name-alg:
        value: sha256
        raw: 0xb
      …
      sym-keybits: 128
      rsa: xxxxxx…
      
      # tpm2_evictcontrol -c key.ctxt 0x81000001
      persistentHandle: 0x81000001
      action: persisted

2. Create a trusted key by using a TPM 2.0 with the syntax of keyctl add trusted <NAME> "new <KEY_LENGTH> keyhandle=<PERSISTENT-HANDLE> [options]" <KEYRING>. In this example, the persistent handle is 81000001.

   # keyctl add trusted kmk "new 32 keyhandle=0x81000001" @u
   642500861

   The command creates a trusted key called kmk with the length of 32 bytes (256 bits) and places it in the user keyring (@u). The keys may have a length of 32 to 128 bytes (256 to 1024 bits).

3. List the current structure of the kernel keyrings:

   # keyctl show
   Session Keyring
          -3 --alswrv    500   500  keyring: ses 97833714 --alswrv 500 -1 \ keyring: uid.1000 642500861 --alswrv 500 500 \ trusted: kmk

4. Export the key to a user-space blob by using the serial number of the trusted key:

   # keyctl pipe 642500861 > kmk.blob

   The command uses the pipe subcommand and the serial number of kmk.

5. Load the trusted key from the user-space blob:

   # keyctl add trusted kmk "load `cat kmk.blob`" @u
   268728824

6. Create secure encrypted keys that use the TPM-sealed trusted key (kmk). Follow this syntax: keyctl add encrypted <NAME> "new [FORMAT] <KEY_TYPE>:<PRIMARY_KEY_NAME> <KEY_LENGTH>" <KEYRING>:

   # keyctl add encrypted encr-key "new trusted:kmk 32" @u
   159771175

Procedure

1. Generate a user key by using a random sequence of numbers.

   # keyctl add user kmk-user "$(dd if=/dev/urandom bs=1 count=32 2>/dev/null)" @u
   427069434

   The command generates a user key called kmk-user which acts as a primary key and is used to seal the actual encrypted keys.

2. Generate an encrypted key using the primary key from the previous step:

   # keyctl add encrypted encr-key "new user:kmk-user 32" @u
   1012412758

3. Optionally, list all keys in the specified user keyring:

   # keyctl list @u
   2 keys in keyring:
   427069434: --alswrv  1000  1000 user: kmk-user
   1012412758: --alswrv  1000  1000 encrypted: encr-key

Procedure

1. Enable IMA and EVM in the fix mode for the current boot entry and allow users to gather and update the IMA measurements by adding the following kernel command-line parameters:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="ima_policy=appraise_tcb ima_appraise=fix evm=fix"

   The command enables IMA and EVM in the fix mode for the current boot entry and allows users to gather and update the IMA measurements.

   The ima_policy=appraise_tcb kernel command-line parameter ensures that the kernel uses the default Trusted Computing Base (TCB) measurement policy and the appraisal step. The appraisal step forbids access to files whose prior and current measures do not match.

2. Reboot to make the changes come into effect.

3. Optional: Verify that the parameters have been added to the kernel command line:

   # cat /proc/cmdline
   BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-1.el9.x86_64 root=/dev/mapper/rhel-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet ima_policy=appraise_tcb ima_appraise=fix evm=fix

4. Create a kernel master key to protect the EVM key:

   # keyctl add user kmk "$(dd if=/dev/urandom bs=1 count=32 2> /dev/null)" @u
   748544121

   The kmk is kept entirely in the kernel space memory. The 32-byte long value of the kmk is generated from random bytes from the /dev/urandom file and placed in the user (@u) keyring. The key serial number is on the first line of the previous output.

5. Create an encrypted EVM key based on the kmk:

   # keyctl add encrypted evm-key "new user:kmk 64" @u
   641780271

   The command uses the kmk to generate and encrypt a 64-byte long user key (named evm-key) and places it in the user (@u) keyring. The key serial number is on the first line of the previous output.

   Important

   It is necessary to name the user key as evm-key because that is the name the EVM subsystem is expecting and is working with.

6. Create a directory for exported keys.

   # mkdir -p /etc/keys/

7. Search for the kmk and export its unencrypted value into the new directory.

   # keyctl pipe $(keyctl search @u user kmk) > /etc/keys/kmk

8. Search for the evm-key and export its encrypted value into the new directory.

   # keyctl pipe $(keyctl search @u encrypted evm-key) > /etc/keys/evm-key

   The evm-key has been encrypted by the kernel master key earlier.

9. Optional: View the newly created keys.

   # keyctl show
   Session Keyring
   974575405   --alswrv     0        0      keyring: ses 299489774 --alswrv 0 65534 \ keyring: uid.0 748544121 --alswrv 0 0 \ user: kmk
   641780271   --alswrv     0        0           \_ encrypted: evm-key
   
   # ls -l /etc/keys/
   total 8
   -rw-r--r--. 1 root root 246 Jun 24 12:44 evm-key
   -rw-r--r--. 1 root root  32 Jun 24 12:43 kmk

10. Optional: If the keys have been removed from the keyring, for example after system reboot, you can import the already exported kmk and evm-key instead of creating new ones.

    1. Import the kmk.

       # keyctl add user kmk "$(cat /etc/keys/kmk)" @u
       451342217

    2. Import the evm-key.

       # keyctl add encrypted evm-key "load $(cat /etc/keys/evm-key)" @u
       924537557

11. Activate EVM.

    # echo 1 > /sys/kernel/security/evm

12. Relabel the whole system.

    # find / -fstype xfs -type f -uid 0 -exec head -n 1 '{}' >/dev/null \;

    Warning

    Enabling IMA and EVM without relabeling the system might make the majority of the files on the system inaccessible.

Procedure

1. Create a test file:

   # echo <Test_text> > test_file

   IMA and EVM ensure that the test_file example file has assigned hash values that are stored as its extended attributes.

2. Inspect the file’s extended attributes:

   # getfattr -m . -d test_file
   # file: test_file
   security.evm=0sAnDIy4VPA0HArpPO/EqiutnNyBql
   security.ima=0sAQOEDeuUnWzwwKYk+n66h/vby3eD

   The example output shows extended attributes with the IMA and EVM hash values and SELinux context. EVM adds a security.evm extended attribute related to the other attributes. At this point, you can use the evmctl utility on security.evm to generate either an RSA-based digital signature or a Hash-based Message Authentication Code (HMAC-SHA1).

1. Check the assigned values for the service of your choice.

   # systemctl show --property <unit file option> <service name>

2. Set the required value of the CPU time allocation policy option:

   # systemctl set-property <service name> <unit file option>=<value>

Procedure

1. To view which cgroup a process belongs to, run the # cat proc/<PID>/cgroup command:

   # cat /proc/2467/cgroup
   0::/system.slice/example.service

   The example output relates to a process of interest. In this case, it is a process identified by PID 2467, which belongs to the example.service unit. You can determine whether the process was placed in a correct control group as defined by the systemd unit file specifications.

2. To display what controllers the cgroup utilizes and the respective configuration files, check the cgroup directory:

   # cat /sys/fs/cgroup/system.slice/example.service/cgroup.controllers
   memory pids
   
   # ls /sys/fs/cgroup/system.slice/example.service/
   cgroup.controllers
   cgroup.events
   …​
   cpu.pressure
   cpu.stat
   io.pressure
   memory.current
   memory.events
   …​
   pids.current
   pids.events
   pids.max

Procedure

1. Display a dynamic account of currently running cgroups with the systemd-cgtop command.

   # systemd-cgtop
   Control Group                            Tasks   %CPU   Memory  Input/s Output/s
   /                                          607   29.8     1.5G        -        -
   /system.slice                              125      -   428.7M        -        -
   /system.slice/ModemManager.service           3      -     8.6M        -        -
   /system.slice/NetworkManager.service         3      -    12.8M        -        -
   /system.slice/accounts-daemon.service        3      -     1.8M        -        -
   /system.slice/boot.mount                     -      -    48.0K        -        -
   /system.slice/chronyd.service                1      -     2.0M        -        -
   /system.slice/cockpit.socket                 -      -     1.3M        -        -
   /system.slice/colord.service                 3      -     3.5M        -        -
   /system.slice/crond.service                  1      -     1.8M        -        -
   /system.slice/cups.service                   1      -     3.1M        -        -
   /system.slice/dev-hugepages.mount            -      -   244.0K        -        -
   /system.slice/dev-mapper-rhel\x2dswap.swap   -      -   912.0K        -        -
   /system.slice/dev-mqueue.mount               -      -    48.0K        -        -
   /system.slice/example.service                2      -     2.0M        -        -
   /system.slice/firewalld.service              2      -    28.8M        -        -
   ...

   The example output displays currently running cgroups ordered by their resource usage (CPU, memory, disk I/O load). The list refreshes every 1 second by default. Therefore, it offers a dynamic insight into the actual resource usage of each control group.

Procedure

1. Modify the /usr/lib/systemd/system/example.service file to limit the memory usage of a service:

   …​
   [Service]
   MemoryMax=1500K
   …​

   The configuration above places a maximum memory limit, which the processes in a control group cannot exceed. The example.service service is part of such a control group which has imposed limitations. You can use suffixes K, M, G, or T to identify Kilobyte, Megabyte, Gigabyte, or Terabyte as a unit of measurement.

2. Reload all unit configuration files:

   # systemctl daemon-reload

3. Restart the service:

   # systemctl restart example.service

Verification

1. Check that the changes took effect:

   # cat /sys/fs/cgroup/system.slice/example.service/memory.max
   1536000

   The example output shows that the memory consumption was limited at around 1,500 KB.

1. Check the values of the CPUAffinity unit file option in the service of your choice:

   $ systemctl show --property <CPU affinity configuration option> <service name>

2. As a root, set the required value of the CPUAffinity unit file option for the CPU ranges used as the affinity mask:

   # systemctl set-property <service name> CPUAffinity=<value>

3. Restart the service to apply the changes.

   # systemctl restart <service name>

1. Set the CPU numbers for the CPUAffinity= option in the /etc/systemd/system.conf file.

2. Save the edited file and reload the systemd service:

   # systemctl daemon-reload

3. Reboot the server to apply the changes.

1. Check the values of the NUMAPolicy unit file option in the service of your choice:

   $ systemctl show --property <NUMA policy configuration option> <service name>

2. As a root, set the required policy type of the NUMAPolicy unit file option:

   # systemctl set-property <service name> NUMAPolicy=<value>

3. Restart the service to apply the changes.

   # systemctl restart <service name>

1. Search in the /etc/systemd/system.conf file for the NUMAPolicy option.
2. Edit the policy type and save the file.

3. Reload the systemd configuration:

   # systemd daemon-reload

4. Reboot the server.

Procedure

1. Create the /sys/fs/cgroup/Example/ directory:

   # mkdir /sys/fs/cgroup/Example/

   The /sys/fs/cgroup/Example/ directory defines a child group. When you create the /sys/fs/cgroup/Example/ directory, some cgroups-v2 interface files are automatically created in the directory. The /sys/fs/cgroup/Example/ directory contains also controller-specific files for the memory and pids controllers.

2. Optionally, inspect the newly created child control group:

   # ll /sys/fs/cgroup/Example/
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.controllers
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 cgroup.events
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.freeze
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.procs
   …​
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 cgroup.subtree_control
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 memory.events.local
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 memory.high
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 memory.low
   …​
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 pids.current
   -r—​r—​r--. 1 root root 0 Jun  1 10:33 pids.events
   -rw-r—​r--. 1 root root 0 Jun  1 10:33 pids.max

   The example output shows general cgroup control interface files such as cgroup.procs or cgroup.controllers. These files are common to all control groups, regardless of enabled controllers.

   The files such as memory.high and pids.max relate to the memory and pids controllers, which are in the root control group (/sys/fs/cgroup/), and are enabled by default by systemd.

   By default, the newly created child group inherits all settings from the parent cgroup. In this case, there are no limits from the root cgroup.

3. Verify that the desired controllers are available in the /sys/fs/cgroup/cgroup.controllers file:

   # cat /sys/fs/cgroup/cgroup.controllers
   cpuset cpu io memory hugetlb pids rdma

4. Enable the desired controllers. In this example it is cpu and cpuset controllers:

   # echo "+cpu" >> /sys/fs/cgroup/cgroup.subtree_control
   # echo "+cpuset" >> /sys/fs/cgroup/cgroup.subtree_control

   These commands enable the cpu and cpuset controllers for the immediate child groups of the /sys/fs/cgroup/ root control group. Including the newly created Example control group. A child group is where you can specify processes and apply control checks to each of the processes based on your criteria.

   Users can read the contents of the cgroup.subtree_control file at any level to get an idea of what controllers are going to be available for enablement in the immediate child group.

   Note

   By default, the /sys/fs/cgroup/cgroup.subtree_control file in the root control group contains memory and pids controllers.

5. Enable the desired controllers for child cgroups of the Example control group:

   # echo "+cpu +cpuset" >> /sys/fs/cgroup/Example/cgroup.subtree_control

   This command ensures that the immediate child control group will only have controllers relevant to regulate the CPU time distribution - not to memory or pids controllers.

6. Create the /sys/fs/cgroup/Example/tasks/ directory:

   # mkdir /sys/fs/cgroup/Example/tasks/

   The /sys/fs/cgroup/Example/tasks/ directory defines a child group with files that relate purely to cpu and cpuset controllers. You can now assign processes to this control group and utilize cpu and cpuset controller options for your processes.

7. Optionally, inspect the child control group:

   # ll /sys/fs/cgroup/Example/tasks
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.controllers
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.events
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.freeze
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.max.depth
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.max.descendants
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.procs
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cgroup.stat
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.subtree_control
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.threads
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cgroup.type
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.max
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.pressure
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.effective
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.cpus.partition
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpuset.mems
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpuset.mems.effective
   -r—​r—​r--. 1 root root 0 Jun  1 11:45 cpu.stat
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.weight
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 cpu.weight.nice
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 io.pressure
   -rw-r—​r--. 1 root root 0 Jun  1 11:45 memory.pressure

Procedure

1. Configure desired CPU weights to achieve resource restrictions within the control groups:

   # echo "150" > /sys/fs/cgroup/Example/g1/cpu.weight
   # echo "100" > /sys/fs/cgroup/Example/g2/cpu.weight
   # echo "50" > /sys/fs/cgroup/Example/g3/cpu.weight

2. Add the applications' PIDs to the g1, g2, and g3 child groups:

   # echo "33373" > /sys/fs/cgroup/Example/g1/cgroup.procs
   # echo "33374" > /sys/fs/cgroup/Example/g2/cgroup.procs
   # echo "33377" > /sys/fs/cgroup/Example/g3/cgroup.procs

   The example commands ensure that desired applications become members of the Example/g*/ child cgroups and will get their CPU time distributed as per the configuration of those cgroups.

   The weights of the children cgroups (g1, g2, g3) that have running processes are summed up at the level of the parent cgroup (Example). The CPU resource is then distributed proportionally based on the respective weights.

   As a result, when all processes run at the same time, the kernel allocates to each of them the proportionate CPU time based on their respective cgroup’s cpu.weight file:

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~50% (150/300)
   g2	100	~33% (100/300)
   g3	50	~16% (50/300)

   The value of the cpu.weight controller file is not a percentage.

   If one process stopped running, leaving cgroup g2 with no running processes, the calculation would omit the cgroup g2 and only account weights of cgroups g1 and g3:

   Child cgroup	cpu.weight file	CPU time allocation
   g1	150	~75% (150/200)
   g3	50	~25% (50/200)

   Important

   If a child cgroup had multiple running processes, the CPU time allocated to the respective cgroup would be distributed equally to the member processes of that cgroup.

Verification

1. Verify that the applications run in the specified control groups:

   # cat /proc/33373/cgroup /proc/33374/cgroup /proc/33377/cgroup
   0::/Example/g1
   0::/Example/g2
   0::/Example/g3

   The command output shows the processes of the specified applications that run in the Example/g*/ child cgroups.

2. Inspect the current CPU consumption of the throttled applications:

   # top
   top - 05:17:18 up 1 day, 18:25,  1 user,  load average: 3.03, 3.03, 3.00
   Tasks:  95 total,   4 running,  91 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 18.1 us, 81.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st
   MiB Mem :   3737.0 total,   3233.7 free,    132.8 used,    370.5 buff/cache
   MiB Swap:   4060.0 total,   4060.0 free,      0.0 used.   3373.1 avail Mem
   
       PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     33373 root      20   0   18720   1748   1460 R  49.5   0.0 415:05.87 sha1sum
     33374 root      20   0   18720   1756   1464 R  32.9   0.0 412:58.33 sha1sum
     33377 root      20   0   18720   1860   1568 R  16.3   0.0 411:03.12 sha1sum
       760 root      20   0  416620  28540  15296 S   0.3   0.7   0:10.23 tuned
         1 root      20   0  186328  14108   9484 S   0.0   0.4   0:02.00 systemd
         2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthread
   ...

   Note

   We forced all the example processes to run on a single CPU for clearer illustration. The CPU weight applies the same principles also when used on multiple CPUs.

   Notice that the CPU resource for the PID 33373, PID 33374, and PID 33377 was allocated based on the weights, 150, 100, 50, you assigned to the respective child cgroups. The weights correspond to around 50%, 33%, and 16% allocation of CPU time for each application.

Procedure

1. Configure the system to mount cgroups-v1 by default during system boot by the systemd system and service manager:

   # grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller"

   This adds the necessary kernel command-line parameters to the current boot entry.

   To add the same parameters to all kernel boot entries:

   # grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller"

2. Reboot the system for the changes to take effect.

Verification

1. Optionally, verify that the cgroups-v1 filesystem was mounted:

   # mount -l | grep cgroup
   tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,size=4096k,nr_inodes=1024,mode=755,inode64)
   cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
   cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
   cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpu,cpuacct)
   cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
   cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
   cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_cls,net_prio)
   cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
   cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
   cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
   cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
   cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,misc)
   cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
   cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,rdma)

   The cgroups-v1 filesystems that correspond to various cgroup-v1 controllers, were successfully mounted on the /sys/fs/cgroup/ directory.

2. Optionally, inspect the contents of the /sys/fs/cgroup/ directory:

   # ll /sys/fs/cgroup/
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 blkio
   lrwxrwxrwx.  1 root root 11 Mar 16 09:34 cpu → cpu,cpuacct
   lrwxrwxrwx.  1 root root 11 Mar 16 09:34 cpuacct → cpu,cpuacct
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 cpu,cpuacct
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 cpuset
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 devices
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 freezer
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 hugetlb
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 memory
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 misc
   lrwxrwxrwx.  1 root root 16 Mar 16 09:34 net_cls → net_cls,net_prio
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 net_cls,net_prio
   lrwxrwxrwx.  1 root root 16 Mar 16 09:34 net_prio → net_cls,net_prio
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 perf_event
   dr-xr-xr-x. 10 root root  0 Mar 16 09:34 pids
   dr-xr-xr-x.  2 root root  0 Mar 16 09:34 rdma
   dr-xr-xr-x. 11 root root  0 Mar 16 09:34 systemd

   The /sys/fs/cgroup/ directory, also called the root control group, by default, contains controller-specific directories such as cpuset. In addition, there are some directories related to systemd.

Procedure

1. Identify the process ID (PID) of the application that you want to restrict in CPU consumption:

   # top
   top - 11:34:09 up 11 min,  1 user,  load average: 0.51, 0.27, 0.22
   Tasks: 267 total,   3 running, 264 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 49.0 us,  3.3 sy,  0.0 ni, 47.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.0 st
   MiB Mem :   1826.8 total,    303.4 free,   1046.8 used,    476.5 buff/cache
   MiB Swap:   1536.0 total,   1396.0 free,    140.0 used.    616.4 avail Mem
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    6955 root      20   0  228440   1752   1472 R  99.3   0.1   0:32.71 sha1sum
    5760 jdoe      20   0 3603868 205188  64196 S   3.7  11.0   0:17.19 gnome-shell
    6448 jdoe      20   0  743648  30640  19488 S   0.7   1.6   0:02.73 gnome-terminal-
       1 root      20   0  245300   6568   4116 S   0.3   0.4   0:01.87 systemd
     505 root      20   0       0      0      0 I   0.3   0.0   0:00.75 kworker/u4:4-events_unbound
   ...

   This example output of the top program reveals that illustrative application sha1sum with PID 6955 consumes a lot of CPU resources.

2. Create a sub-directory in the cpu resource controller directory:

   # mkdir /sys/fs/cgroup/cpu/Example/

   This directory represents a control group, where you can place specific processes and apply certain CPU limits to the processes. At the same time, a number of cgroups-v1 interface files and cpu controller-specific files will be created in the directory.

3. Optional: Inspect the newly created control group:

   # ll /sys/fs/cgroup/cpu/Example/
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cgroup.clone_children
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cgroup.procs
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.stat
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_all
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_sys
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_percpu_user
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_sys
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpuacct.usage_user
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.cfs_period_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.cfs_quota_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.rt_period_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.rt_runtime_us
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 cpu.shares
   -r—​r—​r--. 1 root root 0 Mar 11 11:42 cpu.stat
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 notify_on_release
   -rw-r—​r--. 1 root root 0 Mar 11 11:42 tasks

   This example output shows files, such as cpuacct.usage, cpu.cfs._period_us, that represent specific configurations and/or limits, which can be set for processes in the Example control group. Note that the respective file names are prefixed with the name of the control group controller to which they belong.

   By default, the newly created control group inherits access to the system’s entire CPU resources without a limit.

4. Configure CPU limits for the control group:

   # echo "1000000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us
   # echo "200000" > /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us

   • The cpu.cfs_period_us file represents a period of time in microseconds (µs, represented here as "us") for how frequently a control group’s access to CPU resources should be reallocated. The upper limit is 1 000 000 microsecond and the lower limit is 1 000 microseconds.

   • The cpu.cfs_quota_us file represents the total amount of time in microseconds for which all processes collectively in a control group can run during one period (as defined by cpu.cfs_period_us). When processes in a control group, during a single period, use up all the time specified by the quota, they are throttled for the remainder of the period and not allowed to run until the next period. The lower limit is 1 000 microseconds.

     The example commands above set the CPU time limits so that all processes collectively in the Example control group will be able to run only for 0.2 seconds (defined by cpu.cfs_quota_us) out of every 1 second (defined by cpu.cfs_period_us).

5. Optional: Verify the limits:

   # cat /sys/fs/cgroup/cpu/Example/cpu.cfs_period_us /sys/fs/cgroup/cpu/Example/cpu.cfs_quota_us
   1000000
   200000

6. Add the application’s PID to the Example control group:

   # echo "6955" > /sys/fs/cgroup/cpu/Example/cgroup.procs

   This command ensures that a specific application becomes a member of the Example control group and hence does not exceed the CPU limits configured for the Example control group. The PID should represent an existing process in the system. The PID 6955 here was assigned to process sha1sum /dev/zero &, used to illustrate the use case of the cpu controller.

Verification

1. Verify that the application runs in the specified control group:

   # cat /proc/6955/cgroup
   12:cpuset:/
   11:hugetlb:/
   10:net_cls,net_prio:/
   9:memory:/user.slice/user-1000.slice/user@1000.service
   8:devices:/user.slice
   7:blkio:/
   6:freezer:/
   5:rdma:/
   4:pids:/user.slice/user-1000.slice/user@1000.service
   3:perf_event:/
   2:cpu,cpuacct:/Example
   1:name=systemd:/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service

   This example output shows that the process of the desired application runs in the Example control group, which applies CPU limits to the application’s process.

2. Identify the current CPU consumption of your throttled application:

   # top
   top - 12:28:42 up  1:06,  1 user,  load average: 1.02, 1.02, 1.00
   Tasks: 266 total,   6 running, 260 sleeping,   0 stopped,   0 zombie
   %Cpu(s): 11.0 us,  1.2 sy,  0.0 ni, 87.5 id,  0.0 wa,  0.2 hi,  0.0 si,  0.2 st
   MiB Mem :   1826.8 total,    287.1 free,   1054.4 used,    485.3 buff/cache
   MiB Swap:   1536.0 total,   1396.7 free,    139.2 used.    608.3 avail Mem
   
     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    6955 root      20   0  228440   1752   1472 R  20.6   0.1  47:11.43 sha1sum
    5760 jdoe      20   0 3604956 208832  65316 R   2.3  11.2   0:43.50 gnome-shell
    6448 jdoe      20   0  743836  31736  19488 S   0.7   1.7   0:08.25 gnome-terminal-
     505 root      20   0       0      0      0 I   0.3   0.0   0:03.39 kworker/u4:4-events_unbound
    4217 root      20   0   74192   1612   1320 S   0.3   0.1   0:01.19 spice-vdagentd
   ...

   Note that the CPU consumption of the PID 6955 has decreased from 99% to 20%.

Procedure

1. Install bcc-tools.

   # dnf install bcc-tools

   The BCC tools are installed in the /usr/share/bcc/tools/ directory.

2. Optionally, inspect the tools:

   # ll /usr/share/bcc/tools/
   ...
   -rwxr-xr-x. 1 root root  4198 Dec 14 17:53 dcsnoop
   -rwxr-xr-x. 1 root root  3931 Dec 14 17:53 dcstat
   -rwxr-xr-x. 1 root root 20040 Dec 14 17:53 deadlock_detector
   -rw-r--r--. 1 root root  7105 Dec 14 17:53 deadlock_detector.c
   drwxr-xr-x. 3 root root  8192 Mar 11 10:28 doc
   -rwxr-xr-x. 1 root root  7588 Dec 14 17:53 execsnoop
   -rwxr-xr-x. 1 root root  6373 Dec 14 17:53 ext4dist
   -rwxr-xr-x. 1 root root 10401 Dec 14 17:53 ext4slower
   ...

   The doc directory in the listing above contains documentation for each tool.

1. Run the execsnoop program in one terminal:

   # /usr/share/bcc/tools/execsnoop

2. In another terminal run, for example:

   $ ls /usr/share/bcc/tools/doc/

   The above creates a short-lived process of the ls command.

3. The terminal running execsnoop shows the output similar to the following:

   PCOMM	PID    PPID   RET ARGS
   ls   	8382   8287     0 /usr/bin/ls --color=auto /usr/share/bcc/tools/doc/
   ...

   The execsnoop program prints a line of output for each new process, which consumes system resources. It even detects processes of programs that run very shortly, such as ls, and most monitoring tools would not register them.

   The execsnoop output displays the following fields:

   • PCOMM - The parent process name. (ls)
   • PID - The process ID. (8382)
   • PPID - The parent process ID. (8287)
   • RET - The return value of the exec() system call (0), which loads program code into new processes.
   • ARGS - The location of the started program with arguments.

1. Run the opensnoop program in one terminal:

   # /usr/share/bcc/tools/opensnoop -n uname

   The above prints output for files, which are opened only by the process of the uname command.

2. In another terminal, enter:

   $ uname

   The command above opens certain files, which are captured in the next step.

3. The terminal running opensnoop shows the output similar to the following:

   PID    COMM 	FD ERR PATH
   8596   uname 	3  0   /etc/ld.so.cache
   8596   uname 	3  0   /lib64/libc.so.6
   8596   uname 	3  0   /usr/lib/locale/locale-archive
   ...

   The opensnoop program watches the open() system call across the whole system, and prints a line of output for each file that uname tried to open along the way.

   The opensnoop output displays the following fields:

   • PID - The process ID. (8596)
   • COMM - The process name. (uname)
   • FD - The file descriptor - a value that open() returns to refer to the open file. (3)
   • ERR - Any errors.

   • PATH - The location of files that open() tried to open.

     If a command tries to read a non-existent file, then the FD column returns -1 and the ERR column prints a value corresponding to the relevant error. As a result, opensnoop can help you identify an application that does not behave properly.

1. Run the biotop program in one terminal:

   # /usr/share/bcc/tools/biotop 30

   The command enables you to monitor the top processes, which perform I/O operations on the disk. The argument ensures that the command will produce a 30 second summary.

   Note

   When no argument provided, the output screen by default refreshes every 1 second.

2. In another terminal enter, for example :

   # dd if=/dev/vda of=/dev/zero

   The command above reads the content from the local hard disk device and writes the output to the /dev/zero file. This step generates certain I/O traffic to illustrate biotop.

3. The terminal running biotop shows the output similar to the following:

   PID    COMM             D MAJ MIN DISK       I/O  Kbytes     AVGms
   9568   dd               R 252 0   vda      16294 14440636.0  3.69
   48     kswapd0          W 252 0   vda       1763 120696.0    1.65
   7571   gnome-shell      R 252 0   vda        834 83612.0     0.33
   1891   gnome-shell      R 252 0   vda       1379 19792.0     0.15
   7515   Xorg             R 252 0   vda        280  9940.0     0.28
   7579   llvmpipe-1       R 252 0   vda        228  6928.0     0.19
   9515   gnome-control-c  R 252 0   vda         62  6444.0     0.43
   8112   gnome-terminal-  R 252 0   vda         67  2572.0     1.54
   7807   gnome-software   R 252 0   vda         31  2336.0     0.73
   9578   awk              R 252 0   vda         17  2228.0     0.66
   7578   llvmpipe-0       R 252 0   vda        156  2204.0     0.07
   9581   pgrep            R 252 0   vda         58  1748.0     0.42
   7531   InputThread      R 252 0   vda         30  1200.0     0.48
   7504   gdbus            R 252 0   vda          3  1164.0     0.30
   1983   llvmpipe-1       R 252 0   vda         39   724.0     0.08
   1982   llvmpipe-0       R 252 0   vda         36   652.0     0.06
   ...

   The biotop output displays the following fields:

   • PID - The process ID. (9568)
   • COMM - The process name. (dd)
   • DISK - The disk performing the read operations. (vda)
   • I/O - The number of read operations performed. (16294)
   • Kbytes - The amount of Kbytes reached by the read operations. (14,440,636)
   • AVGms - The average I/O time of read operations. (3.69)

1. Run the xfsslower program in one terminal:

   # /usr/share/bcc/tools/xfsslower 1

   The command above measures the time the XFS file system spends in performing read, write, open or sync (fsync) operations. The 1 argument ensures that the program shows only the operations that are slower than 1 ms.

   Note

   When no arguments provided, xfsslower by default displays operations slower than 10 ms.

2. In another terminal enter, for example, the following:

   $ vim text

   The command above creates a text file in the vim editor to initiate certain interaction with the XFS file system.

3. The terminal running xfsslower shows something similar upon saving the file from the previous step:

   TIME     COMM           PID    T BYTES   OFF_KB   LAT(ms) FILENAME
   13:07:14 b'bash'        4754   R 256     0           7.11 b'vim'
   13:07:14 b'vim'         4754   R 832     0           4.03 b'libgpm.so.2.1.0'
   13:07:14 b'vim'         4754   R 32      20          1.04 b'libgpm.so.2.1.0'
   13:07:14 b'vim'         4754   R 1982    0           2.30 b'vimrc'
   13:07:14 b'vim'         4754   R 1393    0           2.52 b'getscriptPlugin.vim'
   13:07:45 b'vim'         4754   S 0       0           6.71 b'text'
   13:07:45 b'pool'        2588   R 16      0           5.58 b'text'
   ...

   Each line above represents an operation in the file system, which took more time than a certain threshold. xfsslower is good at exposing possible file system problems, which can take form of unexpectedly slow operations.

   The xfsslower output displays the following fields:

   • COMM - The process name. (b’bash')

   • T - The operation type. (R)

     • Read
     • Write
     • Sync
   • OFF_KB - The file offset in KB. (0)
   • FILENAME - The file being read, written, or synced.

Procedure

1. Go to the Kdump field.

2. Enable kdump if not already enabled.

   

3. Define how much memory should be reserved for kdump.

   

Procedure

1. Check whether kdump is installed on your system:

   # rpm -q kexec-tools

   Output if the package is installed:

   # kexec-tools-2.0.22-13.el9.x86_64

   Output if the package is not installed:

   package kexec-tools is not installed

2. Install kdump and other necessary packages by:

   # dnf install kexec-tools

Procedure

1. Configure the default value for crash kernel.

   # kdumpctl reset-crashkernel --kernel=ALL

   When configuring the crashkernel= value, test the configuration by rebooting with kdump enabled. If the kdump kernel fails to boot, increase the memory size gradually to set an acceptable value.

2. To use a custom crashkernel= value:

   1. Configure the required memory reserve.

      # crashkernel=192M

      Alternatively, you can set the amount of reserved memory to a variable depending on the total amount of installed memory using the syntax crashkernel=<range1>:<size1>,<range2>:<size2>. For example:

      # crashkernel=1G-4G:192M,2G-64G:256M

      The example reserves 192 MB of memory if the total amount of system memory is 1 GB or higher and lower than 4 GB. If the total amount of memory is more than 4 GB, 256 MB is reserved for kdump.

   2. (Optional) Offset the reserved memory.

      Some systems require to reserve memory with a certain fixed offset since crashkernel reservation is very early, and it wants to reserve some area for special usage. If the offset is set, the reserved memory begins there. To offset the reserved memory, use the following syntax:

      # crashkernel=192M@16M

      The example reserves 192 MB of memory starting at 16 MB (physical address 0x01000000). If you offset to 0 or do not specify a value, kdump offsets the reserved memory automatically. You can also offset memory when setting a variable memory reservation by specifying the offset as the last value. For example, crashkernel=1G-4G:192M,2G-64G:256M@16M.

   3. Update the bootloader configuration.

      # grubby --update-kernel ALL --args "crashkernel=<custom-value>"

      The <custom-value> must contain the custom crashkernel= value that you have configured for the crash kernel.

3. Reboot for changes to take effect.

   # reboot

1. Activate the sysrq key to boot into the kdump kernel.

   echo c > /proc/sysrq-trigger

   The command causes the kernel to crash and reboots the kernel if required.

2. Display the /etc/kdump.conf file and check if the vmcore file is saved in the target destination.

Procedure

1. As root user, edit the /etc/kdump.conf configuration file to remove the hash sign ("#") from the beginning of the following command:

   core_collector makedumpfile -z -d 31 --message-level 1

2. To enable dump file compression, specify one of the makedumpfile options:

   core_collector makedumpfile -z -d 31 --message-level 1

   where,

   • -z specifies the dump compressed file format.
   • -d specifies dump level as 31.
   • --message-level specifies message level as 1.

Procedure

1. As root, remove the hash sign ("#") from the beginning of the #failure_action line in the /etc/kdump.conf configuration file.

2. Replace the value with a desired action.

   failure_action poweroff

Procedure

1. To enable the kdump service, use the following command:

   # systemctl enable kdump.service

   This enables the service for multi-user.target.

2. To start the service in the current session, use the following command:

   # systemctl start kdump.service

3. To stop the kdump service, type the following command:

   # systemctl stop kdump.service

4. To disable the kdump service, execute the following command:

   # systemctl disable kdump.service

Procedure

1. Reboot the system with kdump enabled.

2. Check if kdump is active.

   # systemctl is-active kdump
   active

3. Force a kernel crash.

   echo c > /proc/sysrq-trigger

   Warning

   The command causes the kernel to crash and reboots the kernel if required.

   On a kernel reboot, the address-YYYY-MM-DD-HH:MM:SS/vmcore file is created at the location you have specified in the /etc/kdump.conf file (by default to /var/crash/).

Procedure

1. Select a kernel module that you intend to block from loading.

   $ lsmod
   
   Module                  Size  Used by
   fuse                  126976  3
   xt_CHECKSUM            16384  1
   ipt_MASQUERADE         16384  1
   uinput                 20480  1
   xt_conntrack           16384  1

   The lsmod command displays a list of modules that are loaded to the currently running kernel.

2. Update the KDUMP_COMMANDLINE_APPEND= variable in the /etc/sysconfig/kdump file.

   # KDUMP_COMMANDLINE_APPEND="rd.driver.blacklist=hv_vmbus,hv_storvsc,hv_utils,hv_netvsc,hid-hyperv"

   Also,consider the following example using the modprobe.blacklist=<modules> configuration option.

   # KDUMP_COMMANDLINE_APPEND="modprobe.blacklist=emcp modprobe.blacklist=bnx2fc modprobe.blacklist=libfcoe modprobe.blacklist=fcoe"

3. Restart the kdump service.

   # systemctl restart kdump

Procedure

1. Print the estimate crashkernel value:

   # kdumpctl estimate
   
   Encrypted kdump target requires extra memory, assuming using the keyslot with minimum memory requirement
      Reserved crashkernel:    256M
      Recommended crashkernel: 652M
      Kernel image size:   47M
      Kernel modules size: 8M
      Initramfs size:      20M
      Runtime reservation: 64M
      LUKS required size:  512M
      Large modules: none
      WARNING: Current crashkernel size is lower than recommended size 652M.

2. Configure the amount of required memory by increasing crashkernel to the desired value.

   # grubby –args=”crashkernel=652M” --update-kernel=ALL

3. Reboot for changes to take effect.

   # reboot

Procedure

1. Add the crashkernel=auto command-line parameter to all installed kernels:

   # grubby --update-kernel=ALL --args="crashkernel=auto"

2. Enable the kdump service.

   # systemctl enable --now kdump.service

Procedure

1. List the kernels installed on the machine.

   # ls -a /boot/vmlinuz-*
   /boot/vmlinuz-0-rescue-2930657cd0dc43c2b75db480e5e5b4a9 /boot/vmlinuz-4.18.0-330.el8.x86_64 /boot/vmlinuz-4.18.0-330.rt7.111.el8.x86_64

2. Add a specific kdump kernel to the system’s Grand Unified Bootloader (GRUB) configuration file.

   For example:

   # grubby --update-kernel=vmlinuz-4.18.0-330.el8.x86_64 --args="crashkernel=auto"

3. Enable the kdump service.

   # systemctl enable --now kdump.service

Procedure

1. To stop the kdump service in the current session:

   # systemctl stop kdump.service

2. To disable the kdump service:

   # systemctl disable kdump.service

Procedure

1. To configure final_action, edit the /etc/kdump.conf file and add one of the following options:

   # final_action <reboot | halt | poweroff>

2. Restart the kdump service for the changes to take effect:

   # kdumpctl restart

Procedure:

1. To configure an action to take if the dump fails, edit the /etc/kdump.conf file and specify one of the failure_action options:

   # failure_action <reboot | halt | poweroff | shell | dump_to_rootfs>

2. Restart the kdump service for the changes to take effect:

   # kdumpctl restart