2.13. System Partitioning

One of the key techniques for real-time tuning is partitioning the system. This means isolating a group of CPU cores for exclusive use of one or more real-time applications running on the system. For best results, partitioning should take into account the CPU topology so that related threads are placed on cores contained on the same Non-Uniform Memory Access (NUMA) node to maximize sharing of second and third-level caches. The lscpu and tuna utilities are used to determine the system CPU topology. The Tuna GUI allows you to dynamically isolate CPUs and move threads and interrupts from one CPU to another so that performance impacts can be measured.

Once a partitioning strategy has been determined based on the system layout and the structure of the application, the next step is to set the system to be partitioned automatically upon boot. For that, use the utilities provided by the tuned-profiles-realtime package. This package is installed by default when the Red Hat Enterprise Linux for Real Time packages are installed. To install tuned-profiles-realtime manually, run the following command as root:

~]# yum install tuned-profiles-realtime

The tuned-profiles-realtime package provides the tuned real-time profile that allows partitioning and other tunings at boot time with no additional user input required. Two configuration files control the behavior of the profile:

The realtime-variables.conf file specifies the group of CPU cores to be isolated. To isolate a group of CPU cores from the system, use the isolated_cores option as in the following example:

# Examples:
# isolated_cores=2,4-7
# isolated_cores=2-23
#
isolated_cores=1-3,5,9-14

In the example above, the profile places the CPUs 1, 2, 3, 5, 9, 10, 11, 12, 13, and 14 into an isolated CPU category; the only threads on these CPUs are kernel threads specifically bound to the cores. These kernel threads are only run when a specific condition is raised, such as the migration thread or the watchdog thread.

Once the isolated_cores variable is set, activate the profile with the tuned-adm command:

~]# tuned-adm profile realtime

The profile uses the bootloader plug-in. When activated, this plug-in adds the following boot parameters to the Linux kernel command line:

In the above example, the kernel boot command-line parameters look as follows:

isolcpus=1-3,5,9-14 nohz=on nohz_full=1-3,5,9-14 intel_pstate=disable nosoftlockup

The profile runs the script.sh shell script specified in the [script] section of tuned.conf. The script adjusts the following entries of the sysfs virtual file system:

The workqueue entry above is set to the inverse of the isolated CPUs mask, while the second entry turns off machine check exceptions.

The script also sets the following variables in the /etc/sysctl.conf file:

kernel.hung_task_timeout_secs = 600
kernel.nmi_watchdog = 0
kernel.sched_rt_runtime_us = 1000000
vm.stat_interval = 10

The script uses the tuna interface to move any non-bound thread on the isolated CPU numbers off of the isolated CPUs.

For further tuning, copy the default /usr/lib/tuned/realtime/script.sh and modify it, then change the tuned.conf JSON file to point to the modified script.