One of the key techniques for real-time tuning is partitioning the system. This means isolating a group of CPU cores for exclusive use of one or more real-time applications running on the system. For best results, partitioning should take into account the CPU topology so that related threads are placed on cores contained on the same Non-Uniform Memory Access (NUMA) node to maximize sharing of second and third-level caches. The
tuna utilities are used to determine the system CPU topology. The Tuna GUI allows you to dynamically isolate CPUs and move threads and interrupts from one CPU to another so that performance impacts can be measured.
Once a partitioning strategy has been determined based on the system layout and the structure of the application, the next step is to set the system to be partitioned automatically upon boot. For that, use the utilities provided by the tuned-profiles-realtime package. This package is installed by default when the Red Hat Enterprise Linux for Real Time packages are installed. To install tuned-profiles-realtime manually, run the following command as root:
yum install tuned-profiles-realtime
The tuned-profiles-realtime package provides the
tuned real-time profile that allows partitioning and other tunings at boot time with no additional user input required. Two configuration files control the behavior of the profile:
realtime-variables.conf file specifies the group of CPU cores to be isolated. To isolate a group of CPU cores from the system, use the
isolated_cores option as in the following example:
In the example above, the profile places the CPUs 1, 2, 3, 5, 9, 10, 11, 12, 13, and 14 into an isolated CPU category; the only threads on these CPUs are kernel threads specifically bound to the cores. These kernel threads are only run when a specific condition is raised, such as the migration thread or the watchdog thread.
isolated_cores variable is set, activate the profile with the
tuned-adm profile realtime
The profile uses the
bootloader plug-in. When activated, this plug-in adds the following boot parameters to the Linux kernel command line:
specifies CPUs listed in the
turns off the timer tick on an idle CPU; set to
off by default
turns off the timer tick on a CPU when there is only one runnable task on that CPU; needs
nohz to be set to
prevents the Intel idle driver from managing power state and CPU frequency
prevents the kernel from detecting soft lockups in user threads
In the above example, the kernel boot command-line parameters look as follows:
isolcpus=1-3,5,9-14 nohz=on nohz_full=1-3,5,9-14 intel_pstate=disable nosoftlockup
The profile runs the
script.sh shell script specified in the
[script] section of
tuned.conf. The script adjusts the following entries of the
sysfs virtual file system:
workqueue entry above is set to the inverse of the isolated CPUs mask, while the second entry turns off machine check exceptions.
The script also sets the following variables in the
kernel.hung_task_timeout_secs = 600
kernel.nmi_watchdog = 0
kernel.sched_rt_runtime_us = 1000000
vm.stat_interval = 10
The script uses the
tuna interface to move any non-bound thread on the isolated CPU numbers off of the isolated CPUs.
For further tuning, copy the default
/usr/lib/tuned/realtime/script.sh and modify it, then change the
tuned.conf JSON file to point to the modified script.