Chapter 23. Isolating CPUs using tuned-profiles-real-time

To give application threads the most execution time possible, you can isolate CPUs. Therefore, remove as many extraneous tasks from a CPU as possible. Isolating CPUs generally involves:

  • Removing all user-space threads.
  • Removing any unbound kernel threads. Kernel related bound threads are linked to a specific CPU and cannot not be moved).
  • Removing interrupts by modifying the /proc/irq/N/smp_affinity property of each Interrupt Request (IRQ) number N in the system.

By using the isolated_cores=cpulist configuration option of the tuned-profiles-realtime package, you can automate operations to isolate a CPU.

Prerequisites

  • You have administrator privileges.

23.1. Choosing CPUs to isolate

Choosing the CPUs to isolate requires careful consideration of the CPU topology of the system. Different use cases require different configuration:

  • If you have a multi-threaded application where threads need to communicate with one another by sharing cache, they need to be kept on the same NUMA node or physical socket.
  • If you run multiple unrelated real-time applications, separating the CPUs by NUMA node or socket can be suitable.

The hwloc package provides utilities that are useful for getting information about CPUs, including lstopo-no-graphics and numactl.

Prerequisites

  • The hwloc package are installed.

Procedure

  1. View the layout of available CPUs in physical packages: 

    # lstopo-no-graphics --no-io --no-legend --of txt

    Figure 23.1. Showing the layout of CPUs using lstopo-no-graphics

    lstopo no graphics output

    This command is useful for multi-threaded applications, because it shows how many cores and sockets are available and the logical distance of the NUMA nodes.

    Additionally, the hwloc-gui package provides the lstopo utility, which produces graphical output.

  2. View more information about the CPUs, such as the distance between nodes: 

    # numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 16159 MB
node 0 free: 6323 MB
node 1 cpus: 4 5 6 7
node 1 size: 16384 MB
node 1 free: 10289 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Additional resources

  • the hwloc(7) man page

23.2. Isolating CPUs using TuneD’s isolated_cores option

The initial mechanism for isolating CPUs is specifying the boot parameter isolcpus=cpulist on the kernel boot command line. The recommended way to do this for RHEL for Real Time is to use the TuneD daemon and its tuned-profiles-realtime package.

Note

In tuned-profiles-realtime version 2.19 and later, the built-in function calc_isolated_cores applies the initial CPU setup automatically. The /etc/tuned/realtime-variables.conf configuration file includes the default variable content as isolated_cores=${f:calc_isolated_cores:2}.

By default, calc_isolated_cores reserves one core per socket for housekeeping and isolates the rest. If you must change the default configuration, comment out the isolated_cores=${f:calc_isolated_cores:2} line in /etc/tuned/realtime-variables.conf configuration file and follow the procedure steps for Isolating CPUs using TuneD’s isolated_cores option.

Prerequisites

  • The TuneD and tuned-profiles-realtime packages are installed.
  • You have root permissions on the system.

Procedure

  1. As a root user, open /etc/tuned/realtime-variables.conf in a text editor.

  2. Set isolated_cores=cpulist to specify the CPUs that you want to isolate. You can use CPU numbers and ranges.

    Examples:

    isolated_cores=0-3,5,7

    This isolates cores 0, 1, 2, 3, 5, and 7.

    In a two socket system with 8 cores, where NUMA node 0 has cores 0-3 and NUMA node 1 has cores 4-8, to allocate two cores for a multi-threaded application, specify: 

    isolated_cores=4,5

    This prevents any user-space threads from being assigned to CPUs 4 and 5.

    To pick CPUs from different NUMA nodes for unrelated applications, specify: 

    isolated_cores=0,4

    This prevents any user-space threads from being assigned to CPUs 0 and 4.

  3. Activate the real-time TuneD profile using the tuned-adm utility. 

    # tuned-adm profile realtime
  4. Reboot the machine for changes to take effect.

Verification

  • Search for the isolcpus parameter in the kernel command line: 

    $ cat /proc/cmdline | grep isolcpus
BOOT_IMAGE=/vmlinuz-4.18.0-305.rt7.72.el8.x86_64 root=/dev/mapper/rhel_foo-root ro crashkernel=auto rd.lvm.lv=rhel_foo/root rd.lvm.lv=rhel_foo/swap console=ttyS0,115200n81 isolcpus=0,4

23.3. Isolating CPUs using the nohz and nohz_full parameters

The nohz and nohz_full parameters modify activity on specified CPUs. To enable these kernel boot parameters, you need to use one of the following TuneD profiles: realtime-virtual-host, realtime-virtual-guest, or cpu-partitioning.

nohz=on

Reduces timer activity on a particular set of CPUs.

The nohz parameter is mainly used to reduce timer interrupts on idle CPUs. This helps battery life by allowing idle CPUs to run in reduced power mode. While not being directly useful for real-time response time, the nohz parameter does not directly impact real-time response time negatively. But the nohz parameter is required to activate the nohz_full parameter that does have positive implications for real-time performance.

nohz_full=cpulist
The nohz_full parameter treats the timer ticks of a list of specified CPUs differently. If a CPU is specified as a nohz_full CPU and there is only one runnable task on the CPU, then the kernel stops sending timer ticks to that CPU. As a result, more time may be spent running the application and less time spent servicing interrupts and context switching.

Additional resources