3.13. Isolating CPUs Using tuned-profiles-realtime

To give application threads the most execution time possible, you can isolate CPUs, which means removing as many extraneous tasks off a CPU as possible. Isolating CPUs generally involves:
  • removing all user-space threads;
  • removing any unbound kernel threads (bound kernel threads are tied to a specific CPU and may not be moved);
  • removing interrupts by modifying the /proc/irq/N/smp_affinity property of each Interrupt Request (IRQ) number N in the system.
This section shows how to automize these operations using the isolated_cores=cpulist configuration option of the tuned-profiles-realtime package.

Choosing CPUs to Isolate

Choosing which CPUs to isolate requires careful consideration of the CPU topology of the system. Different use cases may require different configuration:
  • If you have a multi-threaded application where threads need to communicate with one another by sharing cache, then they may need to be kept on the same NUMA node or physical socket.
  • If you run multiple unrelated real-time applications, then separating the CPUs by NUMA node or socket may be suitable.
The hwloc package provides commands useful for getting information about CPUs, including lstopo-no-graphics and numactl:
  • To show the layout of available CPUs in physical packages, use the lstopo-no-graphics --no-io --no-legend --of txt command:
    Showing the layout of CPUs using lstopo-no-graphics

    Figure 3.1. Showing the layout of CPUs using lstopo-no-graphics

    The above command is useful for multi-threaded applications, because it shows how many cores and sockets are available and the logical distance of the NUMA nodes.
    Additionally, the hwloc-gui package provides the lstopo command, which produces graphical output.
  • For further information about the CPUs, such as the distance between nodes, use the numactl --hardware command: 
    ~]# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 16159 MB
node 0 free: 6323 MB
node 1 cpus: 4 5 6 7
node 1 size: 16384 MB
node 1 free: 10289 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10
For more information about utilities provided by the hwloc package, see the hwloc(7) manpage.

Isolating CPUs Using tuned's isolated_cores Option

The initial mechanism for isolating CPUs is specifying the boot parameter isolcpus=cpulist on the kernel boot command line. The recommended way to do this for Red Hat Enterprise Linux for Real Time is to use the tuned daemon and its tuned-profiles-realtime package.
To specify the isolcpus boot parameter, follow these steps:
  1. Install the tuned package and the tuned-profiles-realtime package: 
    ~]# yum install tuned tuned-profiles-realtime
  2. In file /etc/tuned/realtime-variables.conf, set the configuration option isolated_cores=cpulist, where cpulist is the list of CPUs that you want to isolate. The list is separated with commas and can contain single CPU numbers or ranges, for example: 
    isolated_cores=0-3,5,7
    The above line would isolate CPUs 0, 1, 2, 3, 5, and 7.

    Example 3.3. Isolating CPUs with Communicating threads

    In a two socket system with 8 cores, where NUMA node zero has cores 0-3 and NUMA node one has cores 4-7, to allocate two cores for a multi-threaded application, add this line: 
    isolated_cores=4,5
    Once the tuned-profiles-realtime profile is activated, the isolcpus=4,5 parameter will be added to the boot command line. This will prevent any user-space threads from being assigned to CPUs 4 and 5.

    Example 3.4. Isolating CPUs with Non-communicating threads

    If you wanted to pick CPUs from different NUMA nodes for unrelated applications, you could specify: 
    isolated_cores=0,4
    This would prevent any user-space threads from being assigned to CPUs 0 and 4.
  3. Activate the tuned profile using the tuned-adm utility and then reboot: 
    ~]# tuned-adm profile realtime
~]# reboot
  4. Upon reboot, verify that the selected CPUs have been isolated by searching for the isolcpus parameter at the boot command line: 
    ~]$ cat /proc/cmdline | grep isolcpus
BOOT_IMAGE=/vmlinuz-3.10.0-394.rt56.276.el7.x86_64 root=/dev/mapper/rhel_foo-root ro crashkernel=auto rd.lvm.lv=rhel_foo/root rd.lvm.lv=rhel_foo/swap console=ttyS0,115200n81 isolcpus=0,4

Isolating CPUs Using the nohz and nohz_full Parameters

To enable nohz and nohz_full kernel boot parameters, you need to use one of the following profiles: realtime-virtual-host, realtime-virtual-guest, or cpu-partitioning.
nohz=on
May be used to reduce timer activity on a particular set of CPUs. The nohz parameter is mainly used to reduce timer interrupts happening on idle CPUs. This helps battery life by allowing the idle CPUs to run in reduced power mode. While not being directly useful for real-time response time, the nohz parameter does not directly hurt real-time response time and is required to activate the next parameter which *does* have positive implications for real-time performance.
nohz_full=cpulist
The nohz_full parameter is used to treat a list of CPUs differently, with respect to timer ticks. If a CPU is listed as a nohz_full CPU and there is only one runnable task on the CPU, then the kernel will stop sending timer ticks to that CPU, so more time may be spent running the application and less time spent servicing interrupts and context switching.
For more information on these parameters, see Configuring kernel tick time