3.13. Isolating CPUs Using tuned-profiles-realtime
To give application threads the most execution time possible, you can isolate CPUs, which means removing as many extraneous tasks off a CPU as possible. Isolating CPUs generally involves:
- removing all user-space threads;
- removing any unbound kernel threads (bound kernel threads are tied to a specific CPU and may not be moved);
- removing interrupts by modifying the
/proc/irq/N/smp_affinityproperty of each Interrupt Request (IRQ) number N in the system.
This section shows how to automize these operations using the
isolated_cores=cpulistconfiguration option of the tuned-profiles-realtime package.
Choosing CPUs to Isolate
Choosing which CPUs to isolate requires careful consideration of the CPU topology of the system. Different use cases may require different configuration:
- If you have a multi-threaded application where threads need to communicate with one another by sharing cache, then they may need to be kept on the same NUMA node or physical socket.
- If you run multiple unrelated real-time applications, then separating the CPUs by NUMA node or socket may be suitable.
The hwloc package provides commands useful for getting information about CPUs, including
- To show the layout of available CPUs in physical packages, use the
lstopo-no-graphics --no-io --no-legend --of txtcommand:
Figure 3.1. Showing the layout of CPUs using
lstopo-no-graphicsThe above command is useful for multi-threaded applications, because it shows how many cores and sockets are available and the logical distance of the NUMA nodes.Additionally, the hwloc-gui package provides the
lstopocommand, which produces graphical output.
- For further information about the CPUs, such as the distance between nodes, use the
numactl --hardwareavailable: 2 nodes (0-1) node 0 cpus: 0 1 2 3 node 0 size: 16159 MB node 0 free: 6323 MB node 1 cpus: 4 5 6 7 node 1 size: 16384 MB node 1 free: 10289 MB node distances: node 0 1 0: 10 21 1: 21 10
For more information about utilities provided by the hwloc package, see the
Isolating CPUs Using tuned's
The initial mechanism for isolating CPUs is specifying the boot parameter
isolcpus=cpuliston the kernel boot command line. The recommended way to do this for Red Hat Enterprise Linux for Real Time is to use the
tuneddaemon and its tuned-profiles-realtime package.
To specify the
isolcpusboot parameter, follow these steps:
- Install the tuned package and the tuned-profiles-realtime package:
yum install tuned tuned-profiles-realtime
- In file
/etc/tuned/realtime-variables.conf, set the configuration option
isolated_cores=cpulist, where cpulist is the list of CPUs that you want to isolate. The list is separated with commas and can contain single CPU numbers or ranges, for example:
isolated_cores=0-3,5,7The above line would isolate CPUs 0, 1, 2, 3, 5, and 7.
Example 3.3. Isolating CPUs with Communicating threadsIn a two socket system with 8 cores, where NUMA node zero has cores 0-3 and NUMA node one has cores 4-8, to allocate two cores for a multi-threaded application, add this line:
tuned-profiles-realtimeprofile is activated, the
isolcpus=4,5parameter will be added to the boot command line. This will prevent any user-space threads from being assigned to CPUs 4 and 5.
Example 3.4. Isolating CPUs with Non-communicating threadsIf you wanted to pick CPUs from different NUMA nodes for unrelated applications, you could specify:
isolated_cores=0,4This would prevent any user-space threads from being assigned to CPUs 0 and 4.
- Activate the
tunedprofile using the
tuned-admutility and then reboot:
~]# tuned-adm profile realtime ~]# reboot
- Upon reboot, verify that the selected CPUs have been isolated by searching for the
isolcpusparameter at the boot command line:
cat /proc/cmdline | grep isolcpusBOOT_IMAGE=/vmlinuz-3.10.0-394.rt56.276.el7.x86_64 root=/dev/mapper/rhel_foo-root ro crashkernel=auto rd.lvm.lv=rhel_foo/root rd.lvm.lv=rhel_foo/swap console=ttyS0,115200n81 isolcpus=0,4
Isolating CPUs Using the
nohz_fullkernel boot parameters, you need to use one of the following profiles:
- May be used to reduce timer activity on a particular set of CPUs. The
nohzparameter is mainly used to reduce timer interrupts happening on idle CPUs. This helps battery life by allowing the idle CPUs to run in reduced power mode. While not being directly useful for real-time response time, the
nohzparameter does not directly hurt real-time response time and is required to activate the next parameter which *does* have positive implications for real-time performance.
nohz_fullparameter is used to treat a list of CPUs differently, with respect to timer ticks. If a CPU is listed as a nohz_full CPU and there is only one runnable task on the CPU, then the kernel will stop sending timer ticks to that CPU, so more time may be spent running the application and less time spent servicing interrupts and context switching.
For more information on these parameters, see Configuring kernel tick time