An overview and comparison of the isolcpus kernel parameter

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 10
  • Kernel booted with isolcpus

Issue

  • What does isolcpus do?
  • How to set the isolcpus kernel parameter?

Resolution

  • To enable isolcpus, modify the kernel boot parameters of Red Hat Enterprise Linux as described in the knowledgebase article How to manually modify the boot parameter in grub before the system boots.

  • Below is the general syntax for the isolcpus parameter:

    isolcpus=<cpu number>,...,<cpu number>
or
isolcpus=<cpu number>-<cpu number>
or
isolcpus=<cpu number>,...,<cpu number>-<cpu number>

  • After modifying the kernel boot parameter(s), reboot for the changes to take effect.

  • Once booted, /proc/cmdline can be checked to ensure the isolcpus parameter took effect:

    $ cat /proc/cmdline 
ro root=/dev/mapper/vg_dhcp--1--67-root rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_dhcp-1-67/root rd_NO_MD quiet SYSFONT=latarcyrheb-sun16 rhgb crashkernel=128M rd_LVM_LV=vg_dhcp-1-67/LogVol01  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM isolcpus=1,3,2

  • From RHEL 7.2 onwards (kernel-3.10.0-327.el7.x86_64 and higher), the isolated CPUs information can be verified from sysfs. The information is exported under following file:

    # cat /sys/devices/system/cpu/isolated
1-3

  • If isolcpus is used with managed_irq or nohz flag, then make sure to use domain flag to Isolate from the general SMP balancing and scheduling algorithms. If isolcpus is used only with managed_irq or nohz flag and cpu-list, then /sys/devices/system/cpu/isolated file shows blank output. More details can be found in the kcs: How to verify list of "isolated" and "nohz_full" CPU information from sysfs?

  • Warning Make sure the use of isolcpus follows the recommendations outlined by the product vendor of the product needing to use isolcpus, and the application stack is configured to take advantage of isolcpus. Otherwise, isolating too many CPUs from the process scheduler while the system is expected to handle a heavy workload may cause a negative performance impact on the system.

Root Cause

  • The isolcpus defines a set of CPUs where the kernel processes scheduler will not interact with those CPUs.
  • The Linux kernel contains a process scheduling subsystem responsible for moving processes onto, off of, and around CPUs on a system.
  • Under typical conditions, the process scheduling subsystem, also commonly referred to as the process scheduler, moves processes around on the CPUs in the system in an effort to help provide equal running time to all processes actively executing. The process scheduler may also keep an executing process on a CPUs if there is no need to migrate it to another CPU.

  • The isolcpus kernel parameter designates a set of CPUs where the process scheduler will ignore those CPUs. That is to say, the process scheduler will not include those CPUs as targets to put processes on, migrate processes onto or off of, or take a process off a CPU.

    • Without intervention, the process scheduler will not put newly created processes on a CPU listed in isolcpus.
    • Without intervention, the process scheduler will not migrate processes onto or off of any CPU listed in isolcpus. This includes migrating processes from/to regular CPUs to/from CPUs listed in isolcpus and migrating processes between CPUs listed in isolcpus.
    • A process can be moved onto an isolcpus CPU by a systems administrator or script with tools such as taskset. See the the man page for taskset(1) for more info.

    • A process can also put itself onto a CPU listed in isolcpus with scheduling-specific functions such as sched_setaffinity(). Please see the man page sched_setaffinity(2) for more information. In particular, note the following from the man page;

      NOTES
[...]
       The  isolcpus  boot  option  can be used to isolate one or more CPUs at
       boot time, so that no processes are scheduled onto those CPUs.  Follow‐
       ing  the  use  of  this boot option, the only way to schedule processes
       onto the isolated CPUs is  via  sched_setaffinity()  or  the  cpuset(7)

  • Note isolcpus is different from setting CPU affinity via systemd's CPUAffinity= configuration options and/or using Cgroups.

    • For either Cgroups or systemd, the process scheduler will still automatically migrate processes around in the Cgroup CPU set and/or the list of CPUs designated by CPUAffinity.
    • For isolcpus the process scheduler will not touch processes in the isolcpus list. As such, if a process is on an isolcpus CPU, it will stay on that CPU until it finishes executing or yields the CPU.
    • As such, the usage of isolcpus requires the applications using CPUs in the isolcpus list to manage process scheduling itself or run the risk of starving those CPUs from other applications that may use them and various Linux kernel-specific processes which maintain the health of the system.

  • The usage of isolcpus may help improve performance for jitter-/latency-sensitive applications. The process scheduler must periodically interrupt regular CPUs to determine if a rebalance/process migration is needed. CPUs listed in isolcpus generally do not incur such interrupts mitigating any overhead introduced by the interrupts into the applications running on the isolcpus CPU(s).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments