Chapter 9. Affinity in RHEL for Real Time

In real-time, every thread and interrupt source in the system has a processor affinity property. The operating system scheduler uses this information to determine which threads and interrupts to run on which CPU.

The Affinity in real-time, is represented as a bitmask, where each bit in the mask represents a CPU core. If the bit is set to 1, then the thread or interrupt may run on that core; if 0 then the thread or interrupt is excluded from running on the core. The default value for an affinity bitmask is all ones, meaning the thread or interrupt may run on any core in the system.

By default, processes can run on any CPU. However, processes can be instructed to run on a predetermined selection of CPUs, by changing the affinity of the process. Child processes inherit the CPU affinities of their parents.

Some of the more typical affinity setups include:

  • Reserve one CPU core for all system processes and allow the application to run on the remainder of the cores.
  • Allow a thread application and a given kernel thread (such as the network softirq or a driver thread) on the same CPU.
  • Pair producer and consumer threads on each CPU.
Note

The affinity settings must be designed in conjunction with the program for good expected behavior.

9.1. Processor affinity

In real-time, the processes by default, can run on any CPU. However, you can configure the processes to run on a predetermined selection of CPUs, by changing the affinity of the process. Child processes inherit the CPU affinities of their parents.

The real-time practice for tuning affinities on a system is to determine the number of cores required to run the application and then isolating those cores. This can be achieved with the Tuna tool, or with shell scripts to modify the bitmask value.

The taskset command can be used to change the affinity of a process and modifying the /proc/ filesystem entry changes the affinity of an interrupt. Using the taskset command with the -p or --pid option and the process identifier (PID) of the process, checks the affinity of a process.

The -c or --cpu-list option displays the numerical list of cores, instead of as a bitmask. The affinity can be set by specifying the number of the CPU to bind a specific process. For example, for a process that previously used either CPU 0 or CPU 1, you can change the affinity so that it can only run on CPU 1. In addition to the taskset command, you can also set the processor affinity can using the sched_setaffinity() system call.

Additional resources

  • taskset(1) man page
  • sched_setaffinity(2) man page

9.2. SCHED_DEADLINE and cpusets

The kernel’s deadline scheduling class (SCHED_DEADLINE) implements early deadline first scheduler (EDF) for sporadic tasks with a constrained deadline. It prioritizes the tasks according to the job deadline: earliest absolute deadline first. In addition to the EDF scheduler, the deadline scheduler also implements the constant bandwidth server (CBS). The CBS algorithm is a resource reservation protocol.

The CBS guarantees that each task receives its run time (Q) at every period (T). At the start of every activation of a task, the CBS replenishes the task’s run time. As the job runs, it consumes its runtime and if the task runs out of its runtime, the task is throttled and de-scheduled. The throttling mechanism prevents a single task from running more than its runtime and helps to avoid the performance problems of other jobs.

In real-time, to avoid the overloading the system with deadline tasks, the deadline scheduler implements an acceptance test, which is run every time a task is configured to run with the deadline scheduler. The acceptance test guarantees that SCHED_DEADLINE tasks does not use more CPU time than the specified on the kernel.sched_rt_runtime_us/kernel.sched_rt_period_us files, which is 950 ms over 1s, by default.