Simultaneous Multithreading in Red Hat Enterprise Linux

Resolution

Overview

Simultaneous Multithreading (SMT) allows multiple execution threads to be executed on a single physical CPU core. The technology is known by a number of different names, such as Hyper-Threading, but operate along similar principles.

Intel provides an excellent overview for Hyper-Threading (HT), an implementation of SMT, and when it may or may not be helpful, depending on the application.

Intel® Hyper-Threading Technology
How to Determine the Effectiveness of Hyper-Threading Technology with an Application | Intel® Software

Warning: The above links are to a source that are not authored by Red Hat directly. As such, Red Hat support cannot verify its accuracy and content.

SMT Security Concerns

Various microprocessor flaws have been discovered recently. Certain issues require SMT be disabled in order to more fully mitigate the issue. It is recommended that the following vulnerabilities are reviewed for the SMT specific concerns.

L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646
MDS - Microarchitectural Store Buffer Data - CVE-2018-12130, CVE-2018-12126, CVE-2018-12127, and CVE-2019-11091

Verifying a Systems Hyper-Thread Topology

Each Hyper-Thread for a system will appear to the kernel as a logical CPU core. There may be multiple cores per physical socket, as demonstrated by the lscpu utility:

# lscpu | grep -e Socket -e Core -e Thread
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2

In the above, the system is indicating that there are two sockets, with 6 cores each, and with 2 threads each core. Roughly resulting in the following visual topology:

+-------------------------------------------------------+-------------------------------------------------------+
|                        Socket 1                       |                        Socket 2                       |
| +-------------------------+-------------------------+ | +-------------------------+-------------------------+ |
| |          Core 1         |          Core 2         | | |          Core 1         |          Core 2         | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |            |            |            |            | | |            |            |            |            | |
| |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | | |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | |
| |            |            |            |            | | |            |            |            |            | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |          Core 3         |          Core 4         | | |          Core 3         |          Core 4         | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |            |            |            |            | | |            |            |            |            | |
| |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | | |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | |
| |            |            |            |            | | |            |            |            |            | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |          Core 5         |          Core 6         | | |          Core 5         |          Core 6         | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |            |            |            |            | | |            |            |            |            | |
| |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | | |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | |
| |            |            |            |            | | |            |            |            |            | |
| +---------------------------------------------------+ | +---------------------------------------------------+ |
+-------------------------------------------------------+-------------------------------------------------------+

To determine which CPUs should be disabled, the threads running on the same CPU core have to be identified. Files named /sys/devices/system/cpu/cpuN/topology/thread_siblings_list exist, where N is the logical CPU number. These files will contain the Hyper-Thread CPU pairs as shown below.

# grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0-1
/sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2-3
/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4-5
/sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6-7
/sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8-9
/sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10-11
/sys/devices/system/cpu/cpu12/topology/thread_siblings_list:12-13
/sys/devices/system/cpu/cpu14/topology/thread_siblings_list:14-15
/sys/devices/system/cpu/cpu16/topology/thread_siblings_list:16-17
/sys/devices/system/cpu/cpu18/topology/thread_siblings_list:18-19
/sys/devices/system/cpu/cpu20/topology/thread_siblings_list:20-21
/sys/devices/system/cpu/cpu22/topology/thread_siblings_list:22-23

Note: The separator in the sibling list may be either in the format 0-1 or 0,2 depending on the CPU model.

The output above indicates that Logical CPU 0 and Logical CPU 1 are threads on the same core.

Disabling SMT

There are a number of methods available for disabling SMT. The effective methods differ depending on the kernel version in use. In the event that the method below requires a newer kernel, it is indicated along with the specific method.

Note: For the methods to disable SMT below that do not require a reboot, it is not recommended to re-enable SMT threads after being disabled. Once disabled, a system reboot is the most effective and safest method for re-enabling SMT.

Hardware-level Disabling

This method needs a reboot, and it is persistent. The most effective, and generally recommended way to disable SMT, is within the systems firmware or BIOS. Please consult the systems hardware vendor documentation to determine the correct process to disable SMT. The operating system will honour the firmware's CPU configuration.

Disabling via kernel boot parameters

This method requires a newer kernel, is persistent and requires a reboot. This functionality is available via the nosmt boot parameter, in RHEL6 with 2.6.32-754 (the RHEL6.10GA kernel), and in RHEL7 with these kernels:

• RHEL7.2: kernel 3.10.0-327.71.4 or later (released in the AUS phase of the stream)
• RHEL7.3: kernel 3.10.0-514.55.4 or later (released in the EUS phase of the stream)
• RHEL7.4: kernel 3.10.0-693.37.4 or later (released in the EUS phase of the stream)
• RHEL7.5: kernel 3.10.0-862.11.6 or later
• RHEL7.6: kernel 3.10.0-957 (GA kernel)

        nosmt           [KNL,S390] Disable symmetric multithreading (SMT).
                        Equivalent to smt=1.

                        [KNL,x86] Disable symmetric multithreading (SMT).
                        nosmt=force: Force disable SMT, cannot be undone
                                     via the sysfs control file.

• The parameter can be set for the active kernel via the following:

# grubby --args=nosmt --update-kernel=DEFAULT

• The above change will apply the kernel boot parameter to the default kernel boot entry, and it may be carried forward with kernel updates. For Red Hat Enterprise Linux 7 and previous releases the arguments would be included for newer kernel versions. With Red Hat Enterprise Linux 8, in keeping with the intention of the command above, the change only applies to the default boot kernel entry and the arguments are not carried forward. Regardless of release, in the event that the grub2-mkconfig command is used, the parameter will be overwritten. To ensure that the parameter will carry over a grub2-mkconfig, the nosmt parameter would need to be applied to the GRUB_CMDLINE_LINUX entry within the /etc/sysconfig/grub configuration file.

Disabling at Runtime

This method might require a new kernel. It does not require a reboot, and is not persistent. Newer kernels include the following interface for controlling SMT at runtime.

   /sys/devices/system/cpu/smt/control:

     This file allows to read out the SMT control state and provides the
     ability to disable or (re)enable SMT. The possible states are:

        ==============  ===================================================
        on              SMT is supported by the CPU and enabled. All
                        logical CPUs can be onlined and offlined without
                        restrictions.

        off             SMT is supported by the CPU and disabled. Only
                        the so called primary SMT threads can be onlined
                        and offlined without restrictions. An attempt to
                        online a non-primary sibling is rejected

        forceoff        Same as 'off' but the state cannot be controlled.
                        Attempts to write to the control file are rejected.

        notsupported    The processor does not support SMT. It's therefore
                        not affected by the SMT implications of L1TF.
                        Attempts to write to the control file are rejected.
        ==============  ===================================================

     The possible states which can be written into this file to control SMT
     state are:

     - on
     - off
     - forceoff

   /sys/devices/system/cpu/smt/active:

     This file reports whether SMT is enabled and active, i.e. if on any
     physical core two or more sibling threads are online.

• To disable SMT during runtime, the following can be used:
  > ~~~
  > # echo off > /sys/devices/system/cpu/smt/control
  > ~~~

Disabling at Runtime, alternative

This method does not require a reboot, and is not persistent.

Red Hat strongly recommends against enabling or disabling SMT at runtime. Doing so may lead to unpredictable behavior including loss in performance, or runtime errors. Using the kernel boot command line flags is the recommended method for controlling SMT.

Runtime SMT controls are presented in the following section for completeness, but are not recommended as stated above.

• The kernel's CPU hot-plugging functionality can be used to disable HT on a per-core basis. Please see the following article to see how to proceed with CPU hot-plugging manually:

  Is it possible to "hotplug" a CPU on a running Red Hat Enterprise Linux?

• Alternatively, the following one-liner script can be used to disable one Hyper-Thread of each CPU core on the system:

  for CPU in $( ls /sys/devices/system/cpu/cpu[0-9]* -d | sort); do 
  awk -F '[-,]' '{if(NF > 1) {HOTPLUG="/sys/devices/system/cpu/cpu"$NF"/online"; print "0" > HOTPLUG; close(HOTPLUG)}}' $CPU/topology/thread_siblings_list 2>/dev/null; done
  

From the previous example system, this results in the following:

# grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
/sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2
/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4
/sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6
/sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8
/sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10
/sys/devices/system/cpu/cpu12/topology/thread_siblings_list:12
/sys/devices/system/cpu/cpu14/topology/thread_siblings_list:14
/sys/devices/system/cpu/cpu16/topology/thread_siblings_list:16
/sys/devices/system/cpu/cpu18/topology/thread_siblings_list:18
/sys/devices/system/cpu/cpu20/topology/thread_siblings_list:20
/sys/devices/system/cpu/cpu22/topology/thread_siblings_list:22