Simultaneous Multithreading in Red Hat Enterprise Linux

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 8
  • Red Hat Enterprise Linux (RHEL) 7
  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 5

Issue

  • What is Simultaneous Multithreading (SMT)?
  • Is Simultaneous Multithreading the same as Hyper-Threading?
  • How do we determine if SMT is in use?
  • How to disable Simultaneous Multithreading in RHEL?
  • Are there any security concerns related to SMT?

Resolution

Overview

Simultaneous Multithreading (SMT) allows multiple execution threads to be executed on a single physical CPU core. The technology is known by a number of different names, such as Hyper-Threading, but operate along similar principles.

Intel provides an excellent overview for Hyper-Threading (HT), an implementation of SMT, and when it may or may not be helpful, depending on the application.

Intel® Hyper-Threading Technology
How to Determine the Effectiveness of Hyper-Threading Technology with an Application | Intel® Software

Warning: The above links are to a source that are not authored by Red Hat directly. As such, Red Hat support cannot verify its accuracy and content.

SMT Security Concerns

Various microprocessor flaws have been discovered recently. Certain issues require SMT be disabled in order to more fully mitigate the issue. It is recommended that the following vulnerabilities are reviewed for the SMT specific concerns.

L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646
MDS - Microarchitectural Store Buffer Data - CVE-2018-12130, CVE-2018-12126, CVE-2018-12127, and CVE-2019-11091

Verifying a Systems Hyper-Thread Topology

Each Hyper-Thread for a system will appear to the kernel as a logical CPU core. There may be multiple cores per physical socket, as demonstrated by the lscpu utility:

# lscpu | grep -e Socket -e Core -e Thread
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             2

In the above, the system is indicating that there are two sockets, with 6 cores each, and with 2 threads each core. Roughly resulting in the following visual topology:

+-------------------------------------------------------+-------------------------------------------------------+
|                        Socket 1                       |                        Socket 2                       |
| +-------------------------+-------------------------+ | +-------------------------+-------------------------+ |
| |          Core 1         |          Core 2         | | |          Core 1         |          Core 2         | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |            |            |            |            | | |            |            |            |            | |
| |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | | |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | |
| |            |            |            |            | | |            |            |            |            | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |          Core 3         |          Core 4         | | |          Core 3         |          Core 4         | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |            |            |            |            | | |            |            |            |            | |
| |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | | |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | |
| |            |            |            |            | | |            |            |            |            | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |          Core 5         |          Core 6         | | |          Core 5         |          Core 6         | |
| +------------+-------------------------+------------+ | +------------+-------------------------+------------+ |
| |            |            |            |            | | |            |            |            |            | |
| |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | | |  Thread 1  |  Thread 2  |  Thread 1  |  Thread 2  | |
| |            |            |            |            | | |            |            |            |            | |
| +---------------------------------------------------+ | +---------------------------------------------------+ |
+-------------------------------------------------------+-------------------------------------------------------+

To determine which CPUs should be disabled, the threads running on the same CPU core have to be identified. The files /sys/devices/system/cpu/cpuN/topology/thread_siblings_list where N is the logical CPU number. This file will contain the Hyper-Thread CPU pairs as shown below.

# grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0-1
/sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2-3
/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4-5
/sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6-7
/sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8-9
/sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10-11
/sys/devices/system/cpu/cpu12/topology/thread_siblings_list:12-13
/sys/devices/system/cpu/cpu14/topology/thread_siblings_list:14-15
/sys/devices/system/cpu/cpu16/topology/thread_siblings_list:16-17
/sys/devices/system/cpu/cpu18/topology/thread_siblings_list:18-19
/sys/devices/system/cpu/cpu20/topology/thread_siblings_list:20-21
/sys/devices/system/cpu/cpu22/topology/thread_siblings_list:22-23

Note: The separator in the sibling list may be either in the format 0-1 or 0,2 depending on the CPU model.

The output above indicates that Logical CPU 0 and Logical CPU 1 are threads on the same core.

Disabling SMT

There are a number of methods available for disabling SMT. The effective methods differ depending on the kernel version in use. In the event that the method below requires a newer kernel, it is indicated along with the specific method.

Note: For the methods to disable SMT below that do not require a reboot, it is not recommended to re-enable SMT threads after being disabled. Once disabled, a system reboot is the most effective and safest method for re-enabling SMT.

Hardware-level Disabling - Requiring a reboot and Persistent

  • The most effective, and generally recommended way to disable SMT, is within the systems firmware or BIOS. Please consult the systems hardware vendor documentation to determine the correct process to disable SMT. The operating system will honour the firmware's CPU configuration.

Requires a newer Kernel - Using kernel boot parameters - Requiring a reboot and Persistent

  • With newer kernel releases, the nosmt boot parameter has been expanded to the x86 architecture.
        nosmt           [KNL,S390] Disable symmetric multithreading (SMT).
                        Equivalent to smt=1.

                        [KNL,x86] Disable symmetric multithreading (SMT).
                        nosmt=force: Force disable SMT, cannot be undone
                                     via the sysfs control file.
  • The parameter can be set for the active kernel via the following:
# grubby --args=nosmt --update-kernel=DEFAULT
  • The above change will apply the kernel boot parameter to the default kernel boot entry, and it may be carried forward with kernel updates. For Red Hat Enterprise Linux 7 and previous releases the arguments would be included for newer kernel versions. With Red Hat Enterprise Linux 8, in keeping with the intention of the command above, the change only applies to the default boot kernel entry and the arguments are not carried forward. Regardless of release, in the event that the grub2-mkconfig command is used, the parameter will be overwritten. To ensure that the parameter will carry over a grub2-mkconfig, the nosmt parameter would need to be applied to the GRUB_CMDLINE_LINUX entry within the /etc/sysconfig/grub configuration file.

Requires a newer Kernel - Disabling at Runtime - Not requiring a reboot but not Persistent

  • Newer kernels include the following interface for controlling SMT at runtime.

       /sys/devices/system/cpu/smt/control:
    
         This file allows to read out the SMT control state and provides the
         ability to disable or (re)enable SMT. The possible states are:
    
            ==============  ===================================================
            on              SMT is supported by the CPU and enabled. All
                            logical CPUs can be onlined and offlined without
                            restrictions.
    
            off             SMT is supported by the CPU and disabled. Only
                            the so called primary SMT threads can be onlined
                            and offlined without restrictions. An attempt to
                            online a non-primary sibling is rejected
    
            forceoff        Same as 'off' but the state cannot be controlled.
                            Attempts to write to the control file are rejected.
    
            notsupported    The processor does not support SMT. It's therefore
                            not affected by the SMT implications of L1TF.
                            Attempts to write to the control file are rejected.
            ==============  ===================================================
    
         The possible states which can be written into this file to control SMT
         state are:
    
         - on
         - off
         - forceoff
    
       /sys/devices/system/cpu/smt/active:
    
         This file reports whether SMT is enabled and active, i.e. if on any
         physical core two or more sibling threads are online.
    
  • To disable SMT during runtime, the following can be used:

    # echo off > /sys/devices/system/cpu/smt/control
    

Disabling at Runtime - Not requiring a reboot but not Persistent - Alternative

Red Hat strongly recommends against enabling or disabling SMT at runtime. Doing so may lead to unpredictable behavior including loss in performance, or runtime errors. Using the kernel boot command line flags is the recommended method for controlling SMT.

Runtime SMT controls are presented in the following section for completeness, but are not recommended as stated above.

  • The kernel's CPU hot-plugging functionality can be used to disable HT on a per-core basis. Please see the following article to see how to proceed with CPU hot-plugging manually:

    Is it possible to "hotplug" a CPU on a running Red Hat Enterprise Linux?

  • Alternatively, the following one-liner script can be used to disable one Hyper-Thread of each CPU core on the system:

    for CPU in $( ls /sys/devices/system/cpu/cpu[0-9]* -d | sort); do 
    awk -F '[-,]' '{if(NF > 1) {HOTPLUG="/sys/devices/system/cpu/cpu"$NF"/online"; print "0" > HOTPLUG; close(HOTPLUG)}}' $CPU/topology/thread_siblings_list 2>/dev/null; done
    

From the previous example system, this results in the following:

# grep -H . /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | sort -n -t ':' -k 2 -u
/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
/sys/devices/system/cpu/cpu2/topology/thread_siblings_list:2
/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4
/sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6
/sys/devices/system/cpu/cpu8/topology/thread_siblings_list:8
/sys/devices/system/cpu/cpu10/topology/thread_siblings_list:10
/sys/devices/system/cpu/cpu12/topology/thread_siblings_list:12
/sys/devices/system/cpu/cpu14/topology/thread_siblings_list:14
/sys/devices/system/cpu/cpu16/topology/thread_siblings_list:16
/sys/devices/system/cpu/cpu18/topology/thread_siblings_list:18
/sys/devices/system/cpu/cpu20/topology/thread_siblings_list:20
/sys/devices/system/cpu/cpu22/topology/thread_siblings_list:22

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

5 Comments

The hyperlink from 'Is it possible to "hotplug" a CPU on a running Red Hat Enterprise Linux 5 or 6 system?' leads to the article for RHEL 3 and 4; it should be https://access.redhat.com/site/solutions/15504.

Good catch. This link is now fixed.

Will this required to reboot the server

are these configurations temporary? Do we need to call for loop statement each time after reboot to disable hyper threading?

I think that for a persistent change on RHEL-8 (x86_64, ppc64le, aarch64) is the best way to do:

grub2-editenv - set "$(grub2-editenv - list | grep kernelopts) nosmt"
reboot