Performance considerations for L1 Terminal Fault

Updated -

The Red Hat Performance Engineering team worked with our security and kernel engineering teams to measure the performance impact from L1 Terminal Fault (L1TF) on various workloads. These workloads include, but were not limited to:

  • Microbenchmarks
  • OLTP benchmark (based on TPC-C)
  • Java
  • Network
  • Filesystem
  • SPEC benchmarks
  • Running on bare metal
  • Red Hat Virtualization
  • Red Hat OpenStack
  • OpenShift-on-OpenStack environments

The rest of this document describes L1TF and the performance impact for mitigating its three components.
Details for L1TF and the controls to mitigate it can be found in the following article:
L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646

It is advised to refer to the "Mitigation/Remediation" subsection of the above document as a reference for the various mitigation measures documented here. Testing was conducted with a view to each mitigation measure, and how they impact performance when enabled. The three mitigation techniques and their performance impact is discussed below.

First: PTE Inversion
PTE Inversion fixes the L1TF vulnerability for processes running on the host. It is permanently enabled and does not impact performance.

Second: L1D cache flushing on vmentry
To ensure that a guest cannot attack data which is present in the L1D cache, the hypervisor flushes the L1D before entering the guest. The kernel provides three controllable L1D flush modes, including “never”, “cond” for conditional, and “always”.
It’s difficult to predict which workloads L1D flushing will impact, because so many variables are involved as to which important cache line data resides in the L1D cache when it gets flushed.

  1. Setting “never” indicates to the hypervisor that it should never flush the L1D on a vmentry. It has no performance impact.
  2. Setting “cond”, the default value, causes the L1D to be flushed for a limited number of vmentry code paths.
    1. Most workloads we tested saw no measurable impact.
    2. A small number of workloads exhibited performance losses of upwards of 3-4%. Though seeing such a loss on one test configuration didn’t always mean that loss would be seen on another test configuration running the same workload.
  3. Setting “always” causes an L1D flush for every vmentry.
    1. The majority of the workloads showed little to no impact.
    2. Several large multithreaded applications, like SPECjbb, showed an up to 4% slowdown.
    3. Asynchronous file I/O benchmarks, which trigger high rates of vmexits, did have up to a 14% impact in one hardware configuration, and only 0-5% in another configuration.
    4. This was also the setup where we saw the biggest performance impact from not having updated microcode. The above 14% impact (with microcode) slowed down to a 27% performance impact (without microcode). In most cases the hit from not having updated microcode was much smaller, in the 0-2% range. Thus, updated microcode can help with performance, especially if the L1D flushing is set to “always”.
    5. Custom microbenchmarks (that tend to not represent real workloads), with tight loops that triggered high rates of vmexits did show slowdowns well into the double digit ranges.

Third: Disabling Hyper-Threading
The performance impact from disabling Hyper-Threading (HT) varies widely. Disabling HT reduces the number of available vCPUs by 50%, which reduces the scale of virtualized environments. The Red Hat products that use KVM (Red Hat Virtualization and Red Hat OpenStack) are affected, as are OpenShift-on-OpenStack environments.

Red Hat’s performance testing on Intel hardware is consistent with Intel’s published data on Hyper-Threading. The performance impact when HT is disabled is dependent on many factors. Measured impact ranges from a +30% gain, to -50% loss and beyond. Most HT testing, however, showed losses in the 0-30% range.

Disabling HT in environments with less than ~50% cpu utilization usually showed no performance impact. Disabling HT in environments with higher cpu utilization yields a serious performance drop. The amount of loss was usually dependent on the cpu utilization before HT was disabled.

  1. When HT is disabled in VM guests whose cpu and memory usage fit within one numa node, the performance impact we measured with database, network, and SPEC benchmarks, frequently fell into the -10-30% range. However on larger VMs (where cpu+memory usage spanned multiple numa nodes) the impact tended to be less, usually in the -10% range.
  2. We’ve measured certain applications that ran faster when HT was disabled. Some examples include:
    1. A highly optimized and tightly scheduled linpack benchmark within a VM that spans multiple numa nodes. In that scenario, the performance of this benchmark improved up to a 30% gain with HT disabled.
    2. OLTP benchmark (based on TPC-C) whose guest size fit on one numa node ran faster with HT enabled. However larger OLTP guests that need the memory and cpu resources from multiple numa nodes ran faster with HT disabled.
  3. The impact for clouds hosting VMs may be significant. Disabling HT has the perceived effect of 50% fewer vCPU for scheduling purposes. As a result, OpenStack and Red Hat Virtualization will have a perceived performance impact of up to 50% less capacity. Tools that monitor utilization will indeed view a NON-HT cloud to be up to 50% smaller than with HT enabled. Cloud providers, (e.g. Amazon Web Services, Google Cloud and Microsoft Azure) may have mitigations within their public clouds such that Red Hat Enterprise Linux VM’s in their clouds are not vulnerable to this CVE. It is recommended to verify susceptibility and mitigations measures with public cloud vendors in which Red Hat software is resident.

Considerations for disabling Hyper-Threading on OpenStack, Red Hat Virtualization, and OpenShift

Different environments may require different considerations when disabling Hyper-Threading. Here are the known special considerations for OpenStack, Red Hat Virtualization, and OpenShift.

Red Hat Virtualization

  • Disabling HT will reduce the perceived scale by 50%.
  • No distinctions made between deployments with or without Hyper-Threading. Available resources are verified to determine scheduling.

OpenStack

  • Disabling HT will reduce the perceived scale by 50%.
  • No distinctions made between deployments with or without Hyper-Threading. Available resources are verified to determine scheduling.
  • OpenStack, by default, overcommits hypervisor(s) at a ratio of 16:1. In that state, a system with 32 Hyper-Threads would allow 512 vcpus to be used for that individual hypervisor. Disabling HT in that same example shall leave the hypervisor with only 256 vcpus.
  • Prior to L1TF, the OpenStack hypervisor memory would be the primary limiting factor for users, since no default memory overcommitment is present. Following an effort to disable HT, as a result of CVE-2018-3620 & CVE-2018-3646, the number of vcpus a hypervisor has could become the limiting factor.

OpenShift

  • Disabling Hyper-Threading for L1TF mitigation is only relevant if OpenShift is running on a virtualized infrastructure (e.g. Red Hat Virtualization, OpenStack).
  • Kubernetes relies on cAdvisor to determine hardware information about a system, including number of cores and Hyper-Threads. These attributes are then normalized for each core into what are referred to as “millicores”. User requests for CPU capacity are then limited by these units.
  • For a single server with 8 cores (4 physical cores, 4 HT), 8,000 millicores of schedulable capacity are available from Kubernetes point of view. From a scheduling standpoint, Kubernetes treats all cores (HT or not) equally, as a bucket of compute capacity.
  • Overcommit capabilities are implicitly defined by how the user defines pods. Best practices are to set Requests on a pod.
  • When neither requests nor limits are set, Kubernetes will allow overcommitment of physical resources in the cluster. In this scenario, disabling Hyper-Threading will not cause pods to be unschedulable. Performance may degrade depending on how overcommitted the cluster is, because any performance advantage of Hyper-Threading has been removed.
  • When Requests and/or Limits are set, disabling Hyper-Threads will have the effect of halving the overall capacity of the cluster. This will cause pods to be evicted and eventually a queue of pods that cannot be scheduled due to lack of capacity.
  • In environments with less than 50% cpu utilization, disabling HT should have little or no performance or scheduling impact.

A tool called cluster-capacity is included with OpenShift. This tool can be used to derive capacity on a running cluster, given a pod or deployment config.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.