How to patch my RHV environment for Kernel Side-Channel Attack using Speculative Store Bypass CVE-2018-3639?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization 3.6 ELS
  • Red Hat Virtualization 4.2

Issue

Resolution

To fully mitigate these potential attacks, customers need to do three things:

  • Firmware level update
  • OS level updates (kernel, qemu-kvm-rhev, libvirt)
  • Application level updates (RHV patches in vdsm, rhevm-setup-plugin, rhvh and rhev-m appliance)

Both hosts and guests should be updated and rebooted. Order is not important, but after updating firmware and hypervisors, every virtual machine will need to power off and restart to recognize a new hardware type, and so it might be most convenient to patch and power cycle virtual machine guests last to save virtual machine reboots.

After applying firmware and OS patches customers will also need to upgrade to the latest RHVM and hypervisors/vdsm. These are available in the normal repositories. Details are in the latest Installation Guide on the Red Hat Customer Portal. The latest RHV updates define new CPU Types to distinguish updated hardware from vulnerable hardware. After updating firmware, and then upgrading and rebooting each hypervisor, verify each hypervisor CPU is identified properly as a model with “SSBD” at the end.

Suggested Upgrade Outline:

  1. Upgrade the manager and reboot (firmware, if applicable, kernel patches, RHV patches).

  2. Update each host and reboot(firmware, kernel, RHV).

    For further information regarding firmware availability, please review the following article:
    Is CPU microcode is available via the microcode_ctl package?

  3. Change the Cluster CPU Type to an “-IBRS” variant with the “SSBD” feature (via updated RHV-Manager).

    Example: Intel Haswell-noTSX IBRS SSBD Family

  4. Update each guest and reboot (kernel patches).

  5. Verify that the SSBD mitigations are in place by using the mitigation verification script on both hosts as well as guests.

    Kernel Side-Channel Attack using Speculative Store Bypass - CVE-2018-3639

Detailed

  1. Upgrade the manager and reboot

  • On the manager, run:

     # yum update  rhevm-setup-plugins

  • Continue upgrading the manager as described in the upgrade documentation. In order for this fix to be added, engine-setup run is required and a new kernel.

    Refer to the Upgrade Guide’s “updates between minor versions” for upgrade flow details1
    Or the Self Hosted Engine Guide2

  • Update each host and reboot

    Follow the "updating virtualization hosts" section in the Upgrade Guide3

    • Once rebooted, verify the host CPU is identified properly as a model with SSBD

       # virsh -r capabilities | grep ssbd

    • Within the manager, navigate to Host -> General -> Hardware subtab and verify CPU Type contains -IBRS variant with the "SSBD" feature.

  • Change the Cluster CPU Type to a "-IBRS" variant with the "SSBD" feature:

    After upgrading a sufficient number of hypervisors, customers will need to update their clusters or define new clusters with the new hardware types. To change the Cluster CPU Type to the -IBRS variant with SSBD, in RHVM, navigate to Cluster -> Edit and change CPU Type to -IBRS variant with SSBD. Note that all hypervisors in a cluster must be running the -IBRS CPU type variant with SSBD before changing the cluster CPU type.

    After updating hypervisors and clusters, power cycle every VM in the environment to use clusters with the new CPU types. For convenience, it might make sense to combine this with VM patch and reboot cycles below.

  • Update each guest and reboot

    Note: VMs started/migrated before the Cluster CPU change will keep using the old CPU model

    Download the appropriate updates and follow normal patching procedures to apply them. Kernel patches imply a VM reboot. For VMs not already associated with a cluster with the updated hardware type, it might make sense to power each VM off, move to an updated cluster, and then power back on.

    1. Red Hat Virtualization - 4.2 - Upgrade Guide - Chapter 2. Updates between Minor Releases
      Red Hat Enterprise Virtualization - 3.6 - Upgrade Guide - Chapter 2. Updates between Minor Releases ↩

    2. Red Hat Virtualization - 4.2 - Self-Hosted Engine Guide - Chapter 6. Upgrading the Self-Hosted Engine - 6.1. Upgrading a Self-Hosted Engine Environment
      Red Hat Enterprise Virtualization - 3.6 - 6.2. Upgrading a Self-Hosted Engine Environment ↩

    3. Red Hat Enterprise Virtualization - 4.2 - Upgrade Guide - 2.3. Manually Updating Hosts
      Red Hat Enterprise Virtualization - 3.6 - 2.4. Manually Updating Red Hat Enterprise Linux Virtualization Hosts ↩

    Root Cause

    Full technical details of this CVE can be found in:

    As with previously disclosed Spectre-class of attack vectors, virtualized environments have three primary scenarios

    • Unprivileged guest user attempting to access host's memory (guest->host)
    • Privileged guest user attempting to access other guest memory (guest->(other)guest)
    • Unprivileged guest user attempting to access guest kernel memory (guest->(same)guest).

    To prevent the first two scenarios updates need to be applied on the host and to prevent the third scenario updates need to be applied on both the host and guest system.

    This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

    Comments