How to patch my RHV environment for Meltdown and Spectre CVE( CVE-2017-5754, CVE-2017-5753, and CVE-2017-5715)?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualization 3.6 ELS
  • Red Hat Virtualization 4.1

Issue

Resolution

Refer to the Root Cause section to understand the specifics of the CVE and impact on RHV environment.
Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715

To get the full picture of what VMs and what hosts in your environment has been patched (have IBRS support) and which have not yet, refer to How do I know if my RHV environment has been patched for Meltdown and Spectre CVE?

To fully mitigate these potential attacks, customers need to do three things:

  • firmware level update
  • OS level updates (kernel, qemu-kvm, libvirt)
  • application level updates (RHV patches in vdsm, rhevm-setup-plugin, rhvh and rhev-m appliance)

Both hosts and guests should be updated and rebooted. Order is not important, but after updating firmware and hypervisors, every virtual machine will need to power off and restart to recognize a new hardware type, and so it might be most convenient to patch and power cycle virtual machine guests last to save virtual machine reboots.

After applying firmware and OS patches customers will also need to upgrade to the latest RHVM and hypervisors. These are available in the normal repositories. Details are in the latest Installation Guide on the Red Hat Customer Portal. The latest RHV updates define new CPU Types to distinguish updated hardware from vulnerable hardware. After updating firmware, and then upgrading and rebooting each hypervisor, verify each hypervisor CPU is identified properly as a model with -IBRS at the end.

Upgrade outline:

Note 1: For any physical server (hosts and manager, if running on physical hardware), follow your OEM vendor's specific method of updating your microcode/firmware.
Note 2: For the list of specific errata available check Resolution Tab in https://access.redhat.com/security/vulnerabilities/speculativeexecution.
1. Upgrade the manager and reboot.
2. Update each host and reboot.
3. Change the Cluster CPU Type to -IBRS variant.
4. Update each guest and reboot.

Detailed:

1. Upgrade the manager and reboot.
1.1. On the manager, run:

# yum update  rhevm-setup-plugins

1.2. Continue upgrading the manager as described in the upgrade documentation. In order for this fix to be added, engine-setup run is required and a new kernel.
Refer to the Upgrade Guide’s “updates between minor versions” for upgrade flow details 1.
Or the Self Hosted Engine Guide 2.

2. Update each host and reboot.
Follow the "updating virtualization hosts" section in the Upgrade Guide 3

  • Once rebooted, verify the host CPU is identified properly as a model with -IBRS
# virsh -r cpu-models x86_64 | grep IBRS 
  • Within the manager, navigate to Host -> General -> Hardware subtab and verify CPU Type contains -IBRS variant

3. Change the Cluster CPU Type to -IBRS variant:

After upgrading a sufficient number of hypervisors, customers will need to update their clusters or define new clusters with the new hardware types. To change the Cluster CPU Type to the -IBRS variant, in RHVM, navigate to Cluster -> Edit and change CPU Type to -IBRS variant. Note that all hypervisors in a cluster must be running the -IBRS CPU type variant before changing the cluster CPU type.

After updating hypervisors and clusters, power cycle every VM in the environment to use clusters with the new -IBRS CPU types. For convenience, it might make sense to combine this with VM patch and reboot cycles below.

4. Update each guest and reboot.
Note: VMs started/migrated before the Cluster CPU change will keep using the old CPU model
Download the appropriate updates and follow normal patching procedures to apply them. Kernel patches imply a VM reboot. For VMs not already associated with a cluster with the updated hardware type, it might make sense to power each VM off, move to an updated cluster, and then power back on.

Root Cause

Three CVEs were recently made public (CVE-2017-5754, CVE-2017-5753, CVE-2017-5715) which allowed a local attacker to access unauthorized data. CVE-2017-5753 documents the variant of this attack which allows virtualized guests to interact with the host and other guests on the same physical system.

Modern processors make use of parallel execution of instructions to achieve superior performance. The processor uses multiple cache buffers to facilitate smooth flow of instructions and data. The processor core employs different techniques such as Branch Prediction and Speculative Execution to execute instructions, at times even before the correct sequential execution flow is determined. Results of such execution are stored in temporary buffers (cache area), till the time correct sequence of execution is determined.

A targeted side-channel attack was devised to trick the processor into speculatively executing instructions thus allowing unprivileged users to gain access to the cache buffers. Unprivileged guest users could use this flaw to read host memory and/or guest kernel memory.

The first two variants trigger the speculative execution by performing a bounds-check bypass (CVE-2017-5753) or by utilizing branch target injection (CVE-2017-5715). Both variants rely on the presence of a precisely-defined instruction sequence in the privileged code, as well as the fact that memory accesses populate the cache even when the speculatively executed block is being dropped and never committed (executed). As a result, an unprivileged attacker could use these two flaws to read privileged memory by conducting targeted cache side-channel attacks. These variants could be used not only to cross the syscall boundary (variant #1 and variant #2) but also the guest-host boundary (variant #2).

The third variant (CVE-2017-5754, variant #3) relies on the fact that during speculative execution of instruction permission faults, evaluation is suppressed until the retirement of the whole instruction block and that memory accesses populate the cache even when the block is being dropped and never committed (executed). As a result, an unprivileged local attacker could use this flaw to read privileged (kernel space) memory by conducting targeted cache side-channel attacks. A guest VM can not use this issue to read the VM host’s memory.

Virtualized environment has three primary scenarios, unprivileged guest user attempting to access host's memory (guest->host), privileged guest user attempting to access other guest memory (guest->(other)guest) and unprivileged guest user attempting to access guest kernel's memory (guest->(same)guest). To prevent the first two scenarios updates need to be applied on the host and to prevent the third scenario updates need to be applied on both the host and guest system.

Diagnostic Steps

How do I know if my RHV environment has been patched for Meltdown and Spectre CVE?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.