Trying to better understand the microcode_ctl/kernel RHEL releases for Spectre/Meltdown Vulns
Sorry in advance as I don't claim to be any expert on this.
Since the Spectre & Metldown vulns were announced a few weeks ago I've now patched microcode_ctl twice here in the month of Jan 2018.
I don't even believe we've started to patch from VMWare and our hardware for our servers, however I want to stay on top of it from the RHEL side. Other then watching the Performance tab for the RHEL VMs that I manage is there anything else that I can be watching or trying to learn in the meantime to stay on top of this.
I read thru this a few times, however alot of this is over my head:
https://access.redhat.com/articles/3311301
thanks
Responses
Hi Christopher,
The second microcode update of this month was a roll back to the version before ->
"The latest microcode_ctl and linux-firmware packages from Red Hat do not include resolutions to the CVE-2017-5715 (variant 2) exploit. Red Hat is no longer providing microcode to address Spectre, variant 2, due to instabilities introduced that are causing customer systems to not boot. The latest microcode_ctl and linux-firmware packages are reverting these unstable microprocessor firmware changes to versions that were known to be stable and well tested, released prior to the Spectre/Meltdown embargo lift date on Jan 3rd. Customers are advised to contact their silicon vendor to get the latest microcode for their particular processor."
Reference : What CPU microcode is available via the microcode_ctl package to mitigate CVE-2017-5715 (variant 2)?
What can you do ?
- Check the current vulnerability status with the Spectre And Meltdown Detector.
- Continuously apply all currently available stable software updates from Red Hat.
- Keep yourself informed about solutions regarding Kernel Side-Channel Attacks.
Regards,
Christian
To add to the 'What can you do?'
You should be keeping up to date with processor microcode updates from upstream vendors (eg. Intel) as Red Hat have suggested. I think this latest microcode update issue has 'bitten' Red Hat and as a result we will probably see less microcode updates being packaged/passed on.
Hi ! :)
Thank you for adding this point ... you are right, and me, I too assume that Red Hat will be much more careful with providing new microcode releases after what has happened. I checked the Intel Download Center and saw that they pulled the January release as well. But I think that when things calm down, Red Hat will continue to provide current microcode updates, because some vulnerabilities such like Spectre (I expect more of that kind) can only be mitigated completely by applying fixed microcode.
Regards,
Christian
All 3 variants are primarily risky for hosting providers with multiple tenants and for end-user desktops doing web browsing.
In-house servers, where you control the hardware, OS (bare metal or hypervisor), and the application code running on a server, are at low risk. Pretty much all 3 of the attacks depend on using speculatively executed code to return data which can be used as an index into an array; walking the array and observing the cache timing can then be used to infer what the speculated data value was even though the abusive code can't read it directly.
What you need to do in January 2018:
a) update everything (Linux, Windows, Android, iOS, OS-X, ...) with new vendor kernels to mitigate meltdown entirely and spectre 1 partially
b) update web browsers with timer degradation mitigation, wich makes it really hard to exploit spectre 1 and spectre 2, as they depend on distinguishing cached vs uncached memory access timing. Firefox, IE/Edge, Chrome, Safari, ... all the browsers on all the platforms.
c) AVOID installing new CPU microcode either from a BIOS update or from your OS vendor, as intel is warning their first try was buggy and you might crash a lot
CVE-2017-5754 / rogue data cache load / Meltdown / variant 3 Uses speculative execution to leak kernel-mapped memory, which tends to be almost all of physical memory. E.g. hypervisor guests could not only read their kernel secrets, they might be able to read other guests too. The workaround is to not map kernel memory into user processes, and on older CPU's which don't have process tags on TLB[0] entries, to flush the TLB with every system call into the kernel. So you need a new kernel with "page table isolation" and a reboot, and incur a modest performance penalty which depends on the CPU model and on how many system calls the app makes. Biggest risk is on intel CPU's, but other architectures can have it too.
CVE-2017-5753 / bounds check bypass / Spectre Variant 1 Uses speculative execution to leak memory based on code after conditional branches. Mitigated on servers to a significant extent with modest performance impact by recompiling applications and kernels with extra LFENCE (or equivalent) synchronization instructions in key places. E.g. redhat kernels added about 100 so far. Mitigated on desktops to a large extent by degrading browser timing (there are proof of concept exploits written in javascript alone). Affects most CPU architectures.
CVE-2017-5715 / branch target injection / Spectre Variant 2 Uses speculative execution to leak memory by mis-predicting indirect branches. Indirect branches are very common in both OS kernels and object oriented applications, so the vulnerability is significant. Unfortunately, this one is both very hard to fix (requires new CPU hardware designs in the long run, and at least new CPU microcode in the short run) and the fixes have very high performance penalties, so mostly vendors are basically ignoring this one this month; it will be an ongoing source of pain as 2018 progresses. Intel's first attempts at new CPU microcode are going badly, hence temporary "no update" advice from Intel and microcode package reversions by Redhat. Major cloud vendors are partially mitigating it on many CPU steppings[1] at low performance penalty (1-2%) for OS and hypervisor protection only by using bleeding edge compilers which implement google's "retpoline" workaround[2]; web browser vendors are significantly mitigating it by degrading browser timing. Affects most CPU architectures. Full mitigation for cloud providers requires all four of new CPU microcode, new compiler support, new OS support, and recompiled applications, which is why we are still mostly vulnerable.
Hope this clarify things a bit, -- Jim Leinweber
[0] Translation Lookaside Buffers cache the results of virtual memory address translations, which avoids doing page table walks 2-3x per instruction if the application has favorable locality of reference. Code using virtual addresses (pretty much all of it) runs 5x-90x faster with TLB hardware assistance.
[1] Different CPU's and even different steppings (variants) of the same CPU model have subtle but important differences in the relationships between branch target buffers, return target stacks, and other internal CPU hardware features which unfortunately matter to the workarounds. Assembly language code can often contain up to 30% branches and call/return pairings, so turning off the prediction hardware entirely or even just flushing the prediction buffers frequently can cause Really Nasty slowdowns.
[2] On some but not all CPU steppings return target stacks are less exploitable than branch target buffers, and for those you can use a compiler gimmick to convert indirect branches into call/return pairs at a low performance cost. This helps a lot for protecting OS kernels and cloud hypervisors, but still leaves current applications vulnerable to sandbox escapes.
A month has gone by - any updates? If I chose to put intel-ucode in place, do i need to uninstall the firmware package from Redhat? ( not kernel-firmware, but microcode_ctl)?
root@sapscdci # rpm -ql microcode_ctl
/lib/firmware/amd-ucode
/lib/firmware/amd-ucode/microcode_amd.bin
/lib/firmware/amd-ucode/microcode_amd_fam15h.bin
/lib/firmware/microcode.dat
/lib/udev/rules.d/89-microcode.rules
/sbin/microcode_ctl
/usr/share/doc/microcode_ctl-1.17
/usr/share/doc/microcode_ctl-1.17/INSTALL.microcode_amd
/usr/share/doc/microcode_ctl-1.17/LICENSE.microcode_amd
/usr/share/doc/microcode_ctl-1.17/README.microcode_amd
/usr/share/man/man8/microcode_ctl.8.gz
root@sapscdci #
Hi John,
As of today there is no new microcode available, the latest Linux Microcode Data File is still the one from 2017-11-17.
All you would need to do is replace the microcode firmware files in the /usr/lib/firmware/intel-ucode folder.
Download the Linux Processor Microcode Data File from the Intel website ... extract the file and copy the content to that folder. Then execute sudo dracut -f in order to get the new microcode loaded after the next system reboot.
What makes me wonder though is the output of the command you've provided, do you possibly have an AMD CPU ?
If this is the case, it wouldn't make sense to deploy Intel firmware. For comparison, here the output from my system :
$ sudo rpm -ql microcode_ctl
/lib/firmware/intel-ucode
/lib/firmware/intel-ucode/06-03-02
/lib/firmware/intel-ucode/06-05-00
/lib/firmware/intel-ucode/06-05-01
/lib/firmware/intel-ucode/06-05-02
/lib/firmware/intel-ucode/06-05-03
/lib/firmware/intel-ucode/06-06-00
/lib/firmware/intel-ucode/06-06-05
/lib/firmware/intel-ucode/06-06-0a
/lib/firmware/intel-ucode/06-06-0d
/lib/firmware/intel-ucode/06-07-01
/lib/firmware/intel-ucode/06-07-02
/lib/firmware/intel-ucode/06-07-03
/lib/firmware/intel-ucode/06-08-01
/lib/firmware/intel-ucode/06-08-03
/lib/firmware/intel-ucode/06-08-06
/lib/firmware/intel-ucode/06-08-0a
/lib/firmware/intel-ucode/06-09-05
/lib/firmware/intel-ucode/06-0a-00
/lib/firmware/intel-ucode/06-0a-01
/lib/firmware/intel-ucode/06-0b-01
/lib/firmware/intel-ucode/06-0b-04
/lib/firmware/intel-ucode/06-0d-06
/lib/firmware/intel-ucode/06-0e-08
/lib/firmware/intel-ucode/06-0e-0c
/lib/firmware/intel-ucode/06-0f-02
/lib/firmware/intel-ucode/06-0f-06
/lib/firmware/intel-ucode/06-0f-07
/lib/firmware/intel-ucode/06-0f-0a
/lib/firmware/intel-ucode/06-0f-0b
/lib/firmware/intel-ucode/06-0f-0d
/lib/firmware/intel-ucode/06-16-01
/lib/firmware/intel-ucode/06-17-06
/lib/firmware/intel-ucode/06-17-07
/lib/firmware/intel-ucode/06-17-0a
/lib/firmware/intel-ucode/06-1a-04
/lib/firmware/intel-ucode/06-1a-05
/lib/firmware/intel-ucode/06-1c-02
/lib/firmware/intel-ucode/06-1c-0a
/lib/firmware/intel-ucode/06-1d-01
/lib/firmware/intel-ucode/06-1e-05
/lib/firmware/intel-ucode/06-25-02
/lib/firmware/intel-ucode/06-25-05
/lib/firmware/intel-ucode/06-26-01
/lib/firmware/intel-ucode/06-2a-07
/lib/firmware/intel-ucode/06-2d-06
/lib/firmware/intel-ucode/06-2d-07
/lib/firmware/intel-ucode/06-2f-02
/lib/firmware/intel-ucode/06-3a-09
/lib/firmware/intel-ucode/06-3c-03
/lib/firmware/intel-ucode/06-3d-04
/lib/firmware/intel-ucode/06-3e-04
/lib/firmware/intel-ucode/06-3e-06
/lib/firmware/intel-ucode/06-3e-07
/lib/firmware/intel-ucode/06-3f-02
/lib/firmware/intel-ucode/06-3f-04
/lib/firmware/intel-ucode/06-45-01
/lib/firmware/intel-ucode/06-46-01
/lib/firmware/intel-ucode/06-47-01
/lib/firmware/intel-ucode/06-4e-03
/lib/firmware/intel-ucode/06-4f-01
/lib/firmware/intel-ucode/06-56-02
/lib/firmware/intel-ucode/06-56-03
/lib/firmware/intel-ucode/06-56-04
/lib/firmware/intel-ucode/06-5e-03
/lib/firmware/intel-ucode/0f-00-07
/lib/firmware/intel-ucode/0f-00-0a
/lib/firmware/intel-ucode/0f-01-02
/lib/firmware/intel-ucode/0f-02-04
/lib/firmware/intel-ucode/0f-02-05
/lib/firmware/intel-ucode/0f-02-06
/lib/firmware/intel-ucode/0f-02-07
/lib/firmware/intel-ucode/0f-02-09
/lib/firmware/intel-ucode/0f-03-02
/lib/firmware/intel-ucode/0f-03-03
/lib/firmware/intel-ucode/0f-03-04
/lib/firmware/intel-ucode/0f-04-01
/lib/firmware/intel-ucode/0f-04-03
/lib/firmware/intel-ucode/0f-04-04
/lib/firmware/intel-ucode/0f-04-07
/lib/firmware/intel-ucode/0f-04-08
/lib/firmware/intel-ucode/0f-04-09
/lib/firmware/intel-ucode/0f-04-0a
/lib/firmware/intel-ucode/0f-06-02
/lib/firmware/intel-ucode/0f-06-04
/lib/firmware/intel-ucode/0f-06-05
/lib/firmware/intel-ucode/0f-06-08
/usr/lib/dracut/dracut.conf.d/01-microcode.conf
/usr/lib/systemd/system/microcode.service
/usr/sbin/intel-microcode2ucode
/usr/share/doc/microcode_ctl/README
/usr/share/doc/microcode_ctl/disclaimer
Regards,
Christian
Any update on this? It seems that Microsoft has finally started to include these Microcode Updates in Windows Updates (again).
Would appreciate some feedback from RedHat about this.
Hi ! :)
Intel has released a 'fresh' Linux Processor Microcode Data File a few days ago. The fedora team is already testing it and it has received positive feedback so far. I am testing the new microcode on my CentOS / fedora / RHEL systems and can confirm that it is running without issues until now. So I assume that the Red Hat security team is testing too and will release it as soon as possible, once they are convinced that no negative impacts on production systems are to be expected.
Regards,
Christian
Intel's latest firmware seems to be non-buggy, as far as I have read and seen for myself on some systems.
Having updated microcode_ctl will make spectre mitigation much easier for older systems. Is there any official word on when this will be available?
Thanks, Corey
Same question here - if this microcode patch from Intel is stable, why doesn't RHEL incorporate it into a new version of microcode_ctl? That will promulgate the mitigation a lot more effectively than people having to pull it down from the Intel site themselves and figure out the steps to install it.
I downloaded the spectre/meltdown checker from: https://access.redhat.com/labs/speculativeexecution/
My Dell blade is BIOS patched to latest, which should include applicable microcode updates, and I'm running ESXi 6.5 with the latest patches that supposedly handle the Spectre/Meltdown variants, but for whatever reason, my RHEL6 VM (which is also supposedly fully patched) is telling me that for Variant #2 (Spectre): - HW support / updated microcode: NO
Should I be chasing Redhat, Dell, or VMware about this? It's slightly confusing...
Red Hat has released microcode_ctl-1.17-25.7.el6_9.x86_64.rpm for RHEL 6 on May 30 with the notes: AMD CPU microcode has been updated to the latest upstream version, which also provides new capabilities used by the Linux kernel to enhance mitigation of CVE-2017-5715, known as the Spectre variant 2 vulnerability. (BZ#1576314)
When are these AMD microcode updates going to be released in a microcode_ctl update for RHEL 7?
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
