System hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 or RHSA-2020:3217
Red Hat Insights can detect this issue
Environment
Confirmed in the following releases
- Red Hat Enterprise Linux (RHEL) 7.8
- Red Hat Enterprise Linux (RHEL) 8.2
NOT confirmed in the following releases but potentially affected
- Red Hat Enterprise Linux (RHEL) 8.1 EUS
- Red Hat Enterprise Linux (RHEL) 7.9
Issue
UEFI: system hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 and RHSA-2020:3217
Resolution
Red Hat has fixed the bug in the shim packages. Updated shim packages are now available and can be used in conjunction with previously released grub2, fwupd, and fwupdate packages.
Version | Release |
---|---|
RHEL 7.8 | RHBA-2020:3265 |
RHEL 8.0.0 | RHBA-2020:3264 |
RHEL 8.1.0 | RHBA-2020:3263 |
RHEL 8.2.0 | RHBA-2020:3262 |
Recovering the system if it's in a dead state due to having only applied the previous errata
-
Boot the system with the RHEL DVD in Troubleshooting mode
Refer to How to boot a system into rescue mode for details.
-
Set up the network
Refer to Enabling networking in rescue environment without chrooting for details.
-
Enter the chroot
# chroot /mnt/sysimage
-
Apply the errata
# yum -y update shim\*
-
Exit the chroot and reboot
# exit # exit
Diagnostic Steps
-
Boot the system with the RHEL DVD in Troubleshooting mode
Refer to How to boot a system into rescue mode for details.
-
Enter the chroot
# chroot /mnt/sysimage
-
Check the version of
shim
source package# rpm -qa shim-\* --qf "%{SOURCERPM}\n" | sort | uniq
On a RHEL 7 system, the packages from the following source RPMs are affected:
shim-15-7.el7_8.src.rpm shim-signed-15-7.el7_8.src.rpm
On a RHEL 8 system, the packages from the following source RPMs are affected:
shim-15-14.el8_2.src.rpm
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
22 Comments
This fixed the update, reboot, then no grub at all issue. Be aware it will take a while to re-lable everything due to SELinux. Perhaps someone could test prior to pushing out for the world to use. Or are we the customers now Red Hat QA?
I was able to repair a UEFI system after updating grub2. My fix was Follow steps 1 -3 of "Case where the errata has been applied and system rebooted" 4. configured networking to access yum repo 6. 'yum install grub2-efi-x64-modules' 5. 'grub2-install /boot/efi' 6. 'reboot'
I haven't yet had a chance to verify whether installing the grub2-efi-x64-modules rpm prior to updating the grub2 packages and rebooting the system prevents this problem.
Update: The system crashed due to a power bump, and rebooted without issue.
For Satellite users, best way is to exclude 3216 and 3217 in Content View than publish and promote. For Linux admin, don't shoot systems as soon as Errata or Bug fix comes (whoever Vendor is) , wait for 24/48 hours
Can you please tell us if this only affects UEFI boot type operating systems or if BIOS boot type machines are also affected?
And if only UEFI, is it only UEFI with SecureBoot enabled?
As stated above, yes, only EFI, but cases have been seen with SecureBoot disabled as well.
This issue only affects EFI boot.
Thank you Oliver. We got the same confirmation from Red Hat support today.
You're welcome - sorry, I haven't been faster with the answer. There's a lot of information flowing at the moment, as you can probably imagine.
I'm new to using ansible, but have been using playbooks to update my servers. If I have a playbook that looks like this:
Will ansible honor the exclude statement in the /etc/yum.conf file and will exclude grub2* shim* mokutil?
Thanks, Bruce
It should but why take the risk? What if one system doesn't have the exclude? You could use this until you're ready:
Thanks for the advice Donald Davis. As I said, I'm new to Ansible. I did not see the exclude option...Nice!
Best, Bruce Hodges
Is centos7.8 package grub2-2.02-0.86.el7.centos.src.rpm affected?
A bug within shim packages associated with RHSA-2020:3216 & RHSA-2020:3217 has been addressed with RHBA-2020:3262 & RHBA-2020:3265. RHEL 7 - https://access.redhat.com/errata/RHBA-2020:3265 RHEL 8 - https://access.redhat.com/errata/RHBA-2020:3262
So do you think we now can update our systems without hesitation since the shim packages are updated and the boot crash is fixed on rhel 7 and rhel 8? Or will there also soon be newer grub packages that we still have to wait for?
Yes you can now safely update, only Shim was affected finally. Still I would recommend updating 1 system for a given hardware you have then rebooting it to verify there is no issue, then proceed for your other systems with same hardware.
Just tested a RHEL 7.8 system and it rebooted fine. Looks like the latest patches released have resolved this issue.
Got bit by this hard. Before shim update, booted FIPS system from RHEL 7.8 USB in troubleshooting mode, following guidance to reinstall grub2. Learned to use efibootcfg tool, as grub2-install throws an error from efibootcfg.
Followed update guidance, and can not load any signed kernels. System blanks & reboots. Rescue kernel from Dec 19 refuses to run in FIPS mode (can work to build hmac with dracut, but to what end?) - and rescue kernel does not have vfat or raid5 personality baked into initramfs, so I get emergency mode.
Any advice other than overlay FIPS install of RHEL 7.8?
A re-install will be interesting, because anaconda won't do raid of raids out of the box, but I can.
i have simple query for update of the RHSA-2020:3217 . do we have any issue to update the package on our rhel 7 system now.? As earlier there was booting issue. Can i update RHSA-2020:3217 . which covers the below CVE. Number of Target Files:26 Total Target File Size:17.5 MB CVE:CVE-2020-10713; CVE-2020-14308; CVE-2020-14309; CVE-2020-14310; CVE-2020-14311; CVE-2020-15705; CVE-2020-15706; CVE-2020-15707 . As we use bigfix tool to update the package on the server. Kindly let me know is it safe to update the advisory.
Overlay install failed - saw the old yumdb and it went slideways - even after a wipe of /var, /boot and / - no joy on reinstall.
Platform: ASrock EYPCD8-2T, UEFI BIOS v2.30, BMC firmware v2.00.00 (supports legacy or secure booting) * the above configuration will fail *
Update UEFI BIO to v2.60 and BMC firmware to 2.20.00 (secure boot support only) * the factory EFI keys boot RHEL 7.8 install media (USB made by dd if=) in secure mode * * install successful * * yum update from rhel-7-server-rpms & reboot to new kernel & shim-15-8 successful *
If anyone out there is still caught in this snafu, perhaps a bios update may help you?
after upgrading the server its not getting boot
Redhat Team. I have noticed grub2 is not as reliable as it used to be with random corruption after normal patching. I am seeing it break on legacy bios randomly after patching systems which is a slight variant to this. Even SLES 12 has this issue. I would like to see this work better in terms of reliability. Its a real menace if your patching hundreds or thousands of servers in an enterprise setting especially with slow booting physicals with slow java console gui and media :). Please add this to an internal discussion to see if the reliability of grub2 updates following patching can be improved so it stops breaking so often on legacy boots. EUFI is a whole different can of worms but since some of the developers might see this that work on grub2 I wanted to share.