System hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 or RHSA-2020:3217

Solution Verified - Updated -

Red Hat Insights can detect this issue

Proactively detect and remediate issues impacting your systems.
View matching systems and remediation

Environment

Confirmed in the following releases

  • Red Hat Enterprise Linux (RHEL) 7.8
  • Red Hat Enterprise Linux (RHEL) 8.2

NOT confirmed in the following releases but potentially affected

  • Red Hat Enterprise Linux (RHEL) 8.1 EUS
  • Red Hat Enterprise Linux (RHEL) 7.9

Issue

UEFI: system hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 and RHSA-2020:3217

Resolution

Red Hat has fixed the bug in the shim packages. Updated shim packages are now available and can be used in conjunction with previously released grub2, fwupd, and fwupdate packages.

Version Release
RHEL 7.8 RHBA-2020:3265
RHEL 8.0.0 RHBA-2020:3264
RHEL 8.1.0 RHBA-2020:3263
RHEL 8.2.0 RHBA-2020:3262

Recovering the system if it's in a dead state due to having only applied the previous errata

  1. Boot the system with the RHEL DVD in Troubleshooting mode

    Refer to How to boot a system into rescue mode for details.

  2. Set up the network

    Refer to Enabling networking in rescue environment without chrooting for details.

  3. Enter the chroot

    # chroot /mnt/sysimage

  4. Apply the errata

    # yum -y update shim\*

  5. Exit the chroot and reboot

    # exit
# exit

Diagnostic Steps

  1. Boot the system with the RHEL DVD in Troubleshooting mode

    Refer to How to boot a system into rescue mode for details.

  2. Enter the chroot

    # chroot /mnt/sysimage

  3. Check the version of shim source package

    # rpm -qa shim-\* --qf "%{SOURCERPM}\n" | sort | uniq

    On a RHEL 7 system, the packages from the following source RPMs are affected:

    shim-15-7.el7_8.src.rpm
shim-signed-15-7.el7_8.src.rpm

    On a RHEL 8 system, the packages from the following source RPMs are affected:

    shim-15-14.el8_2.src.rpm

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

22 Comments

Log in to comment
TS Newbie 7 points
30 July 2020 7:38 PM

This fixed the update, reboot, then no grub at all issue. Be aware it will take a while to re-lable everything due to SELinux. Perhaps someone could test prior to pushing out for the world to use. Or are we the customers now Red Hat QA?

EJ Newbie 10 points
30 July 2020 10:18 PM

I was able to repair a UEFI system after updating grub2. My fix was Follow steps 1 -3 of "Case where the errata has been applied and system rebooted" 4. configured networking to access yum repo 6. 'yum install grub2-efi-x64-modules' 5. 'grub2-install /boot/efi' 6. 'reboot'

I haven't yet had a chance to verify whether installing the grub2-efi-x64-modules rpm prior to updating the grub2 packages and rebooting the system prevents this problem.

Update: The system crashed due to a power bump, and rebooted without issue.

MM Newbie 10 points
30 July 2020 11:53 PM

For Satellite users, best way is to exclude 3216 and 3217 in Content View than publish and promote. For Linux admin, don't shoot systems as soon as Errata or Bug fix comes (whoever Vendor is) , wait for 24/48 hours

FN Community Member 38 points
31 July 2020 7:10 AM

Can you please tell us if this only affects UEFI boot type operating systems or if BIOS boot type machines are also affected?

PR Community Member 77 points
31 July 2020 1:35 PM

And if only UEFI, is it only UEFI with SecureBoot enabled?

Oliver Falk's picture Red Hat Active Contributor 206 points
31 July 2020 1:42 PM

As stated above, yes, only EFI, but cases have been seen with SecureBoot disabled as well.

Oliver Falk's picture Red Hat Active Contributor 206 points
31 July 2020 1:41 PM

This issue only affects EFI boot.

FN Community Member 38 points
31 July 2020 1:51 PM

Thank you Oliver. We got the same confirmation from Red Hat support today.

Oliver Falk's picture Red Hat Active Contributor 206 points
31 July 2020 1:56 PM

You're welcome - sorry, I haven't been faster with the answer. There's a lot of information flowing at the moment, as you can probably imagine.

BH Community Member 93 points
31 July 2020 7:10 PM

I'm new to using ansible, but have been using playbooks to update my servers. If I have a playbook that looks like this:

---
- hosts: allhosts
  become: yes

  tasks:
   - name: Perform a yum update
     yum:
       name: '*'
       state: latest
       update_cache: yes

Will ansible honor the exclude statement in the /etc/yum.conf file and will exclude grub2* shim* mokutil?

Thanks, Bruce

DD Newbie 12 points
31 July 2020 9:37 PM

It should but why take the risk? What if one system doesn't have the exclude? You could use this until you're ready:

---
- hosts: allhosts
  become: yes

  tasks:
   - name: Perform a yum update
     yum:
       name: '*'
       state: latest
       update_cache: yes
       exclude: grub2* shim* mokutil
BH Community Member 93 points
2 August 2020 12:31 AM

Thanks for the advice Donald Davis. As I said, I'm new to Ansible. I did not see the exclude option...Nice!

Best, Bruce Hodges

NA Newbie 7 points
1 August 2020 4:47 AM

Is centos7.8 package grub2-2.02-0.86.el7.centos.src.rpm affected?

Cliff Perry's picture Red Hat Community Member 25 points
1 August 2020 1:26 PM

A bug within shim packages associated with RHSA-2020:3216 & RHSA-2020:3217 has been addressed with RHBA-2020:3262 & RHBA-2020:3265. RHEL 7 - https://access.redhat.com/errata/RHBA-2020:3265 RHEL 8 - https://access.redhat.com/errata/RHBA-2020:3262

PJ Newbie 5 points
3 August 2020 6:42 AM

So do you think we now can update our systems without hesitation since the shim packages are updated and the boot crash is fixed on rhel 7 and rhel 8? Or will there also soon be newer grub packages that we still have to wait for?

Renaud Metrich's picture Red Hat Pro 485 points
3 August 2020 6:56 AM

Yes you can now safely update, only Shim was affected finally. Still I would recommend updating 1 system for a given hardware you have then rebooting it to verify there is no issue, then proceed for your other systems with same hardware.

Jamie Butler's picture Active Contributor 263 points
3 August 2020 5:35 PM

Just tested a RHEL 7.8 system and it rebooted fine. Looks like the latest patches released have resolved this issue.

MM Active Contributor 112 points
3 August 2020 8:38 PM

Got bit by this hard. Before shim update, booted FIPS system from RHEL 7.8 USB in troubleshooting mode, following guidance to reinstall grub2. Learned to use efibootcfg tool, as grub2-install throws an error from efibootcfg.

Followed update guidance, and can not load any signed kernels. System blanks & reboots. Rescue kernel from Dec 19 refuses to run in FIPS mode (can work to build hmac with dracut, but to what end?) - and rescue kernel does not have vfat or raid5 personality baked into initramfs, so I get emergency mode.

Any advice other than overlay FIPS install of RHEL 7.8?

A re-install will be interesting, because anaconda won't do raid of raids out of the box, but I can.

LS Newbie 5 points
4 August 2020 12:36 PM

i have simple query for update of the RHSA-2020:3217 . do we have any issue to update the package on our rhel 7 system now.? As earlier there was booting issue. Can i update RHSA-2020:3217 . which covers the below CVE. Number of Target Files:26 Total Target File Size:17.5 MB CVE:CVE-2020-10713; CVE-2020-14308; CVE-2020-14309; CVE-2020-14310; CVE-2020-14311; CVE-2020-15705; CVE-2020-15706; CVE-2020-15707 . As we use bigfix tool to update the package on the server. Kindly let me know is it safe to update the advisory.

MM Active Contributor 112 points
6 August 2020 12:12 AM

Overlay install failed - saw the old yumdb and it went slideways - even after a wipe of /var, /boot and / - no joy on reinstall.

Platform: ASrock EYPCD8-2T, UEFI BIOS v2.30, BMC firmware v2.00.00 (supports legacy or secure booting) * the above configuration will fail *

Update UEFI BIO to v2.60 and BMC firmware to 2.20.00 (secure boot support only) * the factory EFI keys boot RHEL 7.8 install media (USB made by dd if=) in secure mode * * install successful * * yum update from rhel-7-server-rpms & reboot to new kernel & shim-15-8 successful *

If anyone out there is still caught in this snafu, perhaps a bios update may help you?

SS Newbie 11 points
5 October 2020 3:47 PM

after upgrading the server its not getting boot

JU Newbie 15 points
21 February 2022 12:07 AM

Redhat Team. I have noticed grub2 is not as reliable as it used to be with random corruption after normal patching. I am seeing it break on legacy bios randomly after patching systems which is a slight variant to this. Even SLES 12 has this issue. I would like to see this work better in terms of reliability. Its a real menace if your patching hundreds or thousands of servers in an enterprise setting especially with slow booting physicals with slow java console gui and media :). Please add this to an internal discussion to see if the reliability of grub2 updates following patching can be improved so it stops breaking so often on legacy boots. EUFI is a whole different can of worms but since some of the developers might see this that work on grub2 I wanted to share.