System hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 or RHSA-2020:3217

Solution Verified - Updated -

Environment

Confirmed in the following releases

  • Red Hat Enterprise Linux (RHEL) 7.8
  • Red Hat Enterprise Linux (RHEL) 8.2

NOT confirmed in the following releases but potentially affected

  • Red Hat Enterprise Linux (RHEL) 8.1 EUS
  • Red Hat Enterprise Linux (RHEL) 7.9

Issue

UEFI: system hangs after POST and the grub menu never loads after applying the RHSA-2020:3216 and RHSA-2020:3217

Resolution

Red Hat has fixed the bug in the shim packages. Updated shim packages are now available and can be used in conjunction with previously released grub2, fwupd, and fwupdate packages.

Version Release
RHEL 7.8 RHBA-2020:3265
RHEL 8.0.0 RHBA-2020:3264
RHEL 8.1.0 RHBA-2020:3263
RHEL 8.2.0 RHBA-2020:3262

Recovering the system if it's in a dead state due to having only applied the previous errata

  1. Boot the system with the RHEL DVD in Troubleshooting mode

    Refer to How to boot a system into rescue mode for details.

  2. Set up the network

    Refer to Enabling networking in rescue environment without chrooting for details.

  3. Enter the chroot

    # chroot /mnt/sysimage
    
  4. Apply the errata

    # yum -y update shim\*
    
  5. Exit the chroot and reboot

    # exit
    # exit
    

Diagnostic Steps

  1. Boot the system with the RHEL DVD in Troubleshooting mode

    Refer to How to boot a system into rescue mode for details.

  2. Enter the chroot

    # chroot /mnt/sysimage
    
  3. Check the version of shim source package

    # rpm -qa shim-\* --qf "%{SOURCERPM}\n" | sort | uniq
    

    On a RHEL 7 system, the packages from the following source RPMs are affected:

    shim-15-7.el7_9.src.rpm
    shim-signed-15-7.el7_8.src.rpm
    

    On a RHEL 8 system, the packages from the following source RPMs are affected:

    shim-15-14.el8_2.src.rpm
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

20 Comments

This fixed the update, reboot, then no grub at all issue. Be aware it will take a while to re-lable everything due to SELinux. Perhaps someone could test prior to pushing out for the world to use. Or are we the customers now Red Hat QA?

I was able to repair a UEFI system after updating grub2. My fix was Follow steps 1 -3 of "Case where the errata has been applied and system rebooted" 4. configured networking to access yum repo 6. 'yum install grub2-efi-x64-modules' 5. 'grub2-install /boot/efi' 6. 'reboot'

I haven't yet had a chance to verify whether installing the grub2-efi-x64-modules rpm prior to updating the grub2 packages and rebooting the system prevents this problem.

Update: The system crashed due to a power bump, and rebooted without issue.

For Satellite users, best way is to exclude 3216 and 3217 in Content View than publish and promote. For Linux admin, don't shoot systems as soon as Errata or Bug fix comes (whoever Vendor is) , wait for 24/48 hours

Can you please tell us if this only affects UEFI boot type operating systems or if BIOS boot type machines are also affected?

And if only UEFI, is it only UEFI with SecureBoot enabled?

As stated above, yes, only EFI, but cases have been seen with SecureBoot disabled as well.

This issue only affects EFI boot.

Thank you Oliver. We got the same confirmation from Red Hat support today.

You're welcome - sorry, I haven't been faster with the answer. There's a lot of information flowing at the moment, as you can probably imagine.

I'm new to using ansible, but have been using playbooks to update my servers. If I have a playbook that looks like this:

---
- hosts: allhosts
  become: yes

  tasks:
   - name: Perform a yum update
     yum:
       name: '*'
       state: latest
       update_cache: yes

Will ansible honor the exclude statement in the /etc/yum.conf file and will exclude grub2* shim* mokutil?

Thanks, Bruce

It should but why take the risk? What if one system doesn't have the exclude? You could use this until you're ready:

---
- hosts: allhosts
  become: yes

  tasks:
   - name: Perform a yum update
     yum:
       name: '*'
       state: latest
       update_cache: yes
       exclude: grub2* shim* mokutil

Thanks for the advice Donald Davis. As I said, I'm new to Ansible. I did not see the exclude option...Nice!

Best, Bruce Hodges

Is centos7.8 package grub2-2.02-0.86.el7.centos.src.rpm affected?

A bug within shim packages associated with RHSA-2020:3216 & RHSA-2020:3217 has been addressed with RHBA-2020:3262 & RHBA-2020:3265. RHEL 7 - https://access.redhat.com/errata/RHBA-2020:3265 RHEL 8 - https://access.redhat.com/errata/RHBA-2020:3262

So do you think we now can update our systems without hesitation since the shim packages are updated and the boot crash is fixed on rhel 7 and rhel 8? Or will there also soon be newer grub packages that we still have to wait for?

Yes you can now safely update, only Shim was affected finally. Still I would recommend updating 1 system for a given hardware you have then rebooting it to verify there is no issue, then proceed for your other systems with same hardware.

Just tested a RHEL 7.8 system and it rebooted fine. Looks like the latest patches released have resolved this issue.

Got bit by this hard. Before shim update, booted FIPS system from RHEL 7.8 USB in troubleshooting mode, following guidance to reinstall grub2. Learned to use efibootcfg tool, as grub2-install throws an error from efibootcfg.

Followed update guidance, and can not load any signed kernels. System blanks & reboots. Rescue kernel from Dec 19 refuses to run in FIPS mode (can work to build hmac with dracut, but to what end?) - and rescue kernel does not have vfat or raid5 personality baked into initramfs, so I get emergency mode.

Any advice other than overlay FIPS install of RHEL 7.8?

A re-install will be interesting, because anaconda won't do raid of raids out of the box, but I can.

i have simple query for update of the RHSA-2020:3217 . do we have any issue to update the package on our rhel 7 system now.? As earlier there was booting issue. Can i update RHSA-2020:3217 . which covers the below CVE. Number of Target Files:26 Total Target File Size:17.5 MB CVE:CVE-2020-10713; CVE-2020-14308; CVE-2020-14309; CVE-2020-14310; CVE-2020-14311; CVE-2020-15705; CVE-2020-15706; CVE-2020-15707 . As we use bigfix tool to update the package on the server. Kindly let me know is it safe to update the advisory.

Overlay install failed - saw the old yumdb and it went slideways - even after a wipe of /var, /boot and / - no joy on reinstall.

Platform: ASrock EYPCD8-2T, UEFI BIOS v2.30, BMC firmware v2.00.00 (supports legacy or secure booting) * the above configuration will fail *

Update UEFI BIO to v2.60 and BMC firmware to 2.20.00 (secure boot support only) * the factory EFI keys boot RHEL 7.8 install media (USB made by dd if=) in secure mode * * install successful * * yum update from rhel-7-server-rpms & reboot to new kernel & shim-15-8 successful *

If anyone out there is still caught in this snafu, perhaps a bios update may help you?