kernel panic while booting after updating to kernel-3.10.0-693.1.1.el7.x86_64

Latest response

Everything was OK with kernel-3.10.0-514.26.2.el7.x86_64 and old versions, but after i update it to kernel-3.10.0-693.el7.x86_64 and kernel-3.10.0-693.1.1.el7.x86_64 OS failed to boot with CapsLock keep blinking.
the crash starts with
atombios stuck in loop for more than 5secs aborting

and end with
kernel panic not syncing : fatal exception
Kernel Offset : disabled

kernel-3.10.0-693.el7.x86_64 Image :
https://www.dropbox.com/s/f6gok5s23v4ib52/kernel-3.10.0-693.el7.x86_64.jpg?dl=0

kernel-3.10.0-693.1.1.el7.x86_64 Images :
https://www.dropbox.com/s/eyh7sfv7hftfs83/kernel-3.10.0-693.1.1.el7.x86_64_1.jpg?dl=0
https://www.dropbox.com/s/9fiyogxfj1w9ur7/kernel-3.10.0-693.1.1.el7.x86_64_2.jpg?dl=0

forget to mention that every time i boot with my working kernel a notification with kernel issues and cannot be reported because it is a hardware problem :
$abrt-cli list --since 1504030233
reason: mce: [Hardware Error]: Machine check events logged
time: Tue 29 Aug 2017 08:14:02 PM EET
cmdline: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.26.2.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rhgb quiet LANG=en_US.UTF-8
uid: 0
Directory: /var/spool/abrt/oops-2017-08-29-20:14:02-774-0

my laptop is :
Dell inspiron 5567 i7-7500U
Intel® Core™ i7-7500U CPU @ 2.70GHz × 4
Graphic AMD R7 M445 4G

any help please

HA
Log in to join the conversation

Responses

Jamie Bainbridge's picture Red Hat Guru 7682 points
29 August 2017 3:05 AM

I found a few more references to this around the internet, it seems to be an amdgpu regression which is resolved in 4.10 and later, eg: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1656649

I logged Red Hat Private Bug 1486100 - amdgpu Radeon R7 panic in smu7_populate_single_firmware_entry.isra.3 to address this in RHEL.

For now, you could use the previous working kernel as a workaround. You might also have luck using the latest ELRepo kernel-ml package if you want to try that, though it is not supported by us.

Christian Labisch's picture Guru 13625 points
29 August 2017 8:28 AM

Hi Jamie,

Alternatively Hassan can disable the AMD GPU by adding radeon.modeset=0 to the GRUB_CMDLINE_LINUX line in the /etc/default/grub file until this issue will be fixed, and then he can use the latest available Red Hat kernel. :)

Regards,
Christian

HA Newbie
29 August 2017 6:03 PM

I'm thinking of excluding any kernel* updates in yum.conf because if kernel 3.10.0-514.26.2 been replaced i'll not be able to boot again until i found a solution

Christian Labisch's picture Guru 13625 points
30 August 2017 9:33 AM

Hi Hassan,

Good idea ... please check what I wrote in my comment concerning AMD / open source GPU drivers. :)

Regards,
Christian

Christian Labisch's picture Guru 13625 points
29 August 2017 8:53 AM

Hi Hassan,

In case that you want to choose the alternative solution I have posted in my reply to Jamie's suggestion above, please don't forget to update the GRUB configuration afterwards by executing sudo grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg. Good luck !

Cheers :)
Christian

HA Newbie
29 August 2017 6:23 PM

I've tried what you have suggested
my /etc/default/grub

GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rhgb quiet radeon.modeset=0"
GRUB_DISABLE_RECOVERY="true"

and update the GRUB configuration noting change

and when i add radeon.modeset=0 amdgpu.runpm=0 acpi_backlight=intel_backlight to GRUB_CMDLINE_LINUX the only change that happen is the line iwlwifi capa flags index 3 larger than supported by driver appears

Images : https://www.dropbox.com/s/8hoxtr1op1yeefk/kernel_1.jpg?dl=0 https://www.dropbox.com/s/jb1q5x4q5kfml5h/kernel_2.jpg?dl=0

Christian Labisch's picture Guru 13625 points
30 August 2017 9:02 AM

Hi Hassan,

I did some further investigation and have a few questions. Did you install the AMD drivers that are provided on the AMD website ? In case yes, I can't find your AMD R7 M445 4G GPU on the list of supported products -> AMD Linux Download Center Also, I can't find Linux drivers which already support RHEL 7.4, the latest supported version as of today is RHEL 7.3, so my advice would be to remove these drivers and check if the open source radeon drivers work.

Please check if you have these drivers installed : xorg-x11-drv-ati, and in case not, install them and report back if the system boots with these drivers (with and without the parameter radeon.modeset=0). Reboot the system after you uninstalled the AMD drivers and in case you reinstalled the open source drivers, reboot the system afterwards again.

Best practice is to do everything with GUI disabled from a virtual console :
sudo systemctl set-default multi-user.target and sudo reboot
Once you have finished the drivers installation - switch back to GUI mode : sudo systemctl set-default graphical.target and sudo reboot

Regards,
Christian

HA Newbie
30 August 2017 3:55 PM

I've the xorg* already installed but i didn't try to install any AMD amdgpu* package, could this be the problem !! should i try to install the EL7.3 amdgpu version and see what will happens

Christian Labisch's picture Guru 13625 points
30 August 2017 4:11 PM

Why not Hassan ? It cannot get worse - but I recommend to create a system backup before, otherwise you might have to reinstall the whole system ... drivers installation is not a trivial thing. I recommend to do it with Clonezilla, I am using this tool for years and it never let me down. I made presentations that show how to do it : Clonezilla - Create partition backup | Clonezilla - Restore partition backup

Regards,
Christian

HA Newbie
30 August 2017 4:48 PM

Sorry i didn't mention that, my old labtop didn't have external GPU and this is my first linux installation with GPU card

Christian Labisch's picture Guru 13625 points
30 August 2017 4:57 PM

No problem Hassan, drivers management for hybrid GPU solutions is something special. Things that work on one machine may not necessarily work on other machines. AMD graphics generally are a bit more complicated than NVIDIA graphics, so you have to try things out, if the AMD drivers don't work properly, we will have to wait until the issue is fixed in one of the next kernel releases - normally the radeon drivers work fine. By the way, I've checked the notebook specifications before responding. :)

Regards,
Christian

Jamie Bainbridge's picture Red Hat Guru 7682 points
30 August 2017 12:51 AM

We've had another customer report the same issue overnight. Setting radeon.modeset=0 didn't work for them either.

I've raised the priority of the bug and we're tracking this on the knowledgebase at:

Christian Labisch's picture Guru 13625 points
30 August 2017 9:25 AM

Hi Jamie,
Thanks :)

This will be much appreciated by many users ... AMD graphics drivers on Linux systems really are a pain these days.

Regards,
Christian

Jamie Bainbridge's picture Red Hat Guru 7682 points
31 August 2017 12:19 AM

No worries, solving problems is what we're here for :)

I am also an AMD graphics fan at home. I haven't had any experience with amdgpu yet, but the old radeon driver has been very reliable for me for a long run of kernels. I gave up on the proprietary fglrx driver ages ago, last I checked it didn't even work with current Xorg.

Christian Labisch's picture Guru 13625 points
31 August 2017 8:45 AM

Hi Jamie,

I can confirm the same, I have installed RHEL 7.4 on my old DELL Studio machine with a dedicated AMD graphics chip and that thing is running perfectly fine with the open source radeon drivers. :)

Regards,
Christian

TM Community Member 51 points
5 September 2017 5:33 PM

We've seen the same issue with two RHEL 7.4 VMWare guests that were recently rebooted with the 3.10.0-693.1.1.el7 kernel. We discovered independently (before finding this issue) that booting back into an older kernel works around the issue.

JF Newbie 5 points
26 October 2017 6:00 AM

Hey Tim, did you find a working solution? We have the same issue with an ESXi 6.5 Client (RHEL 7.2 and 7.4). But the only working "Kernel" is ther rescue boot-option.

TM Community Member 51 points
27 October 2017 7:26 PM

We haven't found any reasonable solution. I initially indicated in my response that Red Hat had not even acknowledged this problem with a Knowledgebase article, but that was incorrect. Jamie links to the article, above. I guess we go with one of the two options listed there, until there's a final recommendation.

We've experienced the problem on both physical hardware (headless servers) and VM guests -- systems without any AMD GPU. Right now, we're very hesitant about rebooting any of our systems, as there's a strong chance the system will experience this problem.

Trevor Hemsley's picture Community Member 45 points
23 October 2017 9:02 PM

I've now seen two separate reports that adding nomodeset - not radeon.modeset=0 - fixes this.

WH Newbie 9 points
18 December 2017 1:03 AM

We have unfortunately found same here - about 50 out of 141 running RHEL7 VMs exhibited same issue after kernel-3.10.0-693.11.1.el7.x86_64 got installed. yum -y reinstall kernel seems to fix the issue most of the time.

BTW - found same issue after the installation of the previous kernel-3.10.0-514.10.2.el7.x86_64 - random at various VMs.

I ended up writing a script to compare the # of installed kernels against the number of created "/boot/initramfs-3.10..." files and do an automatice re-install of the kernel if the number is a mis-match.

So far so good - except for our developers are not happy at all as an occasional restart of a VM may not work.

Jamie Bainbridge's picture Red Hat Guru 7682 points
15 January 2018 11:44 PM

If you're still looking into this, is it possible to try amdgpu.runpm=0 radeon.runpm=0 on the GRUB kernel line? This was confirmed to resolve the issue in at least one other case.

sa Newbie 5 points
2 February 2018 8:18 PM

Hello, I got the same issue when I try to install the latest rhel alongside my ubuntu and windows 10.

kernel panic not syncing : fatal exception Kernel Offset : disabled

Is there new solutions about this issue. I cant install rhel because of this kernel issue. Tried everything above and still doesnt work.

Jamie Bainbridge's picture Red Hat Guru 7682 points
12 April 2018 10:56 PM

Now that RHEL 7.5 is out, is this behaving differently for anyone?

PT Newbie 10 points
20 March 2019 8:53 PM

I am still seeing many issues with kernel panics on kernel update even with the 3.10.0-957.10.1.el7.x86_64 kernel, these systems are all VMWare guests on ESX 6.5, machine version 13.

James Nauer's picture Expert 963 points
21 March 2019 3:39 AM

Is the VMware server fully patched (6.5 update 2 and any other patches released since then)? If not, I would suggest doing that first. We run over 200 RHEL VMs on ESXi 6.5U2+patches, and have seen zero kernel panics. There was an issue with one specific VMware revision, but it has long since been patched. (Most of our VMs are running old "VM hardware" versions - we have a lot of "version 10" VMs that were built on ESX 5.5 - but we have specifically updated a number of them to test for the ESX 6.5+VM version 13 crashing issue, and we have not seen it).

PT Newbie 10 points
22 March 2019 3:08 PM

We are running 6.5u1, so that could be related. We have about 1200 VM's in our environment, and we see this in less than 10% of system reboots. We're going to ask our VM team about updating to 6.5u2. Thanks for checking, I appreciate it.