RHEL7: mlx5 reports error messages during shutdown then panic with mce
Issue
- Kernel panic with below logs:
[ 200.824725] systemd-shutdown[1]: All filesystems unmounted.
[ 200.824736] systemd-shutdown[1]: Deactivating swaps.
[ 200.824837] systemd-shutdown[1]: All swaps deactivated.
[ 200.824846] systemd-shutdown[1]: Detaching loop devices.
[ 200.829664] systemd-shutdown[1]: All loop devices detached.
[ 200.829674] systemd-shutdown[1]: Detaching DM devices.
[ 200.829920] systemd-shutdown[1]: All DM devices detached.
[ 200.848448] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 200.848779] systemd-shutdown[1]: Rebooting.
[ 201.325192] infiniband mlx5_bond_0: wait_for_async_commands:745:(pid 3633): done with all pending requests
[ 204.919209] mlx5_core 0000:af:00.1: Shutdown was called
[ 205.014965] infiniband mlx5_bond_1: wait_for_async_commands:745:(pid 601): done with all pending requests
[ 210.785156] infiniband mlx5_3: wait_for_async_commands:745:(pid 1): done with all pending requests
[ 211.847192] mlx5_core 0000:af:00.0: Shutdown was called
[ 215.000902] infiniband mlx5_2: wait_for_async_commands:745:(pid 1): done with all pending requests
[ 215.310641] Disabling lock debugging due to kernel taint
[ 215.310732] mce: [Hardware Error]: CPU 12: Machine Check Exception: 5 Bank 6: fb80000000000e0b
[ 215.310733] mce: [Hardware Error]: Machine check events logged
[ 215.311849] mce: [Hardware Error]: RIP !INEXACT! 10:<ffffffffb6b8e764> {intel_idle+0xd4/0x225}
[ 215.312405] mce: [Hardware Error]: TSC 5405d7784ea MISC ae000000
[ 215.312956] mce: [Hardware Error]: PROCESSOR 0:50654 TIME 1650408701 SOCKET 1 APIC 40 microcode 2006c0a
[ 215.313520] mce: [Hardware Error]: Run the above through 'mcelog --ascii'
[ 215.333866] mce: [Hardware Error]: Machine check: Processor context corrupt
[ 215.334472] Kernel panic - not syncing: Fatal machine check
Environment
- Red Hat Enterprise Linux 7
- HPE DL380 Gen10
- Mellanox ConnectX-4 Lx
- mlx5 fw 14.32.1010 (HP_2420110034)
- mlx5 fw 14.31.1200 (HP_2420110034 / HP_2690110034)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.