Kernel panic at dma_pool_alloc() function due to NULL pointer dereference upon reboot

Solution Verified - Updated -

Issue

  • Kernel is getting panic while unloading the mlx5_core module by logging below messages.
crash> log
[..]
[  151.697868] watchdog: watchdog0: watchdog did not stop!
[  151.748255] dracut Warning: Killing all remaining processes
[  151.866029] dracut Warning: Unmounted /oldroot.
[  151.899148] dracut: Disassembling device-mapper devices
[  152.104169] mlx5_core 0000:19:00.1: Shutdown was called
[  152.205916] bond0: link status definitely down for interface eno2, disabling it
[  152.906587] mlx5_core 0000:19:00.0: Shutdown was called
[  153.011110] bond0: link status definitely down for interface eno1, disabling it
[  153.018412] bond0: now running without any active interface!
[  656.175800] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  656.183656] IP: [<ffffffff8880f13b>] dma_pool_alloc+0x1ab/0x280
[  656.189596] PGD 0 
[  656.191638] Oops: 0000 [#1] SMP 
[..]
[  656.307084] CPU: 18 PID: 17160 Comm: kworker/18:2 Kdump: loaded Tainted: P           OE  ------------   3.10.0-1062.12.1.el7.x86_64 #1
[  656.319127] Hardware name: Dell Inc. PowerEdge R740/01YM03, BIOS 2.4.8 11/26/2019
[  656.326592] Workqueue: events linkwatch_event
[  656.330962] task: ffff9901ca8220e0 ti: ffff9901c87d8000 task.ti: ffff9901c87d8000
[  656.338424] RIP: 0010:[<ffffffff8880f13b>]  [<ffffffff8880f13b>] dma_pool_alloc+0x1ab/0x280
[  656.346776] RSP: 0018:ffff9901c87db8c8  EFLAGS: 00010046
[  656.352075] RAX: 0000000000000246 RBX: ffff990214c74280 RCX: 0000000000001000
[  656.359191] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff990214c74210
[  656.366304] RBP: ffff9901c87db920 R08: 000000000001f080 R09: ffff98ebbfc07c00
[  656.373421] R10: ffffffffc05640c4 R11: ffffffff887ddc4d R12: 00000000000080d0
[  656.380535] R13: ffff990214c74210 R14: ffff990214c74200 R15: 0000000000000000
[  656.387653] FS:  0000000000000000(0000) GS:ffff99021f640000(0000) knlGS:0000000000000000
[  656.395719] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  656.401450] CR2: 0000000000000000 CR3: 00000017de402000 CR4: 00000000007607e0
[  656.408565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  656.415680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  656.422796] PKRU: 00000000
[  656.425500] Call Trace:
[  656.427955]  [<ffffffff8882516c>] ? kmem_cache_alloc_trace+0x3c/0x200
[  656.434406]  [<ffffffffc05640e3>] mlx5_alloc_cmd_msg+0xd3/0x2a0 [mlx5_core]
[  656.441372]  [<ffffffffc0566c92>] cmd_exec+0x112/0x860 [mlx5_core]
[  656.447549]  [<ffffffffc056742b>] mlx5_cmd_exec+0x2b/0x50 [mlx5_core]
[  656.453996]  [<ffffffffc0571354>] mlx5_core_access_reg+0xe4/0x130 [mlx5_core]
[  656.461135]  [<ffffffffc0571411>] mlx5_query_port_ptys+0x71/0x90 [mlx5_core]
[  656.468189]  [<ffffffffc0596035>] mlx5e_get_link_ksettings+0x75/0x500 [mlx5_core]
[  656.475650]  [<ffffffff88c56d06>] __ethtool_get_link_ksettings+0xa6/0x210
[  656.482420]  [<ffffffff88d80d17>] ? wait_for_completion+0x37/0x140
[  656.488585]  [<ffffffff887355bf>] ? queue_stop_cpus_work+0x8f/0xa0
[  656.494752]  [<ffffffffc06dd6fd>] bond_update_speed_duplex+0x3d/0x90 [bonding]
[  656.501962]  [<ffffffffc06e4d88>] bond_netdev_event+0x138/0x360 [bonding]
[  656.508738]  [<ffffffff88d88b6f>] notifier_call_chain+0x4f/0x70
[  656.514642]  [<ffffffff886cc3d6>] raw_notifier_call_chain+0x16/0x20
[  656.520894]  [<ffffffff88c4a6ed>] call_netdevice_notifiers_info+0x4d/0x80
[  656.527661]  [<ffffffff88c4ac31>] netdev_state_change+0x41/0x70
[  656.533566]  [<ffffffff88c6a6d0>] linkwatch_do_dev+0x40/0x60
[  656.539212]  [<ffffffff88c6a9e9>] __linkwatch_run_queue+0xf9/0x190
[  656.545376]  [<ffffffff88c6aaa5>] linkwatch_event+0x25/0x30
[  656.550936]  [<ffffffff886be21f>] process_one_work+0x17f/0x440
[  656.556753]  [<ffffffff886bf336>] worker_thread+0x126/0x3c0
[  656.562313]  [<ffffffff886bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[  656.568824]  [<ffffffff886c61f1>] kthread+0xd1/0xe0
[  656.573691]  [<ffffffff886c6120>] ? insert_kthread_work+0x40/0x40
[  656.579768]  [<ffffffff88d8dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[  656.586191]  [<ffffffff886c6120>] ? insert_kthread_work+0x40/0x40
[  656.592268] Code: 4c 89 f6 48 89 df 48 89 45 b0 e8 81 20 19 00 8b 53 24 48 8b 45 b0 49 89 d7 4c 03 7b 10 83 43 20 01 48 03 53 18 48 89 c6 4c 89 ef <41> 8b 0f 89 4b 24 48 8b 4d b8 48 89 11 e8 e3 3e 57 00 41 81 e4 
[  656.612690] RIP  [<ffffffff8880f13b>] dma_pool_alloc+0x1ab/0x280
[  656.618706]  RSP <ffff9901c87db8c8>
[  656.622189] CR2: 0000000000000000

Environment

  • Red Hat Enterprise Linux 7
  • kernel-3.10.0-1062.12.1.el7
  • mlx5_core, bonding

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content