Kernel panic at dma_pool_alloc() function due to NULL pointer dereference upon reboot

Solution In Progress - Updated -

Issue

  • Kernel is getting panic while unloading the mlx5_core module by logging below messages.
crash> log
[..]
[  151.697868] watchdog: watchdog0: watchdog did not stop!
[  151.748255] dracut Warning: Killing all remaining processes
[  151.866029] dracut Warning: Unmounted /oldroot.
[  151.899148] dracut: Disassembling device-mapper devices
[  152.104169] mlx5_core 0000:19:00.1: Shutdown was called
[  152.205916] bond0: link status definitely down for interface eno2, disabling it
[  152.906587] mlx5_core 0000:19:00.0: Shutdown was called
[  153.011110] bond0: link status definitely down for interface eno1, disabling it
[  153.018412] bond0: now running without any active interface!
[  656.175800] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  656.183656] IP: [<ffffffff8880f13b>] dma_pool_alloc+0x1ab/0x280
[  656.189596] PGD 0 
[  656.191638] Oops: 0000 [#1] SMP 
[..]
[  656.307084] CPU: 18 PID: 17160 Comm: kworker/18:2 Kdump: loaded Tainted: P           OE  ------------   3.10.0-1062.12.1.el7.x86_64 #1
[  656.319127] Hardware name: Dell Inc. PowerEdge R740/01YM03, BIOS 2.4.8 11/26/2019
[  656.326592] Workqueue: events linkwatch_event
[  656.330962] task: ffff9901ca8220e0 ti: ffff9901c87d8000 task.ti: ffff9901c87d8000
[  656.338424] RIP: 0010:[<ffffffff8880f13b>]  [<ffffffff8880f13b>] dma_pool_alloc+0x1ab/0x280
[  656.346776] RSP: 0018:ffff9901c87db8c8  EFLAGS: 00010046
[  656.352075] RAX: 0000000000000246 RBX: ffff990214c74280 RCX: 0000000000001000
[  656.359191] RDX: 0000000000000000 RSI: 0000000000000246 RDI: ffff990214c74210
[  656.366304] RBP: ffff9901c87db920 R08: 000000000001f080 R09: ffff98ebbfc07c00
[  656.373421] R10: ffffffffc05640c4 R11: ffffffff887ddc4d R12: 00000000000080d0
[  656.380535] R13: ffff990214c74210 R14: ffff990214c74200 R15: 0000000000000000
[  656.387653] FS:  0000000000000000(0000) GS:ffff99021f640000(0000) knlGS:0000000000000000
[  656.395719] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  656.401450] CR2: 0000000000000000 CR3: 00000017de402000 CR4: 00000000007607e0
[  656.408565] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  656.415680] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  656.422796] PKRU: 00000000
[  656.425500] Call Trace:
[  656.427955]  [<ffffffff8882516c>] ? kmem_cache_alloc_trace+0x3c/0x200
[  656.434406]  [<ffffffffc05640e3>] mlx5_alloc_cmd_msg+0xd3/0x2a0 [mlx5_core]
[  656.441372]  [<ffffffffc0566c92>] cmd_exec+0x112/0x860 [mlx5_core]
[  656.447549]  [<ffffffffc056742b>] mlx5_cmd_exec+0x2b/0x50 [mlx5_core]
[  656.453996]  [<ffffffffc0571354>] mlx5_core_access_reg+0xe4/0x130 [mlx5_core]
[  656.461135]  [<ffffffffc0571411>] mlx5_query_port_ptys+0x71/0x90 [mlx5_core]
[  656.468189]  [<ffffffffc0596035>] mlx5e_get_link_ksettings+0x75/0x500 [mlx5_core]
[  656.475650]  [<ffffffff88c56d06>] __ethtool_get_link_ksettings+0xa6/0x210
[  656.482420]  [<ffffffff88d80d17>] ? wait_for_completion+0x37/0x140
[  656.488585]  [<ffffffff887355bf>] ? queue_stop_cpus_work+0x8f/0xa0
[  656.494752]  [<ffffffffc06dd6fd>] bond_update_speed_duplex+0x3d/0x90 [bonding]
[  656.501962]  [<ffffffffc06e4d88>] bond_netdev_event+0x138/0x360 [bonding]
[  656.508738]  [<ffffffff88d88b6f>] notifier_call_chain+0x4f/0x70
[  656.514642]  [<ffffffff886cc3d6>] raw_notifier_call_chain+0x16/0x20
[  656.520894]  [<ffffffff88c4a6ed>] call_netdevice_notifiers_info+0x4d/0x80
[  656.527661]  [<ffffffff88c4ac31>] netdev_state_change+0x41/0x70
[  656.533566]  [<ffffffff88c6a6d0>] linkwatch_do_dev+0x40/0x60
[  656.539212]  [<ffffffff88c6a9e9>] __linkwatch_run_queue+0xf9/0x190
[  656.545376]  [<ffffffff88c6aaa5>] linkwatch_event+0x25/0x30
[  656.550936]  [<ffffffff886be21f>] process_one_work+0x17f/0x440
[  656.556753]  [<ffffffff886bf336>] worker_thread+0x126/0x3c0
[  656.562313]  [<ffffffff886bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[  656.568824]  [<ffffffff886c61f1>] kthread+0xd1/0xe0
[  656.573691]  [<ffffffff886c6120>] ? insert_kthread_work+0x40/0x40
[  656.579768]  [<ffffffff88d8dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[  656.586191]  [<ffffffff886c6120>] ? insert_kthread_work+0x40/0x40
[  656.592268] Code: 4c 89 f6 48 89 df 48 89 45 b0 e8 81 20 19 00 8b 53 24 48 8b 45 b0 49 89 d7 4c 03 7b 10 83 43 20 01 48 03 53 18 48 89 c6 4c 89 ef <41> 8b 0f 89 4b 24 48 8b 4d b8 48 89 11 e8 e3 3e 57 00 41 81 e4 
[  656.612690] RIP  [<ffffffff8880f13b>] dma_pool_alloc+0x1ab/0x280
[  656.618706]  RSP <ffff9901c87db8c8>
[  656.622189] CR2: 0000000000000000

Environment

  • Red Hat Enterprise Linux 7
  • kernel-3.10.0-1062.12.1.el7
  • mlx5_core, bonding

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In