Kernel panic due to "kmem_cache_alloc+117 from mempool_alloc_slab" on RHEL 7

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7.1 and later

Issue

  • PANIC general protection fault RIP: kmem_cache_alloc+117 from mempool_alloc_slab

Resolution

  • Performance will be impacted by the sanity checks and there will be a continual stream of error messages to the syslog but no additional memory will be used (unlike full debugging). For more information please read https://www.kernel.org/doc/Documentation/vm/slub.txt.

Root Cause

  • This issue is caused by a corrupted freelist pointer.
  • Several drivers have been determined/suspected to cause freelist pointer corruption. The following three modules were resolved in specific errata releases:

    • qla2xxx - Fixed in kernel-3.10.0-862.20.1.el7 - 1624503
    • security - Fixed in kernel-3.10.0-1017.el7 - 1607307
    • security - Fixed in kernel-3.10.0-957.19.1.el7 - 1702286
    • NFS - Fixed in kernel-3.10.0-1160.15.2.el7 - 1679980
  • The third party drivers, scdrv and asrim, are also suspected to cause freelist pointer corruption issues. Consider engaging the respective vendors for the latest driver version.

  • Remaining issues are no longer being tracked in Private Bugzilla 1560630 because many conditions have solidly been linked back to 3rd party drivers performing "use-after-free" or "double-free".

Diagnostic Steps

Steps that can be taken to collect data next time the issue is seen:

Steps to avoid the panic:

  • enable slub_debug=F on the kernel command line as detailed in How to use slub_debug to troubleshoot and debug slab issues. The 'F' options set debugging to "Sanity checks only" (minimal debugging). Setting Sanity check will detect a corrupt free-pointer and prevent a panic. At runtime, it will then zero out that free-pointer and find another cache page to use for object allocation. However, using only the slub_debug=F option will provide little to no debug tracing to assist in identifying a potential driver issue.

Call trace of panic from vmcore:

[14787601.352890] general protection fault: 0000 [#1] SMP 
[14787601.352922] Modules linked in: bonding sg acpi_cpufreq coretemp kvm_intel kvm iTCO_wdt iTCO_vendor_support crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_si ipmi_msghandler serio_raw pcspkr sb_edac edac_core hpilo hpwdt lpc_ich mfd_core ioatdma shpchp acpi_power_meter pcc_cpufreq mperf xfs libcrc32c sd_mod crc_t10dif crct10dif_common mgag200 syscopyarea sysfillrect sysimgblt drm_kms_helper ahci ttm libahci igb drm libata ptp pps_core dca i2c_algo_bit hpsa i2c_core dm_mirror dm_region_hash dm_log dm_mod
[14787601.353172] CPU: 0 PID: 1799 Comm: xfsaild/dm-2 Not tainted 3.10.0-123.20.1.el7.x86_64 #1
[14787601.353202] Hardware name: HP ProLiant DL380e Gen8, BIOS P73 08/02/2014
[14787601.353227] task: ffff88183ac438e0 ti: ffff88183549a000 task.ti: ffff88183549a000
[14787601.353261] RIP: 0010:[]  [] kmem_cache_alloc+0x75/0x1d0
[14787601.353296] RSP: 0018:ffff88183549b9e8  EFLAGS: 00010286
[14787601.353316] RAX: 0000000000000000 RBX: ffff88183adb5330 RCX: 00000000048e1907
[14787601.353343] RDX: 00000000048e1906 RSI: 0000000000011200 RDI: ffff880c43409a00
[14787601.353369] RBP: ffff88183549ba18 R08: 0000000000018200 R09: ffffffff81144355
[14787601.353395] R10: ffefa04c1fedfe00 R11: ffff880c3befe008 R12: 0c189ed848ffff88
[14787601.353421] R13: 0000000000011200 R14: ffff880c43409a00 R15: ffff880c43409a00
[14787601.353448] FS:  0000000000000000(0000) GS:ffff880c4fc00000(0000) knlGS:0000000000000000
[14787601.353478] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14787601.353499] CR2: 00007f1ea4e22000 CR3: 00000000018d0000 CR4: 00000000001407f0
[14787601.353525] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14787601.353552] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[14787601.353577] Stack:
[14787601.353587]  0000000417edf200 ffff88183adb5330 ffff88183549ba68 0000000000011200
[14787601.353622]  ffff88183adb5300 ffff88183ac438e0 ffff88183549ba28 ffffffff81144355
[14787601.353655]  ffff88183549baa8 ffffffff81144499 ffff88183549ba70 00000010a0000cf5
[14787601.353690] Call Trace:
[14787601.353705]  [] mempool_alloc_slab+0x15/0x20
[14787601.353728]  [] mempool_alloc+0x69/0x170
[14787601.353752]  [] get_request+0x31d/0x780
[14787601.353775]  [] ? wake_up_bit+0x30/0x30
[14787601.353797]  [] blk_queue_bio+0x96/0x390
[14787601.353819]  [] generic_make_request+0xe2/0x130
[14787601.353843]  [] submit_bio+0x71/0x150
[14787601.353889]  [] _xfs_buf_ioapply+0x2bb/0x3d0 [xfs]
[14787601.353922]  [] ? xfs_bdstrat_cb+0x55/0xb0 [xfs]
[14787601.353955]  [] xfs_buf_iorequest+0x46/0xa0 [xfs]
[14787601.353986]  [] xfs_bdstrat_cb+0x55/0xb0 [xfs]
[14787601.354018]  [] __xfs_buf_delwri_submit+0x183/0x230 [xfs]
[14787601.354053]  [] ? xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
[14787601.354097]  [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[14787601.354133]  [] xfs_buf_delwri_submit_nowait+0x2f/0x50 [xfs]
[14787601.354172]  [] xfsaild+0x240/0x5e0 [xfs]
[14787601.354206]  [] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[14787601.354234]  [] kthread+0xcf/0xe0
[14787601.355036]  [] ? kthread_create_on_node+0x140/0x140
[14787601.355836]  [] ret_from_fork+0x7c/0xb0
[14787601.356630]  [] ? kthread_create_on_node+0x140/0x140
[14787601.357427] Code: cc 00 00 49 8b 50 08 4d 8b 20 49 8b 40 10 4d 85 e4 0f 84 1f 01 00 00 48 85 c0 0f 84 16 01 00 00 49 63 46 20 48 8d 4a 01 4d 8b 06  8b 1c 04 4c 89 e0 65 49 0f c7 08 0f 94 c0 84 c0 74 b9 49 63 
[14787601.359106] RIP  [] kmem_cache_alloc+0x75/0x1d0
[14787601.359895]  RSP 

May also manifest with odd values for x86 Instruction Pointer or PPC64 Link Register, after previous function calls kmem_cache_alloc(). For example, the following stack has global variable ip_packet_offload as current function, which is impossible:

 #0 [c000007c9ff0bb70] ip_packet_offload at c00000000117c9c0
 #1 [c000007c9ff0bbc0] build_skb at c00000000079c7e8
 #2 [c000007c9ff0bc00] bnx2x_rx_int at d000000033da2384 [bnx2x]
 #3 [c000007c9ff0bd40] bnx2x_poll at d000000033da4d90 [bnx2x]
 #4 [c000007c9ff0bdf0] net_rx_action at c0000000007baf38
 #5 [c000007c9ff0bea0] __do_softirq at c0000000000de734
 #6 [c000007c9ff0bf90] call_do_softirq at c000000000024fb8

Previous function just called kmem_cache_alloc():

crash> dis -lr c00000000079c7e8 | tail -4

0xc00000000079c7dc <build_skb+0x3c>:    ld      r3,28664(r9)
0xc00000000079c7e0 <build_skb+0x40>:    li      r4,32
0xc00000000079c7e4 <build_skb+0x44>:    bl      0xc0000000002c8168 <kmem_cache_alloc+0x8>
0xc00000000079c7e8 <build_skb+0x48>:    nop

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments