Crash kernel goes out of memory because of huge number of FC LUN's
Issue
- kdump is not dumping the core as the crash kernel is running out of memory.
- Crash kernel throwing out of memory events when there is a high number of FC LUNs on the system. Initialization of the large number of LUNs leads to the crash kernel exhausting the reserved memory available for the core.
-
Serial console messages when manually triggered NMI :
[ 307.140656] kworker/u2:13 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 <<<<<<<< here. [ 307.140658] kworker/u2:13 cpuset=/ mems_allowed=0 [ 307.140660] CPU: 0 PID: 518 Comm: kworker/u2:13 Not tainted 3.10.0-327.10.1.el7.x86_64 #1 [ 307.140661] Hardware name: HP ProLiant BL660c Gen8, BIOS I32 02/10/2014 [ 307.140671] Workqueue: khelper __call_usermodehelper [ 307.140673] ffff8800303a8000 00000000d6876668 ffff8800303b3a20 ffffffff816352a4 [ 307.140675] ffff8800303b3ab0 ffffffff8163023f ffff8800303a4d70 ffff8800303a4d88 [ 307.140680] 0000000000000202 ffff8800303a8000 ffff8800303b3a98 ffffffff8112890f [ 307.140681] Call Trace: [ 307.140689] [<ffffffff816352a4>] dump_stack+0x19/0x1b [ 307.140691] [<ffffffff8163023f>] dump_header+0x8e/0x214 [ 307.140695] [<ffffffff8112890f>] ? delayacct_end+0x8f/0xb0 [ 307.140700] [<ffffffff8116ce4e>] oom_kill_process+0x24e/0x3b0 [ 307.140701] [<ffffffff8116c9b6>] ? find_lock_task_mm+0x56/0xc0 [ 307.140703] [<ffffffff8116d676>] out_of_memory+0x4b6/0x4f0 [ 307.140706] [<ffffffff81173855>] __alloc_pages_nodemask+0xa95/0xb90 [ 307.140711] [<ffffffff81078d53>] copy_process.part.25+0x163/0x1610 [ 307.140713] [<ffffffff81099e40>] ? ____call_usermodehelper+0x180/0x180 [ 307.140715] [<ffffffff8107a3e1>] do_fork+0xe1/0x320 [ 307.140722] [<ffffffff81013588>] ? __switch_to+0xf8/0x4b0 [ 307.140724] [<ffffffff8107a646>] kernel_thread+0x26/0x30 [ 307.140725] [<ffffffff8109948e>] __call_usermodehelper+0x2e/0x90 [ 307.140728] [<ffffffff8109d5db>] process_one_work+0x17b/0x470 [ 307.140729] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400 [ 307.140730] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400 [ 307.140733] [<ffffffff810a5acf>] kthread+0xcf/0xe0 [ 307.140735] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [ 307.140739] [<ffffffff81645998>] ret_from_fork+0x58/0x90 [ 307.140741] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
Environment
- Red Hat Enterprise Linux 7
- kdump
- scsi_transport_fc
- lpfc (Emulex HBA storage adapter)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.