Crash kernel goes out of memory because of huge number of FC LUN's

Solution Verified - Updated -

Issue

  • kdump is not dumping the core as the crash kernel is running out of memory.
  • Crash kernel throwing out of memory events when there is a high number of FC LUNs on the system. Initialization of the large number of LUNs leads to the crash kernel exhausting the reserved memory available for the core.
  • Serial console messages when manually triggered NMI :

    [  307.140656] kworker/u2:13 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 <<<<<<<< here.
    [  307.140658] kworker/u2:13 cpuset=/ mems_allowed=0
    [  307.140660] CPU: 0 PID: 518 Comm: kworker/u2:13 Not tainted 3.10.0-327.10.1.el7.x86_64 #1
    [  307.140661] Hardware name: HP ProLiant BL660c Gen8, BIOS I32 02/10/2014
    [  307.140671] Workqueue: khelper __call_usermodehelper
    [  307.140673]  ffff8800303a8000 00000000d6876668 ffff8800303b3a20 ffffffff816352a4
    [  307.140675]  ffff8800303b3ab0 ffffffff8163023f ffff8800303a4d70 ffff8800303a4d88
    [  307.140680]  0000000000000202 ffff8800303a8000 ffff8800303b3a98 ffffffff8112890f
    [  307.140681] Call Trace:
    [  307.140689]  [<ffffffff816352a4>] dump_stack+0x19/0x1b
    [  307.140691]  [<ffffffff8163023f>] dump_header+0x8e/0x214
    [  307.140695]  [<ffffffff8112890f>] ? delayacct_end+0x8f/0xb0
    [  307.140700]  [<ffffffff8116ce4e>] oom_kill_process+0x24e/0x3b0
    [  307.140701]  [<ffffffff8116c9b6>] ? find_lock_task_mm+0x56/0xc0
    [  307.140703]  [<ffffffff8116d676>] out_of_memory+0x4b6/0x4f0
    [  307.140706]  [<ffffffff81173855>] __alloc_pages_nodemask+0xa95/0xb90
    [  307.140711]  [<ffffffff81078d53>] copy_process.part.25+0x163/0x1610
    [  307.140713]  [<ffffffff81099e40>] ? ____call_usermodehelper+0x180/0x180
    [  307.140715]  [<ffffffff8107a3e1>] do_fork+0xe1/0x320
    [  307.140722]  [<ffffffff81013588>] ? __switch_to+0xf8/0x4b0
    [  307.140724]  [<ffffffff8107a646>] kernel_thread+0x26/0x30
    [  307.140725]  [<ffffffff8109948e>] __call_usermodehelper+0x2e/0x90
    [  307.140728]  [<ffffffff8109d5db>] process_one_work+0x17b/0x470
    [  307.140729]  [<ffffffff8109e3ab>] worker_thread+0x11b/0x400
    [  307.140730]  [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400
    [  307.140733]  [<ffffffff810a5acf>] kthread+0xcf/0xe0
    [  307.140735]  [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
    [  307.140739]  [<ffffffff81645998>] ret_from_fork+0x58/0x90
    [  307.140741]  [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
    

Environment

  • Red Hat Enterprise Linux 7
  • kdump
  • scsi_transport_fc
  • lpfc (Emulex HBA storage adapter)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In