Crash kernel goes out of memory because of huge number of FC LUN's

Solution Verified - Updated -

Issue

  • kdump is not dumping the core as the crash kernel is running out of memory.
  • Crash kernel throwing out of memory events when there is a high number of FC LUNs on the system. Initialization of the large number of LUNs leads to the crash kernel exhausting the reserved memory available for the core.
  • Serial console messages when manually triggered NMI :

    [  307.140656] kworker/u2:13 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 <<<<<<<< here.
    [  307.140658] kworker/u2:13 cpuset=/ mems_allowed=0
    [  307.140660] CPU: 0 PID: 518 Comm: kworker/u2:13 Not tainted 3.10.0-327.10.1.el7.x86_64 #1
    [  307.140661] Hardware name: HP ProLiant BL660c Gen8, BIOS I32 02/10/2014
    [  307.140671] Workqueue: khelper __call_usermodehelper
    [  307.140673]  ffff8800303a8000 00000000d6876668 ffff8800303b3a20 ffffffff816352a4
    [  307.140675]  ffff8800303b3ab0 ffffffff8163023f ffff8800303a4d70 ffff8800303a4d88
    [  307.140680]  0000000000000202 ffff8800303a8000 ffff8800303b3a98 ffffffff8112890f
    [  307.140681] Call Trace:
    [  307.140689]  [<ffffffff816352a4>] dump_stack+0x19/0x1b
    [  307.140691]  [<ffffffff8163023f>] dump_header+0x8e/0x214
    [  307.140695]  [<ffffffff8112890f>] ? delayacct_end+0x8f/0xb0
    [  307.140700]  [<ffffffff8116ce4e>] oom_kill_process+0x24e/0x3b0
    [  307.140701]  [<ffffffff8116c9b6>] ? find_lock_task_mm+0x56/0xc0
    [  307.140703]  [<ffffffff8116d676>] out_of_memory+0x4b6/0x4f0
    [  307.140706]  [<ffffffff81173855>] __alloc_pages_nodemask+0xa95/0xb90
    [  307.140711]  [<ffffffff81078d53>] copy_process.part.25+0x163/0x1610
    [  307.140713]  [<ffffffff81099e40>] ? ____call_usermodehelper+0x180/0x180
    [  307.140715]  [<ffffffff8107a3e1>] do_fork+0xe1/0x320
    [  307.140722]  [<ffffffff81013588>] ? __switch_to+0xf8/0x4b0
    [  307.140724]  [<ffffffff8107a646>] kernel_thread+0x26/0x30
    [  307.140725]  [<ffffffff8109948e>] __call_usermodehelper+0x2e/0x90
    [  307.140728]  [<ffffffff8109d5db>] process_one_work+0x17b/0x470
    [  307.140729]  [<ffffffff8109e3ab>] worker_thread+0x11b/0x400
    [  307.140730]  [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400
    [  307.140733]  [<ffffffff810a5acf>] kthread+0xcf/0xe0
    [  307.140735]  [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
    [  307.140739]  [<ffffffff81645998>] ret_from_fork+0x58/0x90
    [  307.140741]  [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
    

Environment

  • Red Hat Enterprise Linux 7
  • kdump
  • scsi_transport_fc
  • lpfc (Emulex HBA storage adapter)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content