kdump kernel causes kernel panic due to Out of Memory in the system which has numerous storage devices.

Solution Verified - Updated -

Issue

  • kdump kernel causes kernel panic due to Out of Memory in the system which has numerous storage devices.

    Sample log is

    [   35.923346] systemd-journal invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
    [   35.933233] systemd-journal cpuset=/ mems_allowed=0
    [   35.939321] CPU: 0 PID: 329 Comm: systemd-journal Tainted: GF          O--------------   3.10.0-229.el7.x86_64 #1
    [   35.965850]  ffff88002d3b0000 000000003c774d89 ffff88002fcbf960 ffffffff81603f36
    [   35.974856]  ffff88002fcbf9f0 ffffffff815feeff ffff88002fc77bb0 ffff88002fc77bc8
    [   35.983850]  ffffffff00000206 ffffffff00000000 0000000000000000 ffffffff81117803
    [   35.992838] Call Trace:
    [   35.996214]  [<ffffffff81603f36>] dump_stack+0x19/0x1b
    [   36.002593]  [<ffffffff815feeff>] dump_header+0x8e/0x214
    [   36.009160]  [<ffffffff81117803>] ? delayacct_end+0x73/0xb0
    [   36.016015]  [<ffffffff8115a07e>] oom_kill_process+0x24e/0x3b0
    [   36.023158]  [<ffffffff8107bd0e>] ? has_capability_noaudit+0x1e/0x30
    [   36.030886]  [<ffffffff8115a8a6>] out_of_memory+0x4b6/0x4f0
    [   36.037739]  [<ffffffff81160a65>] __alloc_pages_nodemask+0xa95/0xb90
    [   36.045466]  [<ffffffff811a24aa>] alloc_pages_vma+0x9a/0x140
    [   36.052416]  [<ffffffff8116d4de>] shmem_alloc_page+0x6e/0xc0
    [   36.059359]  [<ffffffff8126ec0b>] ? cred_has_capability+0x6b/0x120
    [   36.066877]  [<ffffffff812f4c55>] ? percpu_counter_compare+0x25/0x90
    [   36.074586]  [<ffffffff8116fce4>] shmem_getpage_gfp+0x404/0x7b0
    [   36.081807]  [<ffffffff81171a6a>] shme3]  [<ffffffff811c8d18>] ? __sb_start_write+0x58/0x110
    [   36.096056]  [<ffffffff811c4c17>] do_fallocate+0x117/0x1b0
    [   36.102766]  [<ffffffff811c4d0b>] SyS_fallocate+0x5b/0xa0
    [   36.109368]  [<ffffffff81613da9>] system_call_fastpath+0x16/0x1b
    [   36.116640] Mem-Info:
    [   36.119726] Node 0 DMA per-cpu:
    [   36.123810] CPU    0: hi:    0, btch:   1 usd:   0
    [   36.129712] Node 0 DMA32 per-cpu:
    [   36.133985] CPU    0: hi:   42, btch:   7 usd:  13
    [   36.139887] active_anon:1164 inactive_anon:2319 isolated_anon:0
    [   36.139887]  active_file:0 inactive_file:0 isolated_file:0
    [   36.139887]  unevictable:10887 dirty:0 writeback:0 unstable:0
    [   36.139887]  free:501 slab_reclaimable:1764 slab_unreclaimable:10079
    [   36.139887]  mapped:913 shmem:2336 pagetables:154 bounce:0
    [   36.139887]  free_cma:0
    [   36.178206] Node 0 DMA free:516kB min:4kB low:4kB high:4kB active_anon:0kB inale:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:600kB managed:516kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimab   4450 le:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
    [   36.223329] lowmem_reserve[]: 0 140 140 140
    [   36.228671] Node 0 DMA32 free:1488kB min:1508kB low:1884kB high:2260kB active_anon:4656kB inactive_anon:9276kB active_file:0kB inactive_file:0kB unevictable:43548kB isolated(anon):0kB isolated(file):0kB present:165268kB managed:144972kB mlocked:0kB dirty:0kB wr   4452 iteback:0kB mapped:3652kB shmem:9344kB slab_reclaimable:7056kB slab_unreclaimable:40316kB kernel_stack:2096kB pagetables:616kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
    [   36.278393] lowmem_reserve[]: 0 0 0 0
    [   36.283193] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 516kB
    [   36.297908] Node 0 DMA32: 30*4kB (UE) 23*8kB (UEM) 8*16kB (UE) 9*32kB (UEM) 8*64kB (U) 0*128kB 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1488kB
    [   36.314547] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
    [   36.324653] 13223 total pagecache pages
    [   36.329605] 0 pages in swap cache
    [   36.333974] Swap cache stats: add 0, delete 0, find 0/0
    [   36.340482] Free swap  = 0kB
    [   36.344360] Total swap = 0kB
    [   36.348234] 41467 pages RAM
    [   36.352003] 0 pages HighMem/MovableOnly
    [   36.356936] 5095 pages reserved
    [   36.361089] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
    [   36.370499] [  178]     0   178     7335      123      17        0             0 iscsid
    [   36.380107] [  219]     0   219     9179      881      22       220]     0   220     8732      325      20        0             0 systemd-udevd
    [   36.400681] [  221]     0   221     8732      308      20        0             0 systemd-udevd
    [   36.410964] [  228]     0   228     3018      446      15        0             0 dracut-initqueu
    [   36.421437] [  234]     0   234     8178      286      18        0             0 udevadm
    [   36.431133] [  329]     0   329     7156      317      18        0             0 systemd-journal
    [   36.441603] Out of memory: Kill process 228 (dracut-initqueu) score 12 or sacrifice child
    [   36.451402] Killed process 234 (udevadm) total-vm:32712kB, anon-rss:272kB, file-rss:872kB
    :
    [   47.018753] Kernel panic - not syncing: Out of memory and no killable processes...
    [   47.018753] 
    [   47.03md-udevd Tainted: GF          O--------------   3.10.0-229.el7.x86_64 #1
    [   47.056576]  ffffffff81837648 00000000275977a3 ffff88002febba50 ffffffff81603f36
    [   47.065632]  ffff88002febbad0 ffffffff815fd7da ffff880000000008 ffff88002febbae0
    [   47.074676]  ffff88002febba80 00000000275977a3 000000000000023a 0000000000000046
    [   47.083724] Call Trace:
    [   47.087164]  [<ffffffff81603f36>] dump_stack+0x19/0x1b
    [   47.093627]  [<ffffffff815fd7da>] panic+0xd8/0x1e7
    [   47.099696]  [<ffffffff8115a8da>] out_of_memory+0x4ea/0x4f0
    [   47.106645]  [<ffffffff81160a65>] __alloc_pages_nodemask+0xa95/0xb90
    [   47.114467]  [<ffffffff811a24aa>] alloc_pages_vma+0x9a/0x140
    [   47.121519]  [<ffffffff8117ffa5>] do_wp_page+0xd5/0x7e0
    [   47.128078]  [<ffffffff81182734>] handle_mm_fault+0x584/0xd60
    [   47.135218]  [<ffffffff8160f1e6>] __do_page_fault+0x156/0x520
    [   47.142354]  [<ffffffff8160f5ca>] ? do_page_fault+0x1a/0x70
    [   47.149290]  [<ffffffff8160b808>] ? page_fault+0x28/0x30
    [   47.155925]  [<ffffffff810ad726>] ? __dequeue_entity+0x26/0x40
    [   47.163143]  [<ffffffff81012609>] ? __switch_to+0x179/0x4a0
    [   47.170047]  [<ffffffff8160f5ca>] do_page_fault+0x1a/0x70
    [   47.176745]  [<ffffffff8160b808>] page_fault+0x28/0x30
    [   47.311817] drm_kms_helper: panic occurred, switching back to text console
    [   47.342750] Rebooting in 10 seconds..
    

Environment

  • Red Hat Enterprise Linux 7
  • kexec-tools

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In