System hang due to low memory exhaustion on 32-bit x86 kernel
Issue
After start 3 service, wait for a while, system hang and get vmcore by NMI
System log contained messages like this:
kthread invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
[<c045ef33>] out_of_memory+0x72/0x1a3
[<c04604bf>] __alloc_pages+0x24e/0x2cf
[<c04759bc>] cache_alloc_refill+0x275/0x48a
[<c0475894>] cache_alloc_refill+0x14d/0x48a
[<c047573d>] kmem_cache_alloc+0x41/0x4b
[<c0424701>] copy_process+0xa2/0x127e
[<c0436169>] alloc_pid+0x1d3/0x236
[<c0425b1b>] do_fork+0x41/0x168
[<c0438072>] kthread+0x0/0xee
[<c04034df>] kernel_thread+0x70/0x99
[<c0438072>] kthread+0x0/0xee
[<c0405cd0>] kernel_thread_helper+0x0/0x10
[<c0437f4e>] keventd_create_kthread+0x18/0x7f
[<c04352a9>] run_workqueue+0x81/0xc5
[<c0437f36>] keventd_create_kthread+0x0/0x7f
[<c0435d0c>] worker_thread+0xd9/0x10d
[<c0420887>] default_wake_function+0x0/0xc
[<c0435c33>] worker_thread+0x0/0x10d
[<c0438132>] kthread+0xc0/0xee
[<c0438072>] kthread+0x0/0xee
[<c0405cd7>] kernel_thread_helper+0x7/0x10
=======================
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
DMA32 per-cpu: empty
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:37
cpu 0 cold: high 62, batch 15 used:61
cpu 1 hot: high 186, batch 31 used:12
cpu 1 cold: high 62, batch 15 used:57
cpu 2 hot: high 186, batch 31 used:96
cpu 2 cold: high 62, batch 15 used:53
cpu 3 hot: high 186, batch 31 used:11
cpu 3 cold: high 62, batch 15 used:56
HighMem per-cpu:
cpu 0 hot: high 186, batch 31 used:84
cpu 0 cold: high 62, batch 15 used:5
cpu 1 hot: high 186, batch 31 used:149
cpu 1 cold: high 62, batch 15 used:2
cpu 2 hot: high 186, batch 31 used:13
cpu 2 cold: high 62, batch 15 used:7
cpu 3 hot: high 186, batch 31 used:119
cpu 3 cold: high 62, batch 15 used:14
Free pages: 629064kB (622568kB HighMem)
Active:86646 inactive:49291 dirty:0 writeback:0 unstable:0 free:157266 slab:187173 mapped-file:44255 mapped-anon:73843 pagetables:1837
DMA free:3588kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 880 2031
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 880 2031
Normal free:2908kB min:3756kB low:4692kB high:5632kB active:0kB inactive:0kB present:901120kB pages_scanned:857881895 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 9215
HighMem free:622568kB min:512kB low:1740kB high:2972kB active:346584kB inactive:197164kB present:1179596kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3588kB
DMA32: empty
Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2908kB
HighMem: 19376*4kB 28349*8kB 15004*16kB 2274*32kB 67*64kB 5*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 622568kB
69808 pagecache pages
Swap cache: add 41757, delete 30894, find 7658/9375, race 0+0
Free swap = 2020304kB
Total swap = 2096472kB
Free swap: 2020304kB
524275 pages of RAM
294899 pages of HIGHMEM
38310 reserved pages
104409 pages shared
10863 pages swap cached
0 pages dirty
0 pages writeback
44255 pages mapped
187173 pages slab
1837 pages pagetables
Environment
- Redhat Enterprise Linux 5.9
- 32-bit kernel (running on a 64-bit CPU)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.