System hang due to low memory exhaustion on 32-bit x86 kernel

Solution Unverified - Updated -

Issue

After start 3 service, wait for a while, system hang and get vmcore by NMI

System log contained messages like this:

kthread invoked oom-killer: gfp_mask=0xd0, order=0, oomkilladj=0
 [<c045ef33>] out_of_memory+0x72/0x1a3
 [<c04604bf>] __alloc_pages+0x24e/0x2cf
 [<c04759bc>] cache_alloc_refill+0x275/0x48a
 [<c0475894>] cache_alloc_refill+0x14d/0x48a
 [<c047573d>] kmem_cache_alloc+0x41/0x4b
 [<c0424701>] copy_process+0xa2/0x127e
 [<c0436169>] alloc_pid+0x1d3/0x236
 [<c0425b1b>] do_fork+0x41/0x168
 [<c0438072>] kthread+0x0/0xee
 [<c04034df>] kernel_thread+0x70/0x99
 [<c0438072>] kthread+0x0/0xee
 [<c0405cd0>] kernel_thread_helper+0x0/0x10
 [<c0437f4e>] keventd_create_kthread+0x18/0x7f
 [<c04352a9>] run_workqueue+0x81/0xc5
 [<c0437f36>] keventd_create_kthread+0x0/0x7f
 [<c0435d0c>] worker_thread+0xd9/0x10d
 [<c0420887>] default_wake_function+0x0/0xc
 [<c0435c33>] worker_thread+0x0/0x10d
 [<c0438132>] kthread+0xc0/0xee
 [<c0438072>] kthread+0x0/0xee
 [<c0405cd7>] kernel_thread_helper+0x7/0x10
 =======================
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
DMA32 per-cpu: empty
Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:37
cpu 0 cold: high 62, batch 15 used:61
cpu 1 hot: high 186, batch 31 used:12
cpu 1 cold: high 62, batch 15 used:57
cpu 2 hot: high 186, batch 31 used:96
cpu 2 cold: high 62, batch 15 used:53
cpu 3 hot: high 186, batch 31 used:11
cpu 3 cold: high 62, batch 15 used:56
HighMem per-cpu:
cpu 0 hot: high 186, batch 31 used:84
cpu 0 cold: high 62, batch 15 used:5
cpu 1 hot: high 186, batch 31 used:149
cpu 1 cold: high 62, batch 15 used:2
cpu 2 hot: high 186, batch 31 used:13
cpu 2 cold: high 62, batch 15 used:7
cpu 3 hot: high 186, batch 31 used:119
cpu 3 cold: high 62, batch 15 used:14
Free pages:      629064kB (622568kB HighMem)
Active:86646 inactive:49291 dirty:0 writeback:0 unstable:0 free:157266 slab:187173 mapped-file:44255 mapped-anon:73843 pagetables:1837
DMA free:3588kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 0 880 2031
DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 880 2031
Normal free:2908kB min:3756kB low:4692kB high:5632kB active:0kB inactive:0kB present:901120kB pages_scanned:857881895 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 9215
HighMem free:622568kB min:512kB low:1740kB high:2972kB active:346584kB inactive:197164kB present:1179596kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3588kB
DMA32: empty
Normal: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2908kB
HighMem: 19376*4kB 28349*8kB 15004*16kB 2274*32kB 67*64kB 5*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 622568kB
69808 pagecache pages
Swap cache: add 41757, delete 30894, find 7658/9375, race 0+0
Free swap  = 2020304kB
Total swap = 2096472kB
Free swap:       2020304kB
524275 pages of RAM
294899 pages of HIGHMEM
38310 reserved pages
104409 pages shared
10863 pages swap cached
0 pages dirty
0 pages writeback
44255 pages mapped
187173 pages slab
1837 pages pagetables

Environment

  • Redhat Enterprise Linux 5.9
  • 32-bit kernel (running on a 64-bit CPU)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.