Red Hat Enterprise Linux Server went out of memory (OOM) due to high "size-256" slab cache.

Issue

• Kernel panic due to out of memory (OOM) even though memory usage in user-space is very low.
• Red Hat Enterprise Linux Server crashed due to Out Of Memory (OOM) with 99% of memory in slab cache.

Mem-info:
...
Node 0 HighMem per-cpu: empty
Free pages:       70320kB (0kB HighMem)
Active:1320 inactive:20 dirty:0 writeback:0 unstable:0 free:17580 slab:3884713 mapped-file:1043 mapped-anon:232 pagetables:709
Node 0 DMA free:9996kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:9628kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3000 16130 16130
Node 0 DMA32 free:55444kB min:3020kB low:3772kB high:4528kB active:0kB inactive:80kB present:3072160kB pages_scanned:99135 all_unreclaimable? yes
lowmem_reserve[]: 0 0 13130 13130
Node 0 Normal free:4880kB min:13224kB low:16528kB high:19836kB active:5052kB inactive:88kB present:13445120kB pages_scanned:361373 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 5*4kB 3*8kB 4*16kB 3*32kB 1*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 9996kB
Node 0 DMA32: 7*4kB 5*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 13*4096kB = 55444kB
Node 0 Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 1*4096kB = 4880kB
Node 0 HighMem: empty
1328 pagecache pages
Swap cache: add 2259008, delete 2258738, find 209501/617218, race 11+1440
Free swap  = 1807976kB
Total swap = 2096440kB
Free swap:       1807976kB
4456448 pages of RAM
544207 reserved pages
2527 pages shared
277 pages swap cached
Kernel panic - not syncing: out of memory. panic_on_oom is selected

• Slab exhausts entire memory on the system.

# cat /proc/meminfo
MemTotal:     16435652 kB
MemFree:       9105776 kB
Buffers:        341888 kB
Cached:         168484 kB
SwapCached:          0 kB
Active:         401924 kB
Inactive:       200876 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     16435652 kB
LowFree:       9105776 kB
SwapTotal:     2096440 kB
SwapFree:      2096440 kB
Dirty:             668 kB
Writeback:           0 kB
AnonPages:       92384 kB
Mapped:          20264 kB
Slab:          6693116 kB
PageTables:       2340 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  10314264 kB
Committed_AS:   293292 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    264780 kB
VmallocChunk: 34359473015 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

• Slab memory is getting increased in the structure "size-256".

# egrep "size-256 " /proc/slabinfo 
size-256          20410665 20410665    256   15    1 : tunables  120   60    8 : slabdata 1360711 1360711      0