Xen guest runs out of memory leading to OOM killers

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 5
  • 32-bit Xen guest

Issue

  • A Xen server has 24GB of memory and one xen guest. The guest has 20GB memory, but from time to time it runs out of memory and the OOM killer starts killing processes randomly. As per Xen host logs, there should be enough memory.

kernel: Normal: empty
kernel: HighMem: 134889*4kB 27382*8kB 1314*16kB
122*32kB 95*64kB 88*128kB 23*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB =
807796kB
kernel: 4404824 pagecache pages
kernel: Swap cache: add 3043093, delete 2947103, find 1265982/1557913, race 0+4
kernel: Free swap = 6976136kB
kernel: Total swap = 8385824kB
kernel: Free swap: 6976136kB
kernel: 5244928 pages of RAM
kernel: 5058562 pages of HIGHMEM
kernel: 47673 reserved pages
kernel: 6948557 pages shared
kernel: 95996 pages swap cached
kernel: 0 pages dirty
kernel: 0 pages writeback
kernel: 592882 pages mapped
kernel: 52901 pages slab
kernel: 82846 pages pagetables
kernel: Out of memory: Killed process 7919, UID 504 oracle

Resolution

  • Since the issue seems to be occurring regularly, please conider using x86_64 (64-bit) OS, under which the total memory will be in lowmem zone. This will prevent the filling up of lowmem and OOM condition. Note that, the system architecture should be x86_64 and the dom0 should also be running x86_64 OS for the x86_64 domU.

Root Cause

  • The message log of Xen guest shows the following:

kernel: DMA free:3396kB min:3452kB
low:4312kB high:5176kB active:1228kB inactive:1268kB present:745464kB
pages_scanned:150129 all_unreclaimable? yes
kernel: lowmem_reserve[]: 0 0 0 19760
kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 19760
kernel: Normal free:0kB min:0kB low:0kB high:0kB active:0kB
inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
kernel: lowmem_reserve[]: 0 0 0 158080
kernel: HighMem free:8913576kB min:512kB low:23936kB
high:47360kB active:7706736kB inactive:3586512kB present:20234248kB
pages_scanned:0 all_unreclaimable? no

  • The above message shows that the memory in the DMA region (in  the lowmem zone) went below the minimum threshold and OOM killer was  called. As per procinfo, the memory zone sizes are:
HighTotal:     20226056 kB
HighFree :     15010780 kB
LowTotal :       745640 kB
LowFree  :        52980 kB
  • The lowmem zone  is around 745 MB. The rest is highmem. This is because the OS is 32-bit.
Linux client 2.6.18-194.3.1.el5xen #1 SMP Sun May 2 05:02:13 EDT 2010 i686 i686 i386 GNU/Linux
  • Since low memory area is the most  performance critical zone, running of many resource intensive  applications/programs and/or if large physical memory is used, then  "low memory" can become low since more kernel structures must be  allocated in this area. Under heavy IO workloads the kernel may become  starved for LowMem even though there is an abundance of available  HighMem. This can lead to oom-killers or complete system hangs.
  • Component
  • xen

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments