Running out of memory on RHEL 6 Oracle Server

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux, version 6.4

Issue

  • We have an oracle server with 28 GB of RAM with 24 GB of HugePages reserved for Oracle. The server spawned oom-killer at 22:06 on Sat 7:
Sep  7 22:06:07 hostname kernel: oracle invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Sep  7 22:06:07 hostname kernel: oracle invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Sep  7 22:06:07 hostname kernel: oracle cpuset=/ mems_allowed=0
Sep  7 22:06:07 hostname kernel: Pid: 14507, comm: oracle Not tainted 2.6.32-358.14.1.el6.x86_64 #1
  • At the time, we still had over 16 GB of free HugePages for Oracle:
AnonHugePages:    147456 kB
HugePages_Total:   12800
HugePages_Free:     8636
HugePages_Rsvd:     1647
HugePages_Surp:        0
Hugepagesize:       2048 kB

Resolution

  • Adding addition non-dedicated memory to the server resolved the issue.

Root Cause

  • Insufficient RAM installed to the system for the application load put upon it.

Diagnostic Steps

  • From the provided sosreport it seems most likely that too much of this system's memory is allocated to HugePages. Allocating memory to HugePages sets it aside for applications making specific memory allocation calls for shared memory areas, such as an Oracle database's SGA, but blocks that memory from use by applications making standard memory allocation calls, the kernel included. With ~91% of this system's memory allocated to HugePages, it is likely that it simply could not keep up with the demand for non-HugePage memory.
    *At the time this sosreport was collected, only about 1/3 of the allocated HugePages on this system were in use:
AnonHugePages:    147456 kB
HugePages_Total:   12800  <===== Allocated
HugePages_Free:     8640  <===== Free
HugePages_Rsvd:     1651
HugePages_Surp:        0
Hugepagesize:       2048 kB

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.