- Red Hat Enterprise Linux, version 6.4
- We have an oracle server with 28 GB of RAM with 24 GB of HugePages reserved for Oracle. The server spawned oom-killer at 22:06 on Sat 7:
Sep 7 22:06:07 hostname kernel: oracle invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Sep 7 22:06:07 hostname kernel: oracle invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 Sep 7 22:06:07 hostname kernel: oracle cpuset=/ mems_allowed=0 Sep 7 22:06:07 hostname kernel: Pid: 14507, comm: oracle Not tainted 2.6.32-358.14.1.el6.x86_64 #1
- At the time, we still had over 16 GB of free HugePages for Oracle:
AnonHugePages: 147456 kB HugePages_Total: 12800 HugePages_Free: 8636 HugePages_Rsvd: 1647 HugePages_Surp: 0 Hugepagesize: 2048 kB
- Adding addition non-dedicated memory to the server resolved the issue.
- Insufficient RAM installed to the system for the application load put upon it.
- From the provided sosreport it seems most likely that too much of this system's memory is allocated to HugePages. Allocating memory to HugePages sets it aside for applications making specific memory allocation calls for shared memory areas, such as an Oracle database's SGA, but blocks that memory from use by applications making standard memory allocation calls, the kernel included. With ~91% of this system's memory allocated to HugePages, it is likely that it simply could not keep up with the demand for non-HugePage memory.
*At the time this sosreport was collected, only about 1/3 of the allocated HugePages on this system were in use:
AnonHugePages: 147456 kB HugePages_Total: 12800 <===== Allocated HugePages_Free: 8640 <===== Free HugePages_Rsvd: 1651 HugePages_Surp: 0 Hugepagesize: 2048 kB
- Red Hat Enterprise Linux
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.