Warning message

Log in to add comments or rate this document.

Several "page allocation failure. order:1, mode:0x20" messages are seen on the console after upgrade to Red Hat Enterprise Linux 6.2

Updated 2014-09-11T02:52:05+00:00

Issue

The following warning messages are seen on the console after upgrading to update 2:

Mar 18 14:39:17 hostname kernel: glusterfsd: page allocation failure. order:1, mode:0x20
Mar 18 14:39:17 hostname kernel: swapper: page allocation failure. order:1, mode:0x20
Mar 21 11:27:06 hostname kernel: swapper: page allocation failure. order:1, mode:0x20
Mar 21 18:25:45 hostname kernel: swapper: page allocation failure. order:1, mode:0x20
Mar 21 21:59:18 hostname kernel: swapper: page allocation failure. order:1, mode:0x20
  • Swapper is unable to allocate memory due to page allocation failure:

    kernel: swapper: page allocation failure. order:1, mode:0x20
    kernel: Pid: 0, comm: swapper Not tainted 2.6.32-358.2.1.el6.x86_64 #1
    kernel: Call Trace:
    kernel: <IRQ>  [<ffffffff8112c207>] ? __alloc_pages_nodemask+0x757/0x8d0
    kernel: [<ffffffff81166ab2>] ? kmem_getpages+0x62/0x170
    kernel: [<ffffffff811676ca>] ? fallback_alloc+0x1ba/0x270
    kernel: [<ffffffff8116711f>] ? cache_grow+0x2cf/0x320
    kernel: [<ffffffff81167449>] ? ____cache_alloc_node+0x99/0x160
    kernel: [<ffffffff811683cb>] ? kmem_cache_alloc+0x11b/0x190
    kernel: [<ffffffff81439d58>] ? sk_prot_alloc+0x48/0x1c0
    kernel: [<ffffffff8143ae32>] ? sk_clone+0x22/0x2e0
    kernel: [<ffffffff81489d66>] ? inet_csk_clone+0x16/0xd0
    kernel: [<ffffffff814a2c73>] ? tcp_create_openreq_child+0x23/0x450
    kernel: [<ffffffff814a046d>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    kernel: [<ffffffff814a2a16>] ? tcp_check_req+0x226/0x460
    kernel: [<ffffffff8149ff0b>] ? tcp_v4_do_rcv+0x35b/0x430
    kernel: [<ffffffff81082034>] ? mod_timer+0x144/0x220
    kernel: [<ffffffff814a171e>] ? tcp_v4_rcv+0x4fe/0x8d0
    kernel: [<ffffffff814a171e>] ? tcp_v4_rcv+0x4fe/0x8d0
    kernel: [<ffffffff8147f50d>] ? ip_local_deliver_finish+0xdd/0x2d0
    kernel: [<ffffffff8147f798>] ? ip_local_deliver+0x98/0xa0
    kernel: [<ffffffff8147ec5d>] ? ip_rcv_finish+0x12d/0x440
    kernel: [<ffffffff8147f1e5>] ? ip_rcv+0x275/0x350
    kernel: [<ffffffff814483bb>] ? __netif_receive_skb+0x4ab/0x750
    kernel: [<ffffffff8144a798>] ? netif_receive_skb+0x58/0x60
    kernel: [<ffffffffa008b975>] ? vmxnet3_rq_rx_complete+0x365/0x890 [vmxnet3]
    kernel: [<ffffffff8128d2b0>] ? swiotlb_map_page+0x0/0x100
    kernel: [<ffffffffa008c0f3>] ? vmxnet3_poll_rx_only+0x43/0xc0 [vmxnet3]
    kernel: [<ffffffff8144cf63>] ? net_rx_action+0x103/0x2f0
    kernel: [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
    kernel: [<ffffffff810e1720>] ? handle_IRQ_event+0x60/0x170
    kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
    kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
    kernel: [<ffffffff81076d95>] ? irq_exit+0x85/0x90
    kernel: [<ffffffff81516f15>] ? do_IRQ+0x75/0xf0
    kernel: [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11
    kernel: <EOI>  [<ffffffff8103b90b>] ? native_safe_halt+0xb/0x10
    kernel: [<ffffffff8101495d>] ? default_idle+0x4d/0xb0
    kernel: [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
    kernel: [<ffffffff81506d9c>] ? start_secondary+0x2ac/0x2ef
    

Environment

  • Red Hat Enterprise Linux 6 kernels after 2.6.32-220.el6

Resolution

Fix

Update to kernel-2.6.32-358.el6 or higher, which contains the enhancement described in the Root Cause section below.

  • Please note, this update (or newer) does not completely eliminate the possibility of the occurrence of the page allocation failure.
  • The below mentioned workaround also works in 2.6.32-358.el6 and newer if the issue still persists even after the update.
Workaround

The following tunables can be used in an attempt to alleviate or prevent the reported condition:

  • Increase vm.min_free_kbytes value, for example to a higher value than a single allocation request.
  • Change vm.zone_reclaim_mode to 1 if it's set to zero, so the system can reclaim back memory from cached memory.

Both settings can be set in /etc/sysctl.conf, and loaded using sysctl -p /etc/sysctl.conf.

For more information on these tunables, install the kernel-doc package and refer to file /usr/share/doc/kernel-doc-2.6.32/Documentation/sysctl/vm.txt.

Root Cause

Before RHEL 6.4, kswapd does not try to free contiguous pages. This can cause GFP_ATOMIC allocations requests to fail repeatedly, when nothing else in the system defragments memory. With RHEL 6.4 and newer, kswapd will compact (defragment) free memory, when required.

Please note that allocation failures can still happen. For example, when a larger burst of GFP_ATOMIC allocations occur which kswapd may struggle to keep up with. However, these allocations should eventually succeed.

There are also other more specific cases that can result in page allocation failures and cause additional issues. Please refer to the following articles for more information: