With "cpu_policy='dedicated" set in the flavor Nova fails to try scheduling the virtual instance on other NUMA domains with ample resources in RHOSP.

Solution Verified - Updated -

Issue

  • With cpu_policy='dedicated' set in the flavor Nova fails to try scheduling the virtual instance on other NUMA domains with ample resources.
  • Compute always ends up shutting off VM with due to OOM-killer trigger even though memory available on other NUMA.
  • Even there is sufficient memory on NUMA 0 computes shutoff one of the VM and out of memory error is thrown as seen in messages logs:
 kernel: tp_librbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
 kernel: tp_librbd cpuset=emulator mems_allowed=0
 kernel: CPU: 30 PID: 9378 Comm: tp_librbd Tainted: G           OE  ------------   3.10.0-693.11.6.el7.x86_64 #1
 kernel: Hardware name:  XXXXXXXX
 kernel: Call Trace:
 kernel: [<ffffffff816a5ea1>] dump_stack+0x19/0x1b
 kernel: [<ffffffff816a1296>] dump_header+0x90/0x229
 kernel: [<ffffffff812b9dfb>] ? cred_has_capability+0x6b/0x120
 kernel: [<ffffffff81188094>] oom_kill_process+0x254/0x3d0
 kernel: [<ffffffff812b9fde>] ? selinux_capable+0x2e/0x40
 kernel: [<ffffffff811888d6>] out_of_memory+0x4b6/0x4f0
 kernel: [<ffffffff816a1d9a>] __alloc_pages_slowpath+0x5d6/0x724
 kernel: [<ffffffff8118eaa5>] __alloc_pages_nodemask+0x405/0x420
 kernel: [<ffffffff811d6075>] alloc_pages_vma+0xb5/0x200
 kernel: [<ffffffff811b42a0>] handle_mm_fault+0xb60/0xfa0
 kernel: [<ffffffff816b37e4>] __do_page_fault+0x154/0x450
 kernel: [<ffffffff816b3b15>] do_page_fault+0x35/0x90

Per-node system memory usage (in MBs):
                 Node 0 Node 1  Total
                 ------ ------ ------
MemTotal         130951 131072 262023
MemFree           78908 121499 200407
MemUsed           52043   9573  61616
Active            46945   3794  50739
  • VM's are only CPU pinned and no other metadata.
  • Instance goes into shutdown state with due to OOM killer triggered on the compute even when there is ample memory in the other NUMA domain

Environment

  • Red Hat OpenStack Platform 10
  • Red Hat OpenStack Platform 13
  • Nova Scheduler.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content