With "cpu_policy='dedicated" set in the flavor Nova fails to try scheduling the virtual instance on other NUMA domains with ample resources in RHOSP.

Solution Unverified - Updated -

Issue

  • With cpu_policy='dedicated' set in the flavor Nova fails to try scheduling the virtual instance on other NUMA domains with ample resources.
  • Compute always ends up shutting off VM with due to OOM-killer trigger even though memory available on other NUMA.
  • Even there is sufficient memory on NUMA 0 computes shutoff one of the VM and out of memory error is thrown as seen in messages logs:
 kernel: tp_librbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
 kernel: tp_librbd cpuset=emulator mems_allowed=0
 kernel: CPU: 30 PID: 9378 Comm: tp_librbd Tainted: G           OE  ------------   3.10.0-693.11.6.el7.x86_64 #1
 kernel: Hardware name:  XXXXXXXX
 kernel: Call Trace:
 kernel: [<ffffffff816a5ea1>] dump_stack+0x19/0x1b
 kernel: [<ffffffff816a1296>] dump_header+0x90/0x229
 kernel: [<ffffffff812b9dfb>] ? cred_has_capability+0x6b/0x120
 kernel: [<ffffffff81188094>] oom_kill_process+0x254/0x3d0
 kernel: [<ffffffff812b9fde>] ? selinux_capable+0x2e/0x40
 kernel: [<ffffffff811888d6>] out_of_memory+0x4b6/0x4f0
 kernel: [<ffffffff816a1d9a>] __alloc_pages_slowpath+0x5d6/0x724
 kernel: [<ffffffff8118eaa5>] __alloc_pages_nodemask+0x405/0x420
 kernel: [<ffffffff811d6075>] alloc_pages_vma+0xb5/0x200
 kernel: [<ffffffff811b42a0>] handle_mm_fault+0xb60/0xfa0
 kernel: [<ffffffff816b37e4>] __do_page_fault+0x154/0x450
 kernel: [<ffffffff816b3b15>] do_page_fault+0x35/0x90

Per-node system memory usage (in MBs):
                 Node 0 Node 1  Total
                 ------ ------ ------
MemTotal         130951 131072 262023
MemFree           78908 121499 200407
MemUsed           52043   9573  61616
Active            46945   3794  50739
  • VM's are only CPU pinned and no other metadata.
  • Instance goes into shutdown state with due to OOM killer triggered on the compute even when there is ample memory in the other NUMA domain

Environment

  • Red Hat OpenStack Platform (RHOSP)10.0.
  • Nova Scheduler.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In