With "cpu_policy='dedicated" set in the flavor Nova fails to try scheduling the virtual instance on other NUMA domains with ample resources in RHOSP.
Issue
- With
cpu_policy='dedicated'
set in the flavor Nova fails to try scheduling the virtual instance on other NUMA domains with ample resources. - Compute always ends up shutting off VM with due to
OOM-killer
trigger even though memory available on otherNUMA
. - Even there is sufficient memory on
NUMA 0
computesshutoff
one of the VM andout of memory error
is thrown as seen inmessages
logs:
kernel: tp_librbd invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
kernel: tp_librbd cpuset=emulator mems_allowed=0
kernel: CPU: 30 PID: 9378 Comm: tp_librbd Tainted: G OE ------------ 3.10.0-693.11.6.el7.x86_64 #1
kernel: Hardware name: XXXXXXXX
kernel: Call Trace:
kernel: [<ffffffff816a5ea1>] dump_stack+0x19/0x1b
kernel: [<ffffffff816a1296>] dump_header+0x90/0x229
kernel: [<ffffffff812b9dfb>] ? cred_has_capability+0x6b/0x120
kernel: [<ffffffff81188094>] oom_kill_process+0x254/0x3d0
kernel: [<ffffffff812b9fde>] ? selinux_capable+0x2e/0x40
kernel: [<ffffffff811888d6>] out_of_memory+0x4b6/0x4f0
kernel: [<ffffffff816a1d9a>] __alloc_pages_slowpath+0x5d6/0x724
kernel: [<ffffffff8118eaa5>] __alloc_pages_nodemask+0x405/0x420
kernel: [<ffffffff811d6075>] alloc_pages_vma+0xb5/0x200
kernel: [<ffffffff811b42a0>] handle_mm_fault+0xb60/0xfa0
kernel: [<ffffffff816b37e4>] __do_page_fault+0x154/0x450
kernel: [<ffffffff816b3b15>] do_page_fault+0x35/0x90
Per-node system memory usage (in MBs):
Node 0 Node 1 Total
------ ------ ------
MemTotal 130951 131072 262023
MemFree 78908 121499 200407
MemUsed 52043 9573 61616
Active 46945 3794 50739
- VM's are only
CPU
pinned and no other metadata. - Instance goes into shutdown state with due to OOM killer triggered on the compute even when there is ample memory in the other NUMA domain
Environment
- Red Hat OpenStack Platform 10
- Red Hat OpenStack Platform 13
Nova
Scheduler.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.