Sudden instance shutdown due to out of memory
Issue
-
Mulptiple VMs on two compute nodes shutdown suddenly due to out of memory.
-
oom-killer was invoked and killed guests:
Aug 24 12:15:01 overcloud-compute-4 kernel: CPU 1/KVM invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=1, order=0, oom_score_adj=0
Aug 24 12:15:01 overcloud-compute-4 kernel: CPU 1/KVM cpuset=vcpu1 mems_allowed=1
Aug 24 12:15:01 overcloud-compute-4 kernel: CPU: 21 PID: 140442 Comm: CPU 1/KVM Kdump: loaded Not tainted 4.18.0-193.47.1.el8_2.x86_64 #1
Aug 24 12:15:01 overcloud-compute-4 kernel: Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
...
Aug 24 12:15:01 overcloud-compute-4 kernel: active_anon:29978388 inactive_anon:368 isolated_anon:0#012 active_file:352329 inactive_file:192952 isolated_file:0#012 unevictable:50177 dirty:111 writeback:0 unstable:0#012 slab_reclaimable:122043 slab_unreclaimable:246769#012 mapped:53531 shmem:946 pagetables:66326 bounce:0#012 free:171811 free_pcp:276 free_cma:0
Aug 24 12:15:01 overcloud-compute-4 kernel: Node 1 active_anon:59033468kB inactive_anon:828kB active_file:0kB inactive_file:4kB unevictable:119304kB isolated(anon):0kB isolated(file):0kB mapped:9608kB dirty:0kB writeback:0kB shmem:1812kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 58318848kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Aug 24 12:15:01 overcloud-compute-4 kernel: Node 1 Normal free:44792kB min:45224kB low:111240kB high:177256kB active_anon:59033468kB inactive_anon:828kB active_file:0kB inactive_file:4kB unevictable:119304kB writepending:0kB present:67108864kB managed:66025368kB mlocked:119304kB kernel_stack:9368kB pagetables:129444kB bounce:0kB free_pcp:1104kB local_pcp:152kB free_cma:0kB
Aug 24 12:15:01 overcloud-compute-4 kernel: lowmem_reserve[]: 0 0 0 0 0
- VMs are pinned to a specific numa cell:
<cputune>
<shares>4096</shares>
<vcpupin vcpu='0' cpuset='11'/>
<vcpupin vcpu='1' cpuset='39'/>
<vcpupin vcpu='2' cpuset='41'/>
<vcpupin vcpu='3' cpuset='13'/>
<emulatorpin cpuset='11,13,39,41'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
Environment
- Red Hat OpenStack Platform 16.1 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.