Sudden instance shutdown due to out of memory

Solution In Progress - Updated -

Issue

  • Mulptiple VMs on two compute nodes shutdown suddenly due to out of memory.

  • oom-killer was invoked and killed guests:

Aug 24 12:15:01 overcloud-compute-4 kernel: CPU 1/KVM invoked oom-killer: gfp_mask=0x6280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=1, order=0, oom_score_adj=0
Aug 24 12:15:01 overcloud-compute-4 kernel: CPU 1/KVM cpuset=vcpu1 mems_allowed=1
Aug 24 12:15:01 overcloud-compute-4 kernel: CPU: 21 PID: 140442 Comm: CPU 1/KVM Kdump: loaded Not tainted 4.18.0-193.47.1.el8_2.x86_64 #1
Aug 24 12:15:01 overcloud-compute-4 kernel: Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 10/21/2019
...
Aug 24 12:15:01 overcloud-compute-4 kernel: active_anon:29978388 inactive_anon:368 isolated_anon:0#012 active_file:352329 inactive_file:192952 isolated_file:0#012 unevictable:50177 dirty:111 writeback:0 unstable:0#012 slab_reclaimable:122043 slab_unreclaimable:246769#012 mapped:53531 shmem:946 pagetables:66326 bounce:0#012 free:171811 free_pcp:276 free_cma:0
Aug 24 12:15:01 overcloud-compute-4 kernel: Node 1 active_anon:59033468kB inactive_anon:828kB active_file:0kB inactive_file:4kB unevictable:119304kB isolated(anon):0kB isolated(file):0kB mapped:9608kB dirty:0kB writeback:0kB shmem:1812kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 58318848kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Aug 24 12:15:01 overcloud-compute-4 kernel: Node 1 Normal free:44792kB min:45224kB low:111240kB high:177256kB active_anon:59033468kB inactive_anon:828kB active_file:0kB inactive_file:4kB unevictable:119304kB writepending:0kB present:67108864kB managed:66025368kB mlocked:119304kB kernel_stack:9368kB pagetables:129444kB bounce:0kB free_pcp:1104kB local_pcp:152kB free_cma:0kB
Aug 24 12:15:01 overcloud-compute-4 kernel: lowmem_reserve[]: 0 0 0 0 0
  • VMs are pinned to a specific numa cell:
  <cputune>
    <shares>4096</shares>
    <vcpupin vcpu='0' cpuset='11'/>
    <vcpupin vcpu='1' cpuset='39'/>
    <vcpupin vcpu='2' cpuset='41'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <emulatorpin cpuset='11,13,39,41'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
  </numatune>

Environment

  • Red Hat OpenStack Platform 16.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content