The oom_score_adj for aqmon processes, which are run by conmon, should normally be set to 0. However, it is incorrectly set to -1000 after termination by the oom-killer in the context of a Memory Cgroup OOM event.

Solution Unverified - Updated -

Issue

  • The oom_score_adj for aqmon processes, which are run by conmon, should normally be set to 0. However, it is incorrectly set to -1000 after termination by the oom-killer in the context of a Memory Cgroup OOM event.
  • When the processes are running with oom_score_adj of 0 as expected, they can be killed upon memcg OOM as expected:
    ...
Jun 19 09:44:20 hostname kernel: Tasks state (memory values in pages):
Jun 19 09:44:20 hostname kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jun 19 09:44:20 hostname kernel: [2005859]  2000 2005859     2929      641    61440        0             0 aqmon.sh <<-------
Jun 19 09:44:20 hostname kernel: [2005874]  2000 2005874   755653   548439  4907008        0             0 aqmon    <<-------
Jun 19 09:44:20 hostname kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=libpod-xxx.scope,mems_allowed=0,oom_memcg=/machine.slice/libpod-xxx.scope,task_memcg=/machine.slice/libpod-xxx.scope,task=aqmon,pid=2005874,uid=2000
Jun 19 09:44:20 hostname kernel: Memory cgroup out of memory: Killed process 2005874 (aqmon) total-vm:3022612kB, anon-rss:2089288kB, file-rss:104468kB, shmem-rss:0kB, UID:2000 pgtables:4792kB oom_score_adj:0
    ...
  • On the other hand, when the processes are running with oom_score_adj of -1000, the OOM event occurrence ends up as "no killable processes.":
    ...
Jun 19 06:01:13 hostname kernel: Tasks state (memory values in pages):
Jun 19 06:01:13 hostname kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jun 19 06:01:13 hostname kernel: [ 226491]  2000 226491     2929      627    65536        0         -1000 aqmon.sh <<-------
Jun 19 06:01:13 hostname kernel: [ 226504]  2000 226504   736094   527288  4902912        0         -1000 aqmon    <<-------
Jun 19 06:01:13 hostname kernel: Out of memory and no killable processes...
    ...

Environment

  • Red Hat Enterprise Linux 8.9.z
  • RHEL-shipped conmon and podman-docker packages

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content