Why 'podman ps' or 'podman images' command fail frequently with the error "fatal error: runtime: out of memory" on Red Hat Enterprise Linux 8 after starting Quay containers ?

Solution Verified - Updated -

Issue

  • Why podman ps or podman images command fail frequently with the error "fatal error: runtime: out of memory" on Red Hat Enterprise Linux 8 after starting Quay containers ?

Scenario 1: Failure with podman images command(just a short snip)

# podman images
WARN[0000] Error validating CNI config file /etc/cni/net.d/87-podman-bridge.conflist: [netplugin failed: "fatal error: runtime: out of memory\n\nruntime stack:\nruntime.throw(0x558fc8f70bc5, 0x16)\n\t/usr/lib/golang/src/runtime/panic.go:1116 +0x74 fp=0x7ffe99053c10 sp=0x7ffe99053be0 pc=0x558fc8e0ac34\nruntime.sysMap(0xc000000000, 0x4000000, 0x558fc931e578)\n\t/usr/lib/golang/src/runtime/mem_linux.go:169 +0xc7 fp=0x7ffe99053c50 sp=0x7ffe99053c10 pc=0x558fc8dee627\nruntime.(*mheap).sysAlloc(0x558fc9303b80, 0x400000, 0x0, 0x4)\n\t/usr/lib/golang/src/runtime/malloc.go:727 +0x1d4 fp=0x7ffe99053cf8 sp=0x7ffe99053c50 pc=0x558fc8de2954\nruntime.(*mheap).grow(0x558fc9303b80, 0x1, 0x0)\n\t/usr/lib/golang/src/runtime/mheap.go:1344 +0x85 fp=0x7ffe99053d60 sp=0x7ffe99053cf8 pc=0x558fc8dfdae5\nruntime.(*mheap).allocSpan(0x558fc9303b80, 0x1, 0x7571636f70002a00, 0x558fc931e588, 0x72682e7a62)\n

Scenario 2: Failure with podman -v command

# podman -v
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7f2db18f437f m=4 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f2db18f437f
stack: frame={sp:0x7f2d89da87e0, fp:0x0} stack=[0x7f2d895a9288,0x7f2d89da8e88)

Environment

  • Red Hat Enterprise Linux 8
  • Quay and postgresql database installed on the same node for Proof of Concept
  • Half of the total memory allocated for hugepages
  • vm.overcommit_memory = 2
  • Quay enabled with LDAP and clair

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content