High inode usage on /run due to persistent libpod directories prevents pod scheduling on RHCOS nodes (OCP 4.17)

Solution Verified - Updated -

Issue

  • New pods fail to start
  • Node becomes unschedulable
  • Core system functionality (container runtime) is affected
  • On OpenShift Container Platform 4.17, nodes become unschedulable due to /run filling up with podman-related directories, leading to inode exhaustion. This prevents new pods from being scheduled and results in container creation failures.
  • The /run directory, backed by tmpfs, fills up due to thousands of leftover files under /run/libpod/persist/, resulting in 100% inode usage. This leads to failure in creating containers:
Error: creating container storage: open /run/containers/storage/overlay-layers/.tmp-mountpoints.jsonXXXXXX: no space left on device

Environment

  • OpenShift Container Platform: 4.17
  • RHCOS version: 417.94.202502111408-0
  • Podman version: podman-5.2.2-3.rhaos4.17.el9.x86_64
  • Rootless containers
  • Architecture: x86_64
  • Platform: RHEL CoreOS 9.4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content