Error: OCI runtime error: runc: exec failed: container does not exist

Posted on

Hello Community!

I am having some issues with Podman 4.4.1 on RHEL 8.8. I am running an ELK stack on a machine in multiple pods.

  • k8s-elastic = Elasticsearch
  • k8s-kibana = Kibana
  • k8s-logstash = Logstash

These pods are running under the restricted user "elk". The pods are stable, but after some time, I encounter problems accessing the pods via Podman. A podman exec -it k8s-elastic bash command ends with the message:

Error: OCI runtime error: runc: exec failed: container does not exist

A podman stop k8s-elastic ends up in

ERRO[0000] container does not exits.

Even if the bash is no longer accessible - or other commands are not executable via Podman - the pod continues to function; the ELK cluster is operational.

The problem affects all pods. I would be grateful for any hints and/or help.

podman run -d \
-v /opt/k8s-elk/elastic-data/:/usr/share/elasticsearch/data/:z \
-v /opt/k8s-elk/elastic-config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:z \
-v /opt/k8s-elk/elastic-config/jvm.options:/usr/share/elasticsearch/config/jvm.options:z \
-v /opt/k8s-elk/certs/elastic:/usr/share/elasticsearch/config/certs/:z \
-p 9200:9200 \
--env-file=/opt/k8s-elk/elastic-config/elastic.env \
-it --network k8s-elk --name k8s-elastic ac1eef415132


.......................................


# container-4a582d076d40741fadb6ddf1613a49770206a97a22e1fe4921a2933bede5585f.service
# autogenerated by Podman 4.4.1
# Thu Feb  8 09:25:05 CET 2024

[Unit]
Description=Podman container-4a582d076d40741fadb6ddf1613a49770206a97a22e1fe4921a2933bede5585f.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/tmp/podman-run-1001/containers

[Service]
User=elk
Group=elk
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman start 4a582d076d40741fadb6ddf1613a49770206a97a22e1fe4921a2933bede5585f
ExecStop=/usr/bin/podman stop  \
        -t 10 4a582d076d40741fadb6ddf1613a49770206a97a22e1fe4921a2933bede5585f
ExecStopPost=/usr/bin/podman stop  \
        -t 10 4a582d076d40741fadb6ddf1613a49770206a97a22e1fe4921a2933bede5585f
PIDFile=/tmp/podman-run-1001/containers/overlay-containers/4a582d076d40741fadb6ddf1613a49770206a97a22e1fe4921a2933bede5585f/userdata/conmon.pid
Type=forking

[Install]
WantedBy=default.target


podman info
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.6-1.module+el8.8.0+19993+47c8ef84.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: 0dcf8a206aff64a95e52d205642f8a7c7ed45049'
  cpuUtilization:
    idlePercent: 97.09
    systemPercent: 0.57
    userPercent: 2.34
  cpus: 4
  distribution:
    distribution: '"rhel"'
    version: "8.8"
  eventLogger: file
  hostname: myhostname
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 4.18.0-477.36.1.el8_8.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1472061440
  memTotal: 12298850304
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-1.module+el8.8.0+19993+47c8ef84.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      spec: 1.0.2-dev
      go: go1.19.12
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.module+el8.6.0+17704+41732ca0.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 2h 12m 47.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/elk/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 2
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/elk/.local/share/containers/storage
  graphRootAllocated: 104138276864
  graphRootUsed: 10554175488
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /tmp/podman-run-1001/containers
  transientStore: false
  volumePath: /home/elk/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1695119471
  BuiltTime: Tue Sep 19 12:31:11 2023
  GitCommit: ""
  GoVersion: go1.19.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1
SS

Responses