Failed to remove a pod which contains containers created with podman-remote
Issue
-
When we remove a pod which contains containers created with podman-remote, it is possible that podman fails to remove the pod with an error message as shown below. This bug has occurred when the container to be removed only exists.
Error: error removing container 6b587caef6ef5274eb2b1df7fac0cfe0c7ed548f4c97533d238d54c6beeba470 root filesystem: 1 error occurred: * unlinkat /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged: device or resource busy -
The status of the remained pod is "Error" and we can't remove the pod with the rm command.
# podman pod ps POD ID NAME STATUS CREATED INFRA ID # OF CONTAINERS 1161fd4c103b testpo Error 3 minutes ago 9b481b7e04c1 2 WARN[0000] Found incomplete layer "8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4", deleting it # podman pod rm -f testpo Error: error freeing lock for container 9b481b7e04c1f2f3ecde8ed057c356d8165e35ccc2eb56a1c3c518ac254a9c7c: no such file or directoryAlso the status of the remained container is "Removing" and we can't remove the infra container.
# podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9b481b7e04c1 localhost/podman-pause:4.2.0-1661526018 About a minute ago Removing 1161fd4c103b-infra 6b587caef6ef registry.access.redhat.com/ubi8:latest sleep inf About a minute ago Removing testcon WARN[0000] Found incomplete layer "8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4", deleting it # podman rm -f 1161fd4c103b-infra Error: container 9b481b7e04c1f2f3ecde8ed057c356d8165e35ccc2eb56a1c3c518ac254a9c7c is the infra container of pod 1161fd4c103b58c76eb9e5ab4852e7a7b72af42306070b6b17ac46eeeab7ba21 and cannot be removed without removing the pod -
The mount points are remained and cannot be unmounted.
# mount | grep /var/lib/containers/storage overlay on /var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/merged type overlay (rw,nodev,relatime,context="system_u:object_r:container_file_t:s0:c7,c445",lowerdir=/var/lib/containers/storage/overlay/l/QAGE442XL5GDRWPDXETXQQPAJZ,upperdir=/var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/diff,workdir=/var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/work,metacopy=on) overlay on /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged type overlay (rw,nodev,relatime,context="system_u:object_r:container_file_t:s0:c7,c445",lowerdir=/var/lib/containers/storage/overlay/l/IPMRCYO6E7Y2645DHNHQF3ESCO,upperdir=/var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/diff,workdir=/var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/work,metacopy=on) /dev/mapper/rhel-root on /var/lib/containers/storage/overlay type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) # umount var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/merged umount: var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/merged: no mount point specified. # umount /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged umount: /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged: not mounted.
Environment
- Red Hat Enterprise Linux 8.7 and later
- Red Hat Enterprise Linux 9.1 and later
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.