Failed to remove a pod which contains containers created with podman-remote

Solution Verified - Updated -

Issue

  • When we remove a pod which contains containers created with podman-remote, it is possible that podman fails to remove the pod with an error message as shown below. This bug has occurred when the container to be removed only exists.

    Error: error removing container 6b587caef6ef5274eb2b1df7fac0cfe0c7ed548f4c97533d238d54c6beeba470 root filesystem: 1 error occurred:
        * unlinkat /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged: device or resource busy
    
  • The status of the remained pod is "Error" and we can't remove the pod with the rm command.

        # podman pod ps
        POD ID        NAME        STATUS      CREATED        INFRA ID      # OF CONTAINERS
        1161fd4c103b  testpo      Error       3 minutes ago  9b481b7e04c1  2
        WARN[0000] Found incomplete layer "8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4", deleting it
    
        # podman pod rm -f testpo
        Error: error freeing lock for container 9b481b7e04c1f2f3ecde8ed057c356d8165e35ccc2eb56a1c3c518ac254a9c7c: no such file or directory
    

    Also the status of the remained container is "Removing" and we can't remove the infra container.

        # podman ps -a
        CONTAINER ID  IMAGE                                    COMMAND     CREATED             STATUS      PORTS       NAMES
        9b481b7e04c1  localhost/podman-pause:4.2.0-1661526018              About a minute ago  Removing                1161fd4c103b-infra
        6b587caef6ef  registry.access.redhat.com/ubi8:latest   sleep inf   About a minute ago  Removing                testcon
        WARN[0000] Found incomplete layer "8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4", deleting it
    
        # podman rm -f 1161fd4c103b-infra
        Error: container 9b481b7e04c1f2f3ecde8ed057c356d8165e35ccc2eb56a1c3c518ac254a9c7c is the infra container of pod 1161fd4c103b58c76eb9e5ab4852e7a7b72af42306070b6b17ac46eeeab7ba21 and cannot be removed without removing the pod
    
  • The mount points are remained and cannot be unmounted.

        # mount | grep /var/lib/containers/storage
        overlay on /var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/merged type overlay (rw,nodev,relatime,context="system_u:object_r:container_file_t:s0:c7,c445",lowerdir=/var/lib/containers/storage/overlay/l/QAGE442XL5GDRWPDXETXQQPAJZ,upperdir=/var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/diff,workdir=/var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/work,metacopy=on)
        overlay on /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged type overlay (rw,nodev,relatime,context="system_u:object_r:container_file_t:s0:c7,c445",lowerdir=/var/lib/containers/storage/overlay/l/IPMRCYO6E7Y2645DHNHQF3ESCO,upperdir=/var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/diff,workdir=/var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/work,metacopy=on)
        /dev/mapper/rhel-root on /var/lib/containers/storage/overlay type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
    
        # umount var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/merged
        umount: var/lib/containers/storage/overlay/8af6de096453419622bbcdcf673d85ac22416f91e3e3f7bf0eac0168bacbe6f4/merged: no mount point specified.
        # umount /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged
        umount: /var/lib/containers/storage/overlay/ad61b5c3e5e221e9fc4411a2c7713aefa8701d1be3ef67683c13d22bc6ee48a1/merged: not mounted.
    

Environment

  • Red Hat Enterprise Linux 8.7 and later
  • Red Hat Enterprise Linux 9.1 and later

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content