Error: the container name (...) is already in use by (...) But container does not exist in OpenShift 4.x

Solution Verified - Updated -

Issue

Pod cannot spawn due to:

the container name "k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0" is already in use by "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv". You have to remove that container to be able to reuse that name.: that name is already in use

Pods show create container error:

# oc get pods -A -o wide | grep -iv runn | grep -iv compl
NAMESPACE                                               NAME                                                              READY   STATUS                 RESTARTS   AGE     IP               NODE                                     NOMINATED NODE   READINESS GATES
openshift-service-ca-operator                           service-ca-operator-aaaaaaaaaa-bbbbb                              0/1     CreateContainerError   0          114m    192.168.3.34      master01.example.com   <none>           <none>

Events for the pod show

# oc describe pod -n openshift-service-ca-operator service-ca-operator-aaaaaaaaaa-bbbbb  | tail -n 20
  service-ca-operator-token-ccccc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  service-ca-operator-token-ccccc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 120s
                 node.kubernetes.io/unreachable:NoExecute for 120s
Events:
  Type     Reason                  Age                   From                                             Message
  ----     ------                  ----                  ----                                             -------
  Normal   Scheduled               <unknown>             default-scheduler                                Successfully assigned openshift-service-ca-operator/service-ca-operator-aaaaaaaaaa-bbbbb to master01.example.com
  Warning  FailedCreatePodSandBox  111m                  kubelet, master01.example.com  Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  FailedCreatePodSandBox  111m                  kubelet, master01.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = error reserving pod name k8s_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0 for id xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: name is reserved
  Warning  FailedCreatePodSandBox  111m                  kubelet, master01.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = error reserving pod name k8s_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0 for id yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy: name is reserved
  Warning  Failed                  111m                  kubelet, master01.example.com  Error: relabel failed /var/run/containers/storage/overlay-containers/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz/userdata/resolv.conf: lstat /var/run/containers/storage/overlay-containers/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz/userdata/resolv.conf: no such file or directory
  Warning  Failed                  109m (x10 over 111m)  kubelet, master01.example.com  Error: the container name "k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0" is already in use by "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv". You have to remove that container to be able to reuse that name.: that name is already in use
  Normal   Pulled                  78s (x506 over 111m)  kubelet, master01.example.com  Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<checksum>" already present on machine

When inspecting the node, one can see that the container with that ID does not exist:

[root@master01 ~]# crictl ps -a | grep "vvvvvvv"
[root@master01 ~]# podman ps -a | grep "vvvvvv"
[root@master01 ~]# crictl ps -a | grep ca-operator
[root@master01 ~]# podman ps -a | grep ca-operator

However, the container does exist in /var/lib/containers/storage/overlay-containers/containers.json:

[root@master01 ~]# jq . /var/lib/containers/storage/overlay-containers/containers.json | grep "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv"  -B1 -A12
  {
    "id": "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv",
    "names": [
      "k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0"
    ],
    "image": "<image UUID>",
    "layer": "<layer UUID>",
    "metadata": "{\"pod-name\":\"k8s_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0\",\"pod-id\":\"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\",\"image-name\":\"<image UUID>\",\"image-id\":\"<image UUID>\",\"name\":\"k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0\",\"metadata-name\":\"operator\",\"created-at\":1589288627}",
    "created": "2020-05-12T13:03:47.639544652Z",
    "flags": {
      "MountLabel": "system_u:object_r:container_file_t:s0:c699,c769",
      "ProcessLabel": "system_u:system_r:container_t:s0:c699,c769"
    }
  },

Furthermore, it is not possible to delete the container:

[root@master01 ~]# crictl rm vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Removing the container "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv" failed: rpc error: code = Unknown desc = container with ID starting with vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv not found: ID does not exist
[root@master01 ~]# podman rm vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Error: no container with name or ID vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv found: no such container
[root@master01 ~]#

Environment

Red Hat OpenShift Container Platform 4.3.13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content