Error: the container name (...) is already in use by (...) But container does not exist in OpenShift 4.x
Issue
Pod cannot spawn due to:
the container name "k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0" is already in use by "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv". You have to remove that container to be able to reuse that name.: that name is already in use
Pods show create container error:
# oc get pods -A -o wide | grep -iv runn | grep -iv compl
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
openshift-service-ca-operator service-ca-operator-aaaaaaaaaa-bbbbb 0/1 CreateContainerError 0 114m 192.168.3.34 master01.example.com <none> <none>
Events for the pod show
# oc describe pod -n openshift-service-ca-operator service-ca-operator-aaaaaaaaaa-bbbbb | tail -n 20
service-ca-operator-token-ccccc:
Type: Secret (a volume populated by a Secret)
SecretName: service-ca-operator-token-ccccc
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 120s
node.kubernetes.io/unreachable:NoExecute for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned openshift-service-ca-operator/service-ca-operator-aaaaaaaaaa-bbbbb to master01.example.com
Warning FailedCreatePodSandBox 111m kubelet, master01.example.com Failed create pod sandbox: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Warning FailedCreatePodSandBox 111m kubelet, master01.example.com Failed create pod sandbox: rpc error: code = Unknown desc = error reserving pod name k8s_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0 for id xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: name is reserved
Warning FailedCreatePodSandBox 111m kubelet, master01.example.com Failed create pod sandbox: rpc error: code = Unknown desc = error reserving pod name k8s_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0 for id yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy: name is reserved
Warning Failed 111m kubelet, master01.example.com Error: relabel failed /var/run/containers/storage/overlay-containers/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz/userdata/resolv.conf: lstat /var/run/containers/storage/overlay-containers/zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz/userdata/resolv.conf: no such file or directory
Warning Failed 109m (x10 over 111m) kubelet, master01.example.com Error: the container name "k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0" is already in use by "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv". You have to remove that container to be able to reuse that name.: that name is already in use
Normal Pulled 78s (x506 over 111m) kubelet, master01.example.com Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:<checksum>" already present on machine
When inspecting the node, one can see that the container with that ID does not exist:
[root@master01 ~]# crictl ps -a | grep "vvvvvvv"
[root@master01 ~]# podman ps -a | grep "vvvvvv"
[root@master01 ~]# crictl ps -a | grep ca-operator
[root@master01 ~]# podman ps -a | grep ca-operator
However, the container does exist in /var/lib/containers/storage/overlay-containers/containers.json
:
[root@master01 ~]# jq . /var/lib/containers/storage/overlay-containers/containers.json | grep "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv" -B1 -A12
{
"id": "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv",
"names": [
"k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0"
],
"image": "<image UUID>",
"layer": "<layer UUID>",
"metadata": "{\"pod-name\":\"k8s_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0\",\"pod-id\":\"zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz\",\"image-name\":\"<image UUID>\",\"image-id\":\"<image UUID>\",\"name\":\"k8s_operator_service-ca-operator-aaaaaaaaaa-bbbbb_openshift-service-ca-operator_dddddddd-eeee-ffff-gggg-hhhhhhhhhhhh_0\",\"metadata-name\":\"operator\",\"created-at\":1589288627}",
"created": "2020-05-12T13:03:47.639544652Z",
"flags": {
"MountLabel": "system_u:object_r:container_file_t:s0:c699,c769",
"ProcessLabel": "system_u:system_r:container_t:s0:c699,c769"
}
},
Furthermore, it is not possible to delete the container:
[root@master01 ~]# crictl rm vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Removing the container "vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv" failed: rpc error: code = Unknown desc = container with ID starting with vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv not found: ID does not exist
[root@master01 ~]# podman rm vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Error: no container with name or ID vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv found: no such container
[root@master01 ~]#
Environment
Red Hat OpenShift Container Platform 4.3.13
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.