Failing to create pod sandbox on OpenShift 3 and 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 3.x
    • 4.x

Issue

  • Getting the following error when trying to restart a pod:
  • While installing elasticsearch operator in ocp4.x
  Failed create pod sandbox: rpc error code: = Unknown desc = [failed to set up sandbox container.
  • Getting NetworkPlugin cni failed to set up pod error message.

Resolution

  • Delete the OpenShift SDN pod in error state identified in Diagnostics Steps field:

    $ oc delete pod ${podname}

  • Fix of upstream dns sever resolved the issue

Root Cause

  • One of the OpenShift SDN pods in that particular namespace was corrupted. So, there was no network available to run pods.
  • From the operator pod , it was not resolving the quay.io and hence the upstream dns server was checked and found issue was there.

Diagnostic Steps

Run the following command to inspect pods state and check the output for OpenShift SND pods in error state:

$ oc get pods --all-namespaces
openshift-sdn                       ovs-wrzr9                                        1/1       Running             4          94d
openshift-sdn                       ovs-xg2wd                                        1/1       Running             7          94d
openshift-sdn                       ovs-xtrsr                                        1/1       Running             11644      94d
openshift-sdn                       ovs-z6jps                                        1/1       Running             3          94d
openshift-sdn                       ovs-zphdl                                        1/1       Running             8          94d
openshift-sdn                       ovs-zqtfg                                        1/1       Running             6          94d

NOTE: the list above shows that pod ovs-xtrsr had restarted 11644 since creation. That is the one to be recreated.

  • For OCP4.3 and Elasticsearch operator issue
# oc rsh certified-operators-866f85886d-5b6h9
sh-4.2$ nslookup quay.io
Server:         192.168.100.1
Address:        192.168.100.1#53

Non-authoritative answer:
Name:   quay.io.abcd.example.com
Address: 192.168.100.200 <---upstream dns server
** server can't find quay.io.abcd.example..com: SERVFAIL

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.