Pods fail to start in OpenShift 3.11 on OpenStack 13 using Kuryr CNI

Solution Verified - Updated -

Issue

  • OpenShift on OpenShift using Kuryr (OSP 14 Image), the Kuryr CNI Daemon keeps failing with error below (pulled from oc logs kuryr-cni-xxxxx), prevents new Pods from getting IP, and prevents Pod from starting. The error below keeps repeating in the Kuryr CNI Pod logs:
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service [-] Error when processing addNetwork request. CNI Params: {'CNI_IFNAME': u'eth0', 'CNI_NETNS': u'/proc/20609/ns/net', 'C
NI_PATH': u'/opt/cni/bin', 'CNI_ARGS': u'IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=docker-registry-1-deploy;K8S_POD_INFRA_CONTAINER_ID=e2df207e0dbb40479a331ced9fa7655952a73b7e8c
c3fd2e5e3759735b7ffd04', 'CNI_DAEMON': u'True', 'CNI_CONFIG_DIR_PATH': u'/etc/cni/net.d', 'CNI_COMMAND': u'ADD', 'CNI_BIN_DIR_PATH': u'/opt/cni/bin', 'CNI_CONTAINERID': u'e2df207e0dbb40479a3
31ced9fa7655952a73b7e8cc3fd2e5e3759735b7ffd04'}: RetryError: RetryError[Attempts: 61, Value: {'eth0': VIFVlanNested(active=False,address=fa:16:3e:1c:45:a0,has_traffic_filtering=False,id=f1be
ddb2-ab98-46b8-8812-305ae2fc9908,network=Network(bc0cbca2-fd44-46ae-89ea-074321b3f00a),plugin='noop',port_profile=<?>,preserve_on_delete=False,vif_name='tapf1beddb2-ab',vlan_id=3252)}]
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service Traceback (most recent call last):
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/cni/daemon/service.py", line 81, in add
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service     vif = self.plugin.add(params)
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/cni/plugins/k8s_cni_registry.py", line 80, in add
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service     vifs = wait_for_active(pod_name)
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service     return Retrying(*dargs, **dkw).call(f, *args, **kw)
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service   File "/usr/lib/python2.7/site-packages/retrying.py", line 231, in call
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service     raise RetryError(attempt)
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service RetryError: RetryError[Attempts: 61, Value: {'eth0': VIFVlanNested(active=False,address=fa:16:3e:1c:45:a0,has_traffic_filtering=False,id=f1beddb2-ab98-46b8-8812-305ae2fc9908,network=Network(bc0cbca2-fd44-46ae-89ea-074321b3f00a),plugin='noop',port_profile=<?>,preserve_on_delete=False,vif_name='tapf1beddb2-ab',vlan_id=3252)}]
2019-08-23 23:32:40.578 180 ERROR kuryr_kubernetes.cni.daemon.service
2019-08-23 23:32:40.581 180 INFO werkzeug [-] 127.0.0.1 - - [23/Aug/2019 23:32:40] "POST /addNetwork HTTP/1.1" 500 -

Environment

  • Red Hat OpenShift Container Platform
    • 3.11
  • Red Hat OpenStack Platform 13
  • OVN
  • Kuryr CNI 0.5.4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content