Failed to upgrade kube-scheduler when upgrading the cluster to 4.0.0-0.11

Solution In Progress - Updated -

Environment

  • Openshift Container Platform
    • 4.x

Issue

  • Failed to upgrade kube-scheduler when upgrading the cluster to 4.0.0-0.11
  • The errors are:
  Warning  FailedCreatePodSandBox  105s (x732 over 6h58m)  kubelet, ip-192.168.0.1.ec2.internal  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-8-ip-192.168.0.1.ec2.internal_openshift-kube-apiserver_854dfb00-59d3-11e9-a0c2-0ea8a7cdd834_0(be2160403b10ac987e500a234ecc2c0a02fba0b6447c034365121c59b443dbfe): Multus: Err adding pod to network "openshift-sdn": Multus: error in invoke Delegate add - "openshift-sdn": CNI request failed with status 400: 'Unauthorized
  • The journalctl logs shows:
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: time="2019-04-08 15:10:18.388209692Z" level=error msg="Error while adding to cni network: Multus: Err adding pod to network "openshift-sdn": Multus: error in invoke Delegate add - "openshift-sdn": CNI request failed with status 400: 'Unauthorized
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: '"
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: 2019-04-08T15:10:18Z [error] Multus: Err adding pod to network "openshift-sdn": Multus: error in invoke Delegate add - "openshift-sdn": CNI request failed with status 400: 'Unauthorized
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: '
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: time="2019-04-08 15:10:18.404559460Z" level=error msg="Error adding network: Multus: Err adding pod to network "openshift-sdn": Multus: error in invoke Delegate add - "openshift-sdn": CNI request failed with status 400: 'Unauthorized
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: '"
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: time="2019-04-08 15:10:18.404601392Z" level=error msg="Error while adding to cni network: Multus: Err adding pod to network "openshift-sdn": Multus: error in invoke Delegate add - "openshift-sdn": CNI request failed with status 400: 'Unauthorized
Apr 08 15:10:18 ip-192.168.0.1 crio[943]: '"
Apr 08 15:10:18 ip-192.168.0.1 systemd[1]: crio-65981de1f1aa0fc7c65e02c7bfad64ce9994e6a30f5cfebd519ecbe1331d4f54.scope: Consumed 37ms CPU time
-- Subject: Resources consumed by unit runtime
-- Defined-By: systemd
-- Support: https://access.redhat.com/support

Resolution

Diagnostic Steps

  • Check the journalctl logs from the node where kube-scheduler pod is running.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.