Creation of PODs with a secondary network interface fails due to the error event "error setting default gateway in cache: wrong routes format"

Issue

• After upgrading the OCP cluster from 4.12/4.13 to 4.14, creation of Pods with a secondary network interface and a POD annotation to override the default-route fails, resulting in the error event error setting default gateway in cache: wrong routes format.

• In this specific case, the Pod's secondary network interface is configured as a MACVLAN type interface via NetworkAttachmentDefinition, and it acquires its IP address from the infrastructure's DHCP server.

• This specific issue was not found in the OCP cluster < 4.13.

      annotations:
        k8s.v1.cni.cncf.io/networks: '[
         {
           "name": "nad-test",
           "namespace": "demo",
           "interface": "net1",
           "default-route": ["192.168.x.x"] 
         }
        ]'


$ oc get po
NAME                     READY   STATUS              RESTARTS   AGE
nginx-7f7b9f7b7f-hvh4g   0/1     ContainerCreating   0          55m


$ oc get events
LAST SEEN   TYPE      REASON                   OBJECT                        MESSAGE
20s         Normal    Scheduled                pod/nginx-7fb6fdf55f-rcf2m    Successfully assigned demo/nginx-7fb6fdf55f-rcf2m to ramesh-ipi-xsln2-worker-0-v9g9t
4s          Normal    AddedInterface           pod/nginx-7fb6fdf55f-rcf2m    Add eth0 [10.129.x.x/23] from ovn-kubernetes
5s          Normal    AddedInterface           pod/nginx-7fb6fdf55f-rcf2m    Add net1 [192.168.x.x/23] from demo/macvlan-demo
4s          Warning   FailedCreatePodSandBox   pod/nginx-7fb6fdf55f-rcf2m    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_nginx-7fb6fdf55f-rcf2m_demo_1d9d864a-dd94-44cc-8a24-a0064db0090f_0(52aa46b97d1819e0eff9d614d62089b093722ec645c8b5277f9606ebe89c7861): error adding pod demo_nginx-7fb6fdf55f-rcf2m to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:52aa46b97d1819e0eff9d614d62089b093722ec645c8b5277f9606ebe89c7861 Netns:/var/run/netns/f0e9243c-ddf8-4b73-9b3e-c8b1eee9b19e IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=demo;K8S_POD_NAME=nginx-7fb6fdf55f-rcf2m;K8S_POD_INFRA_CONTAINER_ID=52aa46b97d1819e0eff9d614d62089b093722ec645c8b5277f9606ebe89c7861;K8S_POD_UID=1d9d864a-dd94-44cc-8a24-a0064db0090f Path: StdinData:[123 34 98 105 110 68 105 114 34 58 34 47 118 97 114 47 108 105 98 47 99 110 105 47 98 105 110 34 44 34 99 104 114 111 111 116 68 105 114 xxxxxx .....]} ContainerID:"52aa46b97d1819e0eff9d614d62089b093722ec645c8b5277f9606ebe89c7861" Netns:"/var/run/netns/f0e9243c-ddf8-4b73-9b3e-c8b1eee9b19e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=demo;K8S_POD_NAME=nginx-7fb6fdf55f-rcf2m;K8S_POD_INFRA_CONTAINER_ID=52aa46b97d1819e0eff9d614d62089b093722ec645c8b5277f9606ebe89c7861;K8S_POD_UID=1d9d864a-dd94-44cc-8a24-a0064db0090f" Path:"" ERRORED: error configuring pod [demo/nginx-7fb6fdf55f-rcf2m] networking: Multus: [demo/nginx-7fb6fdf55f-rcf2m/1d9d864a-dd94-44cc-8a24-a0064db0090f]: *error setting default gateway in cache: wrong routes format:* <nil>...


// Multus POD logs 

$ oc -n openshift-multus logs multus-m88kn |grep -3 -i "error setting default gateway in cache: wrong routes format"|grep -3 "nginx-7f7b9f7b7f-hvh4g"

I0512 11:08:15.054712 2943416 event.go:298] Event(v1.ObjectReference{Kind:"Pod", Namespace:"demo", Name:"nginx-7f7b9f7b7f-hvh4g", UID:"c67a7b8f-e91d-4d88-a5d1-33cf1b49a238", APIVersion:"v1", ResourceVersion:"5496271", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add net1 [192.168.2.137/23] from demo/macvlan-demo

2024-05-12T11:08:15Z [error] Multus: [demo/nginx-7f7b9f7b7f-hvh4g/c67a7b8f-e91d-4d88-a5d1-33cf1b49a238]: error setting default gateway in cache: wrong routes format: <nil>

2024-05-12T11:08:15Z [verbose] ADD finished CNI request ContainerID:"a66748ea8108d93f88da48acef0250ee632722c7205f84da48eaf1cff11a1b36" Netns:"/var/run/netns/64996672-e5e4-4982-9252-1cb731ce6646" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=demo;K8S_POD_NAME=nginx-7f7b9f7b7f-hvh4g;K8S_POD_INFRA_CONTAINER_ID=a66748ea8108d93f88da48acef0250ee632722c7205f84da48eaf1cff11a1b36;K8S_POD_UID=c67a7b8f-e91d-4d88-a5d1-33cf1b49a238" Path:"", result: "", err: error configuring pod [demo/nginx-7f7b9f7b7f-hvh4gcp] networking: Multus: [demo/nginx-7f7b9f7b7f-hvh4gcp/c67a7b8f-e91d-4d88-a5d1-33cf1b49a238]: error setting default gateway in cache: wrong routes format: <nil>