External gateway pod deletes may not clean up ECMP routes in OVNKubernetes ICNI 2.0 with Red Hat OpenShift Container Platform 4.7

Issue

External gateway pod deletes may not clean up ECMP routes in OVNKubernetes ICNI 2.0 with Red Hat OpenShift Container Platform 4.7

There are a few scenarios where deleting a pod acting as an external gateway will not clean up routes in the served namespaces. The OVN lr-route entries will persist even after deletion of the ICNI router namespace (including the pod with the required annotations) and the application namespace.

When one inspects the lr-route-list in OVN's northbound DB, one can see invalid routes to non-existing pods, for example routes may persist even though no more ICNI 2.0 gateway pod with the annotation exists.

The following was taken after the ICNI 2.0 gateway pod including its namespace as well as all ICNI application pods including their namespace were deleted, yet the routes persisted in the OVN database:

# oc exec -n openshift-ovn-kubernetes -it $POD -- ovn-nbctl lr-route-list GR_openshift-worker-1
Defaulting container name to ovn-controller.
Use 'oc describe pod/ovnkube-node-7s2kh -n openshift-ovn-kubernetes' to see all of the containers in this pod.
IPv4 Routes
              172.25.2.18           192.168.123.120 src-ip ecmp-symmetric-reply    # <--- invalid route
             172.25.3.171           192.168.123.120 src-ip ecmp-symmetric-reply    # <--- invalid route
            172.24.0.0/14                100.64.0.1 dst-ip
                0.0.0.0/0             192.168.123.1 dst-ip rtoe-GR_openshift-worker-1

The issue may also manifest itself in the presence of multiple different gateways for the same source IP address:

# oc exec -n openshift-ovn-kubernetes -it $POD -- ovn-nbctl lr-route-list GR_openshift-worker-1
IPv4 Routes
            192.168.0.124                192.168.123.123 src-ip ecmp-symmetric-reply
            192.168.0.125                192.168.123.123 src-ip ecmp-symmetric-reply
            192.168.0.129                192.168.123.123 src-ip ecmp ecmp-symmetric-reply
            192.168.0.129                192.168.123.123 src-ip ecmp ecmp-symmetric-reply
(...)