Intermittent packet loss for connections between Nodes with OVN-Kubernetes IPv6

Solution Verified - Updated -

Environment

  • OpenShift 4.x
  • OVN-Kubernetes

Issue

  • There is a small amount of packet loss seen in the cluster using OVN-Kubernetes using IPv6.
  • Using TCP in the overlay network with communication between two Pods on different Nodes, a number of retransmissions can be seen.

Resolution

Red Hat is aware of this issue and is currently working with the relevant engineering teams to have this permanently resolved.

To track this issue, bug-ticket BZ 2078026 can be used.

This issue will be resolved in the OVN-Controller and will be released into OpenShift through the normal release streams.

If you are experiencing this issue or require further information about this issue, please open a case with Red Hat Support.

Root Cause

There is a bug in some versions of the OVN-Controller which results in an incorrect copy of the MAC address for IPv6 Neighbor Discovery packets.

This happens when an IPv6 Neighbor Advertisement packet is received without the ICMPv6 target link layer (TLL) option set. This field is commonly used but is not required by the ICMPv6 specification.

The TLL field is not inspected by the OVN-Controller before the TLL values are copied, resulting in zero values copied. One of these zero values is set during the MAC address learning and is stored in the southbound database (SBDB) until a new rule is learnt.

During this time traffic will route incorrectly for a short period resulting in packet loss.

Diagnostic Steps

1) Enable the Debug Logs for the OVN-Controller using the following commands:

    PODS=$( oc get pods -n openshift-ovn-kubernetes -l app=ovnkube-node -o name | cut -d'/' -f2 )
    for POD in $PODS; do
      oc exec -n openshift-ovn-kubernetes -c ovn-controller $POD -- sh -c "ovn-appctl -t ovn-controller vlog/set jsonrpc:dbg && ovn-appctl -t ovn-controller vlog/set vconn:dbg && ovn-appctl -t ovn-controller vlog/set rconn:dbg && ovn-appctl -t ovn-controller vlog/set inc_proc_eng:dbg"
    done

2) Inspect the logs for lines that contain the following "mac":"00:00:00:00:00:00". An example line has been provided below:

jsonrpc|DBG|tcp:XX.XX.XX.XX:6642: send request, method="transact", params=["OVN_Southbound",{"uuid-name":"row12336049_33a4_4b29_8996_c515f4bbdae5","row":{"ip":"XXXX:XXXX:XXXX::XXBB","mac":"00:00:00:00:00:00","logical_port":"rtoe-GR_ovn-worker2","datapath":["uuid","d6a53cb6-6bec-44d1-bb0b-a883237e87"]},"op":"insert","table":"MAC_Binding"},{"uuid-name":"row82af86dd_f01d_4a9c_a0b6_ac03275417ed","row":{"ip":"XXXX:XXXX:XXXX::XXAA","mac":"00:00:00:00:00:00","logical_port":"rtoe-GR_ovn-worker2","datapath":["uuid","d6a53cb6-6bec-44d1-bb0b-a8903f529e87"]},"op":"insert","table":"MAC_Binding"},{"comment":"ovn-controller","op":"comment"}], id=39

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments