OCP4: Network performance issues for Pod to Pod communications when OCP cluster is hosted on OCP-Virt

Issue

• Throughput for pod to pod communications on Container Native Virtualization (CNV) nodes is considerably lower when nodes are on different bare metal hosts when an MTU is set to 8900 for the nodes.
• Throughput for bare metal to bare metal (bm2bm), pod to pod (pod2pod), and CNV node to CNV node (cnv2cnv) is still high.
• Lowering MTU to 3600 for the CNV nodes does increase throughput for pod to pod on CNV (pod_on_cnv2pod_on_cnv), but cnv2cnv throughput decreases by 50%.
• pod_on_cnv2pod_on_cnv throughput on CNV nodes with an MTU of 8900 is less than 1% of cnv2cnv throughput.
• Throughput testing was replicated with iperf3
• After each iperf3 test there is an increase of InCsumErrors from netstat -s output on the receiving CNV node.
• Additionally using retis on the CNV node receiving the traffic will log UDP_CSUM.
• Packet captures on the CNV node receiving traffic will show traffic flowing to br-ex, but not forwarded to the veth interface of the receiving pod.