Severe network latency observed when communicating to another application server within same subnet.

Issue

• Network latency observed between the newly added RHEL server and the existing application RHEL server of same subnet is not more than 5 Mbps even though the NIC/link speed is of 10000Mb/s on both the ends.
• Another RHEL system of same subnet acting as a client is correctly delivering throughput of around 8 Gbps when connecting to same destination RHEL server.
• There is a Juniper 5100 switch in between the client and server.
• Throughput is tested via iperf3 and below are the observations;

SERVER:
-------
[root@SERVER ~]# iperf3 -i 2 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.0.1, port 27636
[  5] local 10.0.0.0 port 5201 connected to 10.0.0.1 port 27638
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-2.31   sec  1.62 MBytes  5.90 Mbits/sec
[  5]   2.31-4.25   sec  1.38 MBytes  5.96 Mbits/sec
[...]
[...]
[ ID] Interval           Transfer     Bandwidth       Retr
[  5]   0.00-60.24  sec  35.9 MBytes  4.99 Mbits/sec    0             sender
[  5]   0.00-60.24  sec  35.8 MBytes  4.98 Mbits/sec                  receiver


CLIENT:
-------
[root@CLIENT ~]# iperf3 -i 2 -t 60 -c SERVER-IP
Connecting to host 10.0.0.0, port 5201
[  4] local 10.0.0.1 port 27638 connected to 10.0.0.0 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-2.00   sec  1.69 MBytes  7.08 Mbits/sec    0   42.4 KBytes
[  4]   2.00-4.00   sec  1.37 MBytes  5.73 Mbits/sec    0   42.4 KBytes
[  4]   4.00-6.00   sec  1.49 MBytes  6.26 Mbits/sec    0   42.4 KBytes
[...]
[...]
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  35.9 MBytes  5.01 Mbits/sec    0             sender
[  4]   0.00-60.00  sec  35.8 MBytes  5.00 Mbits/sec                  receiver