The LACP of the two interfaces is not stable. Ping falls to 20%

Latest response

The LACP of the two interfaces is not stable.
Ping falls to 20%

Hello,
I am trying to connect RHEL 7.6 to Switch Juniper EX4300 in LACP mode, I mean bond mode = 4 / 802.3ad.
The switch has the following configuration:

show configuration interfaces ae10 
description DESC;
aggregated-ether-options {
    lacp {
        active;
        periodic slow;
    }
}
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members [ OOB_MGNT_DCE_RHEW OOB_MGNT_DCE_Monitoring OOB_MGNT_DCE_HW ];
        }
    }
}

On the server side, this configuration:

# cat /etc/sysconfig/network-scripts/ifcfg-bond0_slave_1
HWADDR=AA:BB:CC:DD:EE:FF
TYPE=Ethernet
NAME="bond0 slave 1"
#UUID=488e980b-2145-4b37-a151-15be8cf2dee0
DEVICE=ens3f0
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ZONE=public
UUID=1c4a0d15-e00d-a2fa-60ef-a83a0f504e79

# cat /etc/sysconfig/network-scripts/ifcfg-bond0_slave_2
HWADDR=00:11:22:33:44:55
TYPE=Ethernet
NAME="bond0 slave 2"
#UUID=504f0fb2-211e-4b38-8fe6-24a7b63dc1ef
DEVICE=ens3f1
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ZONE=public
UUID=9df58229-b454-00ec-03ba-78099296b37f

# cat /etc/sysconfig/network-scripts/ifcfg-bond0
# Generated by VDSM version 4.30.17.1.git0de043f
DEVICE=bond0
BONDING_OPTS="downdelay=0 lacp_rate=fast miimon=1 mode=802.3ad updelay=0"
MACADDR=AA:BB:CC:DD:EE:FF
ONBOOT=yes
MTU=1500
DEFROUTE=no
IPV6INIT=no
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
NAME="Bond bond0"
UUID=ad33d8b0-1f7b-cab9-9447-ba07f855b143

# cat /etc/sysconfig/network-scripts/ifcfg-bond0.343
# Generated by VDSM version 4.30.17.1.git0de043f
DEVICE=bond0.343
VLAN=yes
BRIDGE=ovirtmgmt
ONBOOT=yes
MTU=1500
DEFROUTE=no
NM_CONTROLLED=yes
TYPE=Vlan
PHYSDEV=bond0
VLAN_ID=343
REORDER_HDR=yes
GVRP=no
MVRP=no
NAME="Vlan bond0.343"
UUID=183cec78-9fe0-8701-58de-59bc537df364

# cat /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt
# Generated by VDSM version 4.30.17.1.git0de043f
DEVICE=ovirtmgmt
TYPE=Bridge
STP=no
ONBOOT=yes
IPADDR=10.x.x.19
NETMASK=255.255.255.192
GATEWAY=10.x.x.1
BOOTPROTO=none
MTU=1500
DEFROUTE=yes
NM_CONTROLLED=yes
IPV6INIT=no
DNS1=10.x.x.11
PROXY_METHOD=none
BROWSER_ONLY=no
PREFIX=26
IPV4_FAILURE_FATAL=no
NAME="Bridge ovirtmgmt"
UUID=9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0

Bond looks healthy, but ping responses have about 20% loss.

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 1
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: aa:bb:cc:dd:ee:ff
Active Aggregator Info:
        Aggregator ID: 20
        Number of ports: 2
        Actor Key: 9
        Partner Key: 11
        Partner Mac Address: aa:aa:aa:aa:aa:aa

Slave Interface: ens3f0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: aa:bb:cc:dd:ee:ff
Slave queue ID: 0
Aggregator ID: 20
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: aa:bb:cc:dd:ee:ff
    port key: 9
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 127
    system mac address: aa:aa:aa:aa:aa:aa
    oper key: 11
    port priority: 127
    port number: 22
    port state: 63

Slave Interface: ens3f1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:11:22:33:44:55
Slave queue ID: 0
Aggregator ID: 20
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: aa:bb:cc:dd:ee:ff
    port key: 9
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 127
    system mac address: aa:aa:aa:aa:aa:aa
    oper key: 11
    port priority: 127
    port number: 23
    port state: 63

I thought by default it would work without the need to try different bonding parameters.
Please help find the mistake.

Thank you and best regards Jiří Kameník

Attachments

JK
Log in to join the conversation

Responses

Jamie Bainbridge's picture Red Hat Guru 8357 points
3 September 2019 10:49 PM

Looks like the switch is configured to LACP Slow (one LACPDU every 30 secs) and the bond is configured to LACP Fast (one LACPDU per second).

These need to match at each end, so either change the switch to fast, or the bond to slow.

JK Community Member 34 points
4 September 2019 4:52 AM

OK, good match. I change a bond to:

# cat /etc/sysconfig/network-scripts/ifcfg-bond0
# Generated by VDSM version 4.30.17.1.git0de043f
DEVICE=bond0
BONDING_OPTS="downdelay=0 lacp_rate=slow miimon=1 mode=802.3ad updelay=0"
MACADDR=AA:BB:CC:DD:EE:FF
ONBOOT=yes
MTU=1500
DEFROUTE=no
IPV6INIT=no
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
NAME="Bond bond0"
UUID=ad33d8b0-1f7b-cab9-9447-ba07f855b143

but:

--- 10.x.x.1 ping statistics ---
290 packets transmitted, 222 received, 23% packet loss, time 406565ms
rtt min/avg/max/mdev = 0.297/0.472/1.050/0.116 ms
Jamie Bainbridge's picture Red Hat Guru 8357 points
4 September 2019 6:24 AM

If you look in /proc/net/bonding/bond0 does it now say LACP rate: slow confirming the setting is applied?

If not, you will have to apply the new config file with nmcli con reload then put the bond down and up with nmcli con down bond0 and nmcli con up bond0 (if using NetworkManager), or ifdown bond0; ifup bond0 (if using network initscript).

If you can confirm the bond is running in slow mode, then I'd troubleshoot the physical layer.

Change cables. Try each bond slave individually. Try each switchport individually. If the Juniper interface crosses chassis (MC-LAG) then try just the one chassis.

You could packet capture on each bond slave at the same time (tcpdump -n -i ens3f0 -w /tmp/$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S")_slave1.pcap and tcpdump -n -i ens3f1 -w /tmp/$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S")_slave2.pcap) and on each switchport at the same time (I presume the switch has a way to do this) and see where the lack of ICMP traffic occurs.

Keep in mind lack ping doesn't necessarily mean there's a problem. The device you're pinging could have strict ICMP Ratelimiting or just be too busy to respond to all those pings.

You could also check for packet loss on the RHEL NICs, the easiest way is probably the xsos tool and run xsos --net.

You're very welcome to open a support case if you'd like our help looking at all that data.

JK Community Member 34 points
4 September 2019 8:23 AM 
# cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 1
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow

I don't think it could be on the physical layer, I see this behavior on multiple servers. I have the same configuration. I just don't know where I have a configuration error.

My colleague network administrator sent me the following information:

> show lacp statistics interfaces ae10
Aggregated Interface: ae10
     LACP Rx LACP Tx Unknown Rx Illegal Rx
       ge-0/0/47 29987 125639 0 0
       ge-1/0/28 31242 124005 0 0

For reference, the aggregated interface is called ae10 and consists of the physical ports ge-0/0/47 and ge-1/0/28.

In the attachment I am sending LACP debug log from switch.

Capture must be done on site in DC. Please wait.

JK Community Member 34 points
4 September 2019 10:02 AM

Attachment: dc1sldcerhv41.dcegov.virt-2019-09-04-11-53_slave12.zip (3.54 KB)

it contains tcpdump from both interfaces. I will have the Capture from the switch in a moment.

Jamie Bainbridge's picture Red Hat Guru 8357 points
4 September 2019 10:23 PM

If you filter those on icmp in Wireshark, you can see there are several pings where the remote end never replies. You can merge the two slave captures with mergecap to make that analysis easier.

So the problem is on the switch, or on the network between the switch and the ping target, or on the ping target itself. The cause could be something "bad" like packet loss or it could be something "harmless" like ICMP Ratelimiting.

btw you might not want to attach your organization's binary packet captures here where everyone can access them. I've been sanitising hostnames, MACs, and IPs out of your messages above too.

JK Community Member 34 points
5 September 2019 6:34 AM

I have attached another attachment, it is the capture of packets from the switch side. lacp_from_switch.zip

A colleague of a netter says that while ratelimitting is there, he doubts that we are hitting him. It normally passes from 0 source IP addresses to 0% packet loss. But there is no LACP on.

Other TCP communications, such as SSH, are disintegrating due to LACP errors.

Jamie Bainbridge's picture Red Hat Guru 8357 points
5 September 2019 11:45 PM

Yes, you had that one uploaded earlier. It also shows the ping target not replying.

If RHEL never receives the ping replies, then we will report ping packet loss.

As you say, that packet loss seems to also apply to actual traffic like SSH and other TCP.

So I think you have quantified that given the following network:

System (a) --- (b) switch (c) --- (d) ping target (default gateway)

You have quantified that traffic leaves (a) and reaches (b) successfully, but replies are not delivered back from (b) to (a).

The loss appears to be either within the switch which owns (b) and (c), or on the network between (c) and (d), or on (d) itself.