Bonding Mode: IEEE 802.3ad Showing Slave Interface: nic6 - MII Status: down

Latest response

Hi Experts

I have configured Bonding Mode: IEEE 802.3ad Dynamic link aggregation on my 8 linux servers. I am getting issue only on 3 servers rest 5 are looks good.

If i am checking the status of NIC bonding. Slave Interface: nic6 - MII Status: showing down anyone having idea why MII Status is showing down?

Related Information
Server OS version RHEL 7.5
Kernel Version - 3.10.0-862.11.6.el7.x86_64

Command OutPut
cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 14:02:ec:86:24:e0
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 15
Partner Key: 32897
Partner Mac Address: 00:23:04:ee:be:01

Slave Interface: nic3
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 14:02:ec:86:24:e0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 14:02:ec:86:24:e0
port key: 15
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 4000
system mac address: 00:23:04:ee:be:01
oper key: 32897
port priority: 32768
port number: 12064
port state: 61

Slave Interface: nic6
MII Status: down
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 14:02:ec:86:24:e1
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 2
Partner Churned Count: 3
details actor lacp pdu:
system priority: 65535
system mac address: 14:02:ec:86:24:e0
port key: 0
port priority: 255
port number: 2
port state: 5
details partner lacp pdu:
system priority: 4000
system mac address: 00:23:04:ee:be:01
oper key: 32897
port priority: 32768
port number: 12064
port state: 69

As i have checked both NIC`s individually both are showing up see the below output.
NIC -1
ethtool nic3
Settings for nic3:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

NIC - 2

ethtool nic6
Settings for nic6:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Device Info -

ethtool -i nic6
driver: ixgbe
version: 5.1.0-k-rh7.5
firmware-version: 0x80000835
expansion-rom-version:
bus-info: 0000:05:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Both the ports are up and running but still MII Status: down. Need your suggestions.

Let me know if you require any other information aprart from this.

Regards
JS

Responses

The bond isn't polling the slaves:

MII Status: up
MII Polling Interval (ms): 0

Try add miimon=100 to your bonding options, that is BONDING_OPTS in the ifcfg-bond0 file.

If you are running NetworkManager, make sure you nmcli con reload to make NM aware of the changes. You could also edit with nmcli or nmtui.

I expect that nmcli con down bond0 and nmcli con up bond0 should apply the change. Confirm with procfs as you have above.

If you have tried to configure ARP monitoring, that isn't compliant with 802.3ad so shouldn't work.

Hi Thanks for the response, Could you please explain more about miimon=100 as my other servers working fine with miimon=100 in Bonding_Opts.

Currently i have - BONDING_OPTS=" mode=802.3ad"

Moreover, we are not using NetworkManager in this.

Bonding needs a method to monitor its slave link, otherwise it doesn't know the link state of the slave, nor does bonding react when the slave link state changes. The two monitors are the miimon which watches physical link, and the ARP monitor which uses traffic but isn't compatible with LACP.

The default should be to miimon at an interval of 100ms. It's not clear why this didn't apply to your systems. It's a good idea to always specify the proper link monitoring mode.

As I understand to the issue correctly the bond does not respect the configured miimon in the BONDING_OPTS and leaves the miimon on the default zero. This is definitely BUG and the LACP will not cope with this in case of failover. The miimon threads will never run and the link is never detected. Manual fix is to set up the miimon in the /sys/class/net//bonding/miimon to some value.

The problem of having miimon on zero with LACP is nicely described here: https://lkml.org/lkml/2018/7/18/840

The problem is currently under investigation and possibly some BZ will be used to track this.

Regards Michal Tesar