Intelligently autoconfiguring LACP/802.3ad?

Latest response

Hello, everyone --

I was wondering if it's possible for a host to determine which network interfaces are supposed to be included in a LACP bond without sniffing for LACPDUs on each interface.

Let me add a little bit more detail around the scenario that I'm thinking about.

Let's say that I have a host with interfaces eth[0123], and all four interfaces are plugged into a switch. On the switch, the switchports that eth0 and eth1 are plugged into are configured with (Cisco syntax) "channel-group X mode active", but eth2 and eth3 are not in a port-channel. Is it possible, without sniffing frames off each interface, for the host to "see" that eth0 and eth1 should be configured as a bond with 802.3ad/LACP while eth2 and eth3 should be configured individually? Has anyone done this before?

If possible, I'd like to do this at the tail-end of a Kickstart, or maybe from a Puppet-script.

I'm considering lumping all of the interfaces into a bond, then looking at /proc/net/bonding/bond0 to see which interfaces aren't getting along with the rest of them, and pulling the dissidents out. But that seems clunky and I wanted to see if anyone had done anything like this before.

Thanks for your help!
-joe.-

ci
Log in to join the conversation

Responses

Jamie Bainbridge's picture Red Hat Guru 7257 points
25 August 2014 10:38 PM

Both the methods you covered were the first things which sprung to my mind as well.

You could also just brute-force the pair - script bond configuration in all possible pairs (eth0+eth1, eth0+eth2, and so on through to eth2+eth3) and stop when you get to a bond which has two aggregated slaves.

Personally I like your first idea of capturing Slow Protocols on each interface and marking it as "to bond" or "not to bond".

thomas.jones2@dodiis.mil's picture Guru 6435 points
26 August 2014 11:43 PM

LACP is a negotiated configuration. The only way I'm aware of to tell if a link/path is LACP-ready is to look for the handshake packets sent from the (other) active LACP end-point (i.e., "snoop the interfaces").

This, of course, assumes that your switch is set to LACP active-mode. If the switch is in passive-mode, you'd have to solicit, first (sequentially create single-NIC "bundles" to force the LACP capability-query to be sent out, record which links get a reply, then configure your real bundles).

The down side to the brute force method (against passive-mode switch-configs) is that, if your host is configured across redundant switches and the switches haven't been configured to allow switch-spanning bundles, your automated-results may be sub-optimal. You can also get sub-optimal results if your host-side ports are using different NIC-drivers (so, you'll want logic to address this possibility).

james.radtke@siriusxm.com's picture Guru 6863 points
27 August 2014 2:37 AM

Does anyone know if Spanning Tree would flip out of if you configured your 4 interfaces to be in one port-group, yet the interfaces were instead plugged in to potentially several groups? In particular, if that host-side bond presented a single mac address across all the interfaces (or is that only active-passive that does that?).

Also - another reason I would anticipate why that plan may not work. If you happen to be kickstarting your host and your kickstart expects to use eth0 (em1) based on the VLAN for your environment, but instead... eth3 ends up being the primary/preferred for the bond (which happens to be on the wrong VLAN), I would expect that to fail.

I have wondered the same thing as you - does a switch port "advertise" configuration data (in addition to the capabilities that it may advertise - i.e. speed, duplex, etc...)?

thomas.jones2@dodiis.mil's picture Guru 6435 points
27 August 2014 10:54 AM

MACs are generally discrete in a LACP-bond: it's part of how the negotiation keeps track of which leg of the bond to send packets across and keep things properly sequenced.

KickStart is one of the reasons why you'd be running your switches in passive LACP-mode. It allows you to run in "standard" mode until such time as the other end-point announces, "I'm ready for LACP: how bout you?"

As to capabilities advertisement, it's more a question of "what do you mean by advertise". For GigE and higher, autoneg is the default mode (used to be, some 10/100/1000 NICs simply wouldn't run above 100 if you didn't autoneg). Capabilities are "advertised" in as much as the endpoints say to each other, "hey, wanna talk? You do? Can you talk at this speed? You can? Awesome: let's go."

ci Community Member 20 points
28 August 2014 12:05 AM

Hey, everyone --

Thanks for all the responses so far.

So, my buddy (Bob Izenberg) and I were experimenting with various approaches to this, and here's what we found.

When we created a bond and included interfaces that weren't configured on the switch for LACP, we encountered some behavior we didn't expect. When you look at /proc/net/bonding/bond0, the non-LACP enabled interfaces don't appear with a specific error, but rather seem okay since their link is good. They do, however, get their own Aggregator ID. So, it seems like our approach will end up being add all interfaces to a single bond and look for Aggregator IDs that have two or more slave interfaces. See below:

[root@localhost ~]# modprobe -r bonding
[root@localhost ~]# modprobe bonding mode=4 lacp_rate=1
[root@localhost ~]# ifconfig bond0 up
[root@localhost ~]# ifenslave bond0 eno1 eno2 eno3 eno4
[root@localhost ~]# cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 2
    Actor Key: 17
    Partner Key: 32847
    Partner Mac Address: 00:11:22:33:44:01

Slave Interface: eno1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: a1:b2:c3:d4:e5:51
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eno2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: a1:b2:c3:d4:e5:53
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eno3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: a1:b2:c3:d4:e5:55
Aggregator ID: 3
Slave queue ID: 0

Slave Interface: eno4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: a1:b2:c3:d4:e5:57
Aggregator ID: 4
Slave queue ID: 0

So, in this example, the switch is configured to have eno1 and eno2 in a bond, but treat eno3 and eno4 as unbonded ports. You can see that that eno1 and eno2 have the same Aggregator ID, but the Agg ID for eno3 and eno4 are respectively unique. The kernel actually mentions that the "active" Aggregator ID is 1:

Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 2

I haven't tested yet to see what happens when you actually have two bonds configured on the switch (i.e., eno1+eno2->bond0 and eno3+eno4->bond1) but for the time being ... Without parsing /proc/net/bonding/bond0, is there a script-friendly way to see which interfaces are unhappy with the bond? Maybe something in sysfs?

Thanks again... I appreciate it.
-joe.-

thomas.jones2@dodiis.mil's picture Guru 6435 points
28 August 2014 2:01 AM

What I've observed from attempting to LACP-bond interfaces with mismatched drivers (or bundles that span switch-pairs that don't support bond-spanning), is that one set will get one aggregator ID and the other set will get another aggregator ID. You should see similar ID-groupings if there switch side is presenting two different bonding-groups.

BL Community Member 39 points
28 August 2014 10:21 PM

Joe, I was playing with two LLDP clients on RHEL 6 and 7 with a Cat 4948E running 15.2(1)E3. The client which provides the most promise is https://github.com/vincentbernat/lldpd/. There are premade RPMs available here: http://software.opensuse.org/download.html?project=home:vbernat&package=lldpd. I tried the "lldpad" RPM made available by Red Hat, but it does not offer much help (in my limited effort). The "lldpd" and "lldpcli" offer a "show neighbors" which can show link aggregation details (as referenced here: https://github.com/vincentbernat/lldpd/issues/36). Unfortunately I was unable to see these link aggregation details when running it against the Cat4948E using either CDP or LLDP. I plan to try it on Cat 3750G switches and possibly some others in my lab.

There is a documented implementation of using LLDP to detect when LACP should be dynamically configured (http://rickardnobel.se/lacp-and-esxi-5-1/ references VMware ESXi and their virtual switch).

ci Community Member 20 points
29 September 2014 9:30 PM

Hey, everyone --

So, I've written the code that lives in the %pre block and does all the work of figuring which interfaces are meant to be in a bond. It formulates the network command and throws it out to /tmp, to be sourced by a %include directive. I'll share the code once I get this last thing sorted out, but .... I have one last problem.

For some reason, the network command that I'm generating produces an error, even though the exact same command included in the kickstart file statically works without error.

Here's the network command that ends up getting produced:

network --device=bond0 --bootproto=dhcp --onboot=yes --bondopts=mode=active-active --bondslaves=eno1,eno2 --activate

That command, when included statically in the kickstart file works without a problem. However, when the same command is generated from the %pre block and sourced through an %include, I get the following error:

Starting installer, one moment...
anaconda 19.31.79-1 for CentOS 7 started.
03:26:35 Running pre-installation scripts
03:27:36 Not asking for VNC because of an automated install
03:27:36 Not asking for VNC because text mode was explicitly asked for in kickstart


** (anaconda:2263): WARNING **: Could not open X display
An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details


===============================================================================
An unknown error has occurred
===============================================================================
anaconda 19.31.79-1 exception report
Traceback (most recent call first):
  File "/usr/lib64/python2.7/site-packages/gi/types.py", line 113, in function
    return info.invoke(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/gi/overrides/Gio.py", line 171, in __call__
    None)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/nm.py", line 719, in nm_add_connection
    connection = proxy.AddConnection('(a{sa{sv}})', settings)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/network.py", line 553, in add_connection_for_ksdata
    nm.nm_add_connection(values)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/network.py", line 1182, in apply_kickstart
    added_connections = add_connection_for_ksdata(network_data, dev_name)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/network.py", line 1199, in networkInitialize
    devnames = apply_kickstart(ksdata)
  File "/sbin/anaconda", line 1085, in <module>
    networkInitialize(ksdata)
GError: GDBus.Error:org.freedesktop.DBus.GLib.UnmappedError.NmSettingBondErrorQuark.Code3: bond.options: invalid option 'mode' or its value 'active-active'

What do you want to do now?
1) Report Bug
2) Debug
3) Quit

Does anyone have any ideas as to why the network command, when sourced through a %include might be interpreted differently than the same command embedded in the kickstart file?

Thanks again,
-joe.-

BC Community Member 36 points
16 January 2018 10:22 PM

I'm a little late to the party, but for others wandering in from Google, may I suggest quoting the "mode=active-active" statement? produce this network command:

network --device=bond0 --bootproto=dhcp --onboot=yes --bondopts="mode=active-active" --bondslaves=eno1,eno2 --activate

.. and see whether the quoting reduces stumbling around the two equals signs?

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.