Bonding NICs with differing speeds.
I have a server with dual 10Gb network devices as well as dual 1Gb cards. Currently the two 10Gb cards are bonded and the two 1Gb cards are not enabled. The current bonding mode is active-backup. As I understand it this means that all of our traffic is being handled by one interface. I want to separate the traffic on the server between application data and database traffic which is through a private interconnect to another database server. I've proposed bonding each of the 10Gb cards with one of the 1Gb cards in mode 1. Is there any reason why this wouldn't work?
Thanks,
Mike
Responses
Guru
6435 points
We've been running our NetBackup media servers with pairs of 10/1, asymmetrical, active/passive bonds for years now. Performs well and no hitches with the design we've put in place.
Are you looking for more than a "yes you can/should" or "no you can't/shouldn't" response (e.g., "make sure you're pedantic in your bond-config so as to force preference for the 10Gbps bond-member")?
Guru
6863 points
Tom - I would like to hear about the details. With 10GB becoming more available and 1GB not going away soon, I know I will be in the exact same situation as you with my media servers.
One specific question I have, can you bond a bond? (all mode=active/passive in this example)
bond3
/ \
/ \
bond0 bond1
/ \ / \
p1p1 em1 p2p1 em2
(hopefully that text formatting worked out ;-) There are several different bonding scenarios that I have wondered about, both mode=active/passive and mode=802.3ad and each of them "cascaded" but I don't have the ability to test any of these approaches in my environment.
Thanks!
Hello, No, nesting of bonds will not work.
See Is it possible to configure bonding over bonded interface in Red Hat Enterprise Linux?
Guru
6435 points
You can't bond bonds within the bonding driver itself, no. If you were running an additional abstraction layers on top (some clustering solutions afford that type of functionality) you can do it. However, the primary benefit is resiliency rather than carrying-capacity. TL;DR the effort to do it generally isn't worth the payoff.
With ours, the choice of two 10/1s was driven by a couple of factors:
* High port-costs for 10Gbps: management freaked at the idea of idle 10Gbps ports
* Our networking folks (globe-spanning enterprise's data regions) ability to effectively and efficiently deal with LACP was lacking. Most of our networking expertise was vested at the architecture and engineering level, not the Ops level.
* The desire to separate client<->media server traffic from media server <->(deduplicating) nearline storage arrays (NAS devices). This was mostly driven by not knowing whether putting ingest and NAS traffic on the same links would introduce contention.
What we settled on was a 10/1 (A/P) pair for client ingest and a 10/1 (A/P) pair for media server to image-storage transit. Ultimately, with the optimization-ratios we'd tracked between the amount of data ingested from clients versus the amount written to the NASes, we probably could have gotten away with 10/1 client-ingest with 1/1 between server-to-NAS. To understate things: OST-enabled deduplicating NAS devices are kinda cool. =)
At each site, we had two media servers for resiliency. Server A's 10Gbps links were patched through switch A and its 1Gbps links through switch B. Server B's 10Gbps links were patched through switch B and its 1Gbps links through switch A. That way, in the event of loss of either switch, we could ensure having a minimum of one media server cranking at 10Gbps of ingest and 10Gbps of egress. Who knew that the NBU optimization plugins would have made it so that wasn't strictly necessary.
We could have done 10/10 LACPs, possibly with VLAN tagging on top and IP-based hashing. However, that would really only have made sense for the client link and/or a shared ingest/egress bond (with VLANing). The storage array's network I/O algorithms precluded us from realizing meaningful, LACP-based carrying-capacity benefits on (what was essentially) a point-to-point link. So, it wasn't worth trying to walk the regional NetOps guys through enabling it ...was hard enough getting them to actually do the resilient-wiring correctly. :p
Hello
From the Networking Guide:
primary=
Specifies the interface name, such as eth0, of the primary device. The primary device is the first of the bonding interfaces to be used and is not abandoned unless it fails. This setting is particularly useful when one NIC in the bonding interface is faster and, therefore, able to handle a bigger load.
This setting is only valid when the bonding interface is in active-backup mode. See https://www.kernel.org/doc/Documentation/networking/bonding.txt for more information.
Source: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Using_Channel_Bonding.html#s3-modules-bonding-directives
Thank you
Guru
6435 points
Sounds like you're doing something RAC-like? If so, you might want to factor in the I/O characteristics of you client-facing interfaces, the carrying-requirements of your node-to-node communications and the clustering-solution's capabilities to balance I/O.
If your DB clients are able to regularly saturate a 10Gbps link, you might benefit from 10/10 LACP on the front end.
Depending on how you've distributed your concurrent database's data-handling (how well you're able encourage to node-affinity), it may make more sense to let your DB or clustering software handle the balancing of inter-node I/O across the slower links ...but that's a big if kind of thing. Still, several of the DB-clustering technologies I've dealt with prefer to own the raw NICs and do the I/O balancing themselves rather than relying on the OS providing a resilient interface.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.