KVM Bridge networking is a real pain
Hello everyone,
I am experiencing lots of troubles with KVM and bridge networking. I know bridges in KVM are painful, but I am ready to get them work at any price.
My setup is the classic one:
LAN --> Firewall --> DMZ --> Gateway --> Internet
The issue here is that Firewall is a KVM-virtualized VM on RHEL host. The host has this configuration:
1) RHEL 6.5 as bare metal KVM Host with TWO network interfaces (eth0 is the motherboard-embedded one, and eth1 is the PCI one).
2) RHEL 6.5 is hosting two VM:
2.1) VM1 is basically a firewall router, so it needs bridge access to both network interfaces (eth0 for LAN, eth1 for DMZ)
2.2) VM2 is a MySQL server that needs bridged LAN connectivity (eth0), so LAN computers can reach VM2 with its LAN IP address or domain name.
3) RHEL 6.5 Host LAN/DMZ connectivity are both required for maintenance (LAN) and updates (DMZ). Host can connect to DMZ vía VM1 or directly vía eth1, choosing either way is up to you.
At network level, LAN and DMZ are two different VLAN that works fine. RHEL Host does not recieve VLAN-tagged traffic because is connected to different access ports on the switch. Say eth0 is connected to port 1 and eth1 is connected to port 2. STP in enabled on all switches.
I have created two network bridges on the RHEL Host, br0 and br1, mapped one each with their appropiate pysical devices eth0 and eth1. VM1 uses both bridges (both set in virt-manager --> VM1 --> networking --> bridge br0/1). VM1 guest have static IP on guest eth0 (br0) and dhcp IP on guest eth1 (br1). The same thing on the Host, which has static IP on eth0 and currently no IP at all on eth1 (to get Internet access vía VM1). Host has only one specified gateway on eth0.
The result is that VM1 randomly loses connectivity on both LAN and WAN ports for short periods of time.
I am currently thinking that it could be a MAC address table mismatch between the host and the VMs or an IP misconfiguration on the RHEL Host.
So please, anyone have tried to virtualize with KVM one server with two bridged network adapters that works?
Thanks a lot :)
Responses
Red Hat
Guru
7257 points
The hypervisor should not have IPs on its physical interfaces, it should have its IPs on the bridges. Say you have eth0 in br0, then put the hypervisor's IP on br0.
Other than that, what you've described should be a working setup.
Do your NICs have SR-IOV? We've seen where the internal SR-IOV switch reflects broadcast (ie: ARP) which confuses the bridge's MAC table, so the bridge thinks the VM is accessible via the external port instead of the vnet tap. Try turning SR-IOV off.
If you suspect some other MAC table issue, you can convert the bridge from a "switch" (builds Layer 2 forwarding database) into a "hub" (broadcasts on all ports) with brctl setageing <brname> 0 to test that theory.
If any of this is multicast communication, RHEL 6.5 will drop multicast memberships where there is no querier on the LAN. This is somewhat improved in RHEL 6.6, which broadcasts mcast out all bridgeports if no querier is detected on the LAN.
Maybe it's a spanning tree thing, try turning STP off and setting portfast on your physical switchports?
The Linux bridge is a pretty good piece of kit, not painful at all. It's just a software implementation of a Layer 2 switch. Think of it like a Cisco 3550 and your vnet tap devices as cables.
Oh, and don't use NetworkManager, it can't handle bridges in EL6. You can turn it off in chkconfig and set NM_CONTROLLED=no in your ifcfg-* files, or uninstall the NetworkManager package altogether.
In EL7 NM is great but not in EL6.
Hi Jorge,
I am sorry you had so much trouble getting this setup to work. I am one of the technical writers responsible for documenting KVM and want to make sure your experience is not shared by others. Can you tell me if you had read a misguided procedure somewhere that gave you the wrong impression or didn't let you know that you needed to remove the IP address from the physical interface and place it on the bridge? Let me know and I will open a bug against the documentation and make sure it is clearer.
Thanks in advance
Laura N.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.