== KVM / VLAN / Bonding / Bridging ==

Latest response

Hi I would like to know the best setup up if I have a 3 VLANs will be used over two physical links enp2s0 , enp3s0 trunked server by default has 2 NICs (note : server capable of creating vNICs as well) so in the current case server by default have vNIC enp2s0 , enp3s0 prepresented to OS

So I will have VLAN 100 with one IP address for management Host bare metal RHEL 7.0

On top of the host will install KVM to deploy 2 VMs

VM1 : using VLAN 200 and - one interface - one IP address VM2 : using VLAN 200, VLAN300 and two interfaces two IP addresses

  • How many bond should I use ... I assume one ? (enp2s0 , enp3s0) ?

    Do I need to create all the VLANs over that bond ? or create multiple vNIC
    like enp4 , enp5 ... on the physical server ?
    what is the proper pinning VLANs ?

  • How many bridge should be created ? knowing the VMs also need to be communicated with each other.

I am looking at any help on how this can be implemented when it comes to routing , pinning ,,, networking etc ?

Thanks in advance

Responses

There are several ways you could set this up, but I would do it like this:

  • Hypervisor

    • enp2s0 and enp3s0 in bond0
    • bond0.100 with hypervisor management IP
    • bond0.200 in br200
    • bond0.300 in br300
  • VMs

    • VLAN200 interface in br200
    • VLAN300 interface in br300

All the VLAN tagging is done on the hypervisor, the VMs just need to have the correct IP addresses for the correct interface.

Think of the bridge as a Layer 2 network switch. It just learns location of stations based on source MAC, and forwards frames based on dest MAC.

If the hypervisor needs an IP in VLAN200 or VLAN300, then add that hypervisor IP on br200 or br300 respectively.

Thanks Jamie for your input :) I also hope this setup works with accessing VMs from in and out of band if possible?

By the way should I care about the device model while setting up the virtual network interface: Hypervisor default - e1000 - rtl8139 - virtio as I see different thoughts on performance and throughput, speed ..etc ? In this design we are using 10G connections. Thanks again

I'm not quite sure what in-band out OOB would be, but you can access the VMs by the VNC/Spice VM console on the hypervisor, and via the VLAN200 and VLAN300 IPs if the rest of your network (firewalls/routers/etc) allows that access.

If you need a VM management IP in VLAN100, then make a br100 which contains bond0.100, put the hypervisor management IP on br100 instead of bond0.100, and put the VM's VLAN100 management interface in br100 as well. The same setup as VLAN200 and 300.

As long as the guest OS supports it, use virtio-net, it will provide the best performance. If the guest OS doesn't support virtio-net then the emulated Intel e1000 will be the next best choice. The emulated Realtek card has the same limitations on max frame size as the real hardware (4000-something bytes) and is just a super old driver.

If virtio-net doesn't meet your performance requirements, then see if your hypervisor's physical NIC supports SR-IOV. If so, create SR-IOV Virtual Functions and pass those through to the VM. That setup is covered in the product doc.

It has previously been not possible to migrate VMs using SR-IOV VFs. I haven't kept up on new developments with that, so it's something to look into if migrating VMs is of concern to you.

You can easily do live and offline migration of VMs using virtio or emulated NICs.

Thanks a lot!

Hi, This is similar to what I am trying to accomplish. I have 4 physical NICs, one I have made a bridge already; ifcfg-bridge1-en1 ifcfg-enp2 ifcfg-enp3 and ifcfg-enp4. Each of those NICs are different VLANs and Subnets. VLAN 24, 101, 102 and 104 respectively. I want to have a KVM/QEMU qcow/VM attach to all of those subnets/VLANs. I am not sure how achieve this. In short I would do this; in VMware by creating virtual switches and add 4 NICs to the VM and add each NIC to each Virtual switch. I tried creating 4 bridges but as soons as I create the second bridge I lost connectivity to VLAN 24 (ifcfg-bridge1-en1). I then have to iLO to the physical box and reset network-scripts back to normal. Could someone point me in the right direction or if this is even possible with KVM. Example:

[root@hostname network-scripts]# cat ifcfg-bridge1-en1
DEVICE="bridge1-en7"
ONBOOT="yes"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="10.24.0.20"
NETMASK="255.255.255.224"
GATEWAY="10.24.0.1"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
DHCPV6C="no"
STP="on"
DELAY="0.0"

[root@hostname bkup]# cat ifcfg-bridge2-en2
DEVICE="bridge2-en8"
ONBOOT="yes"
TYPE="Bridge"
BOOTPROTO="none"
IPADDR="192.168.1.1"
NETMASK="255.255.255.0"
GATEWAY="192.168.1.2"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
DHCPV6C="no"
STP="on"
DELAY="0.0"

As I understand, the switch is doing the VLAN tagging, so you've currently got something like:

VLAN24 ---- NIC en1 ---- bridge-en1
VLAN101 ---- NIC enp2
VLAN102 ---- NIC enp3
VLAN104 ---- NIC enp4

You are correct that you create more bridges, to end up with something like:

VLAN24 ---- NIC en1 ---- bridge-en1
VLAN101 ---- NIC enp2 ---- bridge-enp2
VLAN102 ---- NIC enp3 ---- bridge-enp3
VLAN104 ---- NIC enp4 ---- bridge-enp4

Where you probably went wrong is adding that second GATEWAY parameter to the second bridge, which likely started sending all external traffic via 192.168.1.2 instead of 10.24.0.1 and so you lost connectivity to the hypervisor. You can only have one default gateway on the system.

The bridge is a software implementation of a Layer 2 switch. It just shuffles frames back and forth between the external NIC and any VMs which are placed in the bridge. I haven't used VMware in a long time, but think of the bridge like a vSwitch and that'll be pretty close.

Inside the VM you can use whatever routing and gateway config you like. The Layer 3 configuration inside the VMs is independent of the Layer 3 configuration on the hypervisor.

Thanks Jamie Bainbridge I thought I was overlooking the obvious which is usually the case. Jeff

Hello All, I have a similar situation which as raised by user: AA earlier in this thread. And I did the configuration as pointed out by Jamie on 16-Jan-2018. Following are my requirement:

Hypervisor with IP segment: 10.2.3.40 (VLAN303)
Three VMs : 10.2.3.46 (VLAN303), 10.19.1.46 (VLAN319) and 10.2.4.100 (VLAN304)

So, I created bond0 (from NICs eth0 and eth1), followed by bond0.303, bond0.319 and bond0.304. Following to this, three bridges br303, br319 and br304 were created. The required IP addresses are created in respective bridges.

  • Configuration file for bond0
        BONDING_OPTS=mode=active-backup
       TYPE=Bond
    BONDING_MASTER=yes
    MTU=1500
    NAME=bond0
    UUID=fcfa3f19-890f-457a-9d67-b8d3aad1a43b
    DEVICE=bond0
    ONBOOT=yes
  • Configuration file for bond0.303
    VLAN=yes
    TYPE=Vlan
    PHYSDEV=bond0
    VLAN_ID=303
    REORDER_HDR=yes
    GVRP=no
    MVRP=no
    MTU=1500
    NAME=bond0.303
    DEVICE=bond0.303
    ONBOOT=yes
    BRIDGE=br303
  • Configuration file for br303
    STP=yes
    BRIDGING_OPTS=priority=32768
    TYPE=Bridge
    MTU=1500
    PROXY_METHOD=none
    BROWSER_ONLY=no
    IPV6INIT=no
    NAME=br303
    #UUID=1b2b5f59-af60-41f5-bf55-74b18f5e1776
    DEVICE=br303
    ONBOOT=yes
    IPADDR=10.2.3.40
    NETMASK=255.255.255.0
    GATEWAY=10.2.3.1

With the above configuration, with virt-manager, I created my first VM (10.2.3.46) and chose Network Source as Bridge br303: Host device bond0.303.

Following are IP address configuration inside VM

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.2.3.46  netmask 255.255.255.0  broadcast 10.2.3.255
        inet6 fe80::85c6:c2:f155:2328  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:15:81:b2  txqueuelen 1000  (Ethernet)
        RX packets 1587  bytes 183470 (179.1 KiB)
        RX errors 0  dropped 10  overruns 0  frame 0
        TX packets 349  bytes 39156 (38.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

But now, I am unable to communicate between VM and Host Machine even though both are under same IP segment. Can someone throw me some light on where I missed in configuration files. After this, I have to work with VMs of other IP segments.

Thanks in Advance..

regards, M. Dinesh Ram

That looks correct to me.

We sometimes see a situation where the external network reflects the VM's broadcast traffic (like ARP Requests) back to the hypervisor, so the bridge learns the VM is "outside" the hypervisor and connectivity fails:

Try bridge fdb | grep b2 to look for where the bridge thinks the VM is connected.

You should see something like this, which is the permanent vnet interface to the VM, and the VM's MAC address behind that vnet:

fe:54:00:15:81:b2 dev vnetX master br303 permanent
52:54:00:15:81:b2 dev vnetX master br303 

However, if you see this, where the VM's MAC address is behind the bond:

fe:54:00:15:81:b2 dev vnetX master br303 permanent
52:54:00:15:81:b2 dev bond0.303 master br303
                      ^^wrong^^

Then that's wrong. As described on the above knowledgebase solution, you can temporarily work around that by disabling ageing timer on the bridge, which turns it into a broadcasting "hub" instead of a learning "switch". You can then work with your network team to find out why that reflection happens and stop it, so you can remove the workaround later.

Dear Jamie,

Thanks for your Response !!

I changed the bridge configuration as provided in the link ie..,

 BRIDGING_OPTS="ageing_time=0"

However I found the some issues on placing IP address values at > bond0 and > bridge303 and presented here with two cases for understanding:

Case-I

ifcfg-bond0 => No IPADDR configured
ifcfg-br303 =>  IP configured as 10.2.3.40

*Observation*

Hypervisor (10.2.3.40) to Gateway (10.2.3.1) Communication => Not Communicating
Hypervisor to VM (10.2.3.46)                                                            => Communicating


# bridge fdb | grep b2
fe:54:00:15:81:b2 dev vnet0 vlan 1 master br303 permanent 
52:54:00:15:81:b2 dev vnet0 master br303 
fe:54:00:15:81:b2 dev vnet0 master br303 permanent 
33:33:ff:15:81:b2 dev vnet0 self permanent

Case-II

ifcfg-bond0 => IP configured as 10.2.3.40 (and also tried with 10.2.3.47 as case-III)
ifcfg-br303 => IP configured 10.2.3.40 

*Observation*

Hypervisor to Gateway Communication => Communicating
Hypervisor to VM                                           => Not Communicating


# bridge fdb | grep b2
33:33:ff:15:81:b2 dev vnet0 self permanent

# bridge fdb  |grep 6c
33:33:ff:4f:e3:6c dev em1 self permanent
b0:26:28:4f:e3:6c dev bond0.303 vlan 1 master br303 permanent
b0:26:28:4f:e3:6c dev bond0.303 master br303 permanent
33:33:ff:4f:e3:6c dev br303 self permanent

Under the above cases, there is a trade off in communication between Gateway, Hypervisor and VM. Can you suggest on how to overcome this situation and have a communication among all the three devices.

regards, M. Dinesh Ram

It looks like the switchport to the hypervisor is already in VLAN303. It could also be a trunk port with native VLAN 303.

Case 1 has the hypervisor bond0 already in VLAN303, then you add an IP on br303 which adds an additional VLAN303 tag and you can communicate with the VM.

Then Case 2 has the hypervisor communicating on bond0 with untagged traffic and can reach the rest of VLAN303 (because the port is already in VLAN303 natively), but the bond0.303 into br303 setup adds another VLAN303 tag to the VM which is incorrect for the apparent switchport setup.

I suspect you wish to ask your network team to configure the hypervisor's switchports as trunk ports, and to trunk VLAN303, 304, and 319 down that trunk port.

Then go back to your first config (bond0, bond0.303 in br303, bond0.304 in br304, bond0.319 in br139, with the hypervisor IP addresses on the bridges not the bonds) and things should work.

Thanks Jamie for you inputs,

Good to see the VM with VLAN303 and Gateway are pinging now....

I am going to create other VMs (with br304, br319) with the same Hypervisor VLAN IP ie., > 10.2.3.40 and same Gateway on all the bridges.

regards, M. Dinesh Ram