Unable to start or migrate RHEV VM - fails with "libvirtError: error creating macvtap type of interface: Invalid argument"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Virtualisation (RHEV) 3.4
  • Red Hat Enterprise Linux (RHEL) 6.6 hosts
    • vdsm-4.14.17-1, vdsm-4.14.17-2
    • libvirt-0.10.2-46.el6_6.1, libvirt-0.10.2-46.el6_6.2 <-- notice, libvirt 6.6

Issue

  • VM will not migrate off Hypervisor.
  • New VMs will not start.
  • VDSM reports "libvirtError: error creating macvtap type of interface: Invalid argument".
  • The VM in question uses a custom hook called vmfex. This is not a supported feature for RHEV, however it is available in the upstream project oVirt.

Resolution

Fixed in libvirt-0.10.2-46.el6_6.4, bz#1203367.
Instruction to apply the fix to currently broken environment:

Recommended:

In order to apply the fix, all VMs in the environment would have to be stopped and hosts put in maintenance.
Then libvirt package should be updated on each host.

# yum update -y
- Confirm new libvirt package is installed:
# rpm -q libvirt
libvirt-0.10.2-46.el6_6.4.x86_64

However, if you cannot stop your VMs, there is an option to do this using libvirt qemu hook, attached.

Using libvirt-qemu hook:

Here are the instructions to implement libvirt hook to allow live migration of your current VMs without shutting them down.

IMPORTANT:
The hook will modify definition of all the VMs that have in VM's definition.
Which means, if you are running on the same environment non-RHEV VMs that are using , this hook would mess up those VMs definition on migration.
But I believe you are not running any VMs outside of RHEV on those host, and the hook would be applied correctly only to the VMs using vmfex.

Step I: upgrade libvirt to latest build on your hosts.
Since we do not want to stop the VMs, it's going to be tricky to upgrade the hosts without putting them in maintenance.
We need to apply this to all the hosts in the environment.
!!! All except SPM.
!!! Leave SPM for the end, once you are done with all the rest of the hosts. Then put SPM host in maintenance to allow SPM to relocate to another host.

  • Make sure Power Management is disabled across all the hosts in the environment (including SPM host).
# yum update -y
- Confirm new libvirt package is installed:
# rpm -q libvirt
libvirt-0.10.2-46.el6_6.4.x86_64

Step II: install libvirt hook
- On each host, create the folder:

# mkdir /etc/libvirt/hooks
  • Download to this folder the attached qemu file (basically create that file with the following content):
# cat /etc/libvirt/hooks/qemu

#!/bin/sh

   if [ "${2}" = "migrate" -a "${3}" = "begin" ]; then

     sed -e "s/interface type='direct'/interface type='network'/" \
         -e "s/source dev='eth/source network='direct-pool' dev='eth/"
   fi
  • Change permissions:
# chmod +x /etc/libvirt/hooks/qemu

Step III: restart libvirt and vdsm:
!!! Do not execute this on SPM host

# service vdsmd stop
# initctl stop libvirtd
# service vdsmd start
# service vdsmd status
VDS daemon server is running

  • Activate the host.

  • Enable power management on all the hosts, once done restarting libvirt and vdsm on all the hosts.

Step IV: Perform live migration:

IMPORTANT: this is a must requirement:
You would need to have 1 host to start with without any VMs running on it.
Otherwise there would be conflicts within nics allocations and the results are not predictable.
The problem in the bug is that VMs got nic assigned from the host, but not from vmfex network pool.
That's why we need to start from a clean host, that has all the nics available for incoming migrations.

  • Get 1 updated host without any running VMs on it.
  • Start migrating all VMs from another host into this host.
  • Once all successfully migrated, you can use that host as a new host to migrate VMs from another, affected host.
  • Continue until you have all VMs from all hosts migrated; host by host.
  • At the end also migrate VMs from the SPM to another host.

How would you check that VMs on migration host have the correct definition and libvirt hook actually worked?
For each running VM, you need to confirm that it has <interface type='network> defined for vmfex nic and has also .
For example:

# virsh -r dumpxml vm_name --inactive | grep -A8 'interface type='
    <interface type='network'>
      <mac address='00:1a:4a:9f:01:8a'/>
      <source network='direct-pool'/>
      <virtualport type='802.1Qbh'>
        <parameters profileid='VMFEX_portprofile_1'/>
      </virtualport>
      <target dev='macvtap2'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>

Step V: update SPM host:
Once all the VMs are successfully migrated from SPM host, put the host in maintenance.
Restart libvirt and vdsmd:

# service vdsmd stop
# initctl stop libvirtd
# service vdsmd start
# service vdsmd status
VDS daemon server is running

Activate the host on the Admin Portal.

Step VI: remove libvirt hook:
Once all the VMs are migrated with the hook installed and once all the hosts are updated to the hotfix libvirt build, there is no need to use libvirt hook anymore.
And it should be removed. In order to prevent bugs and confusion in the future.
This will require same procedure as above - stopping vdsmd and libvirtd, preferably when the host is in maintenance; if impossible - without powermgmt enabled and not non SPM host.

I highly recommend removing the hook and testing live migration without it after you are done with steps 1-5.

To remove the hook:

# rm -f /etc/libvirt/hooks/qemu
# service vdsmd stop
# initctl stop libvirtd
# service vdsmd start
# service vdsmd status
VDS daemon server is running

DONE!

Root Cause

The problem was introduced by libvirt6.6 bug:
BZ#1203367 - Fail to Migrate with Bridged network, eth + macvtap ,with different interface name on two hosts

Due to the bug, libvirt didn't copy the correct xml during live migration, which caused those VMs, that successfully migrated to loose their vmfex definition, i.e. interface attached to libvirt vmfex network pool 'direct-pool'. And instead, use one of the host's interfaces dedicated to vmfex directly. So next time a VM would try to start or migrate with correct definition (mixed libvirt versions in a cluster), it will not be able to allocate the device from the pool, since the device would be used outside of the pool directly.

Diagnostic Steps

  • src host vdsm log;
Thread-11582::DEBUG::2014-12-13 14:02:23,587::vm::403::vm.Vm::(_startUnderlyingMigration) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::starting migration to qemu+tls://x.x.x.x/system with miguri tcp://y.y.y.y
Thread-11583::DEBUG::2014-12-13 14:02:23,588::vm::802::vm.Vm::(run) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::migration downtime thread started
Thread-11584::DEBUG::2014-12-13 14:02:23,589::vm::841::vm.Vm::(run) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::starting migration monitor thread
Thread-11582::DEBUG::2014-12-13 14:02:23,824::libvirtconnection::124::root::(wrapper) Unknown libvirterror: ecode: 38 edom: 11 level: 2 message: error creating macvtap type of interface: Invalid argument
Thread-11582::DEBUG::2014-12-13 14:02:23,824::vm::817::vm.Vm::(cancel) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::canceling migration downtime thread
Thread-11582::DEBUG::2014-12-13 14:02:23,824::vm::905::vm.Vm::(stop) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::stopping migration monitor thread
Thread-11583::DEBUG::2014-12-13 14:02:23,824::vm::814::vm.Vm::(run) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::migration downtime thread exiting
Thread-11582::ERROR::2014-12-13 14:02:23,825::vm::266::vm.Vm::(_recover) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::error creating macvtap type of interface: Invalid argument
Thread-11582::ERROR::2014-12-13 14:02:25,896::vm::365::vm.Vm::(run) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::Failed to migrate
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 351, in run
    self._startUnderlyingMigration(time.time())
  File "/usr/share/vdsm/vm.py", line 433, in _startUnderlyingMigration
    None, maxBandwidth)
  File "/usr/share/vdsm/vm.py", line 928, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1210, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: error creating macvtap type of interface: Invalid argument
  • src libvirtd log;
2014-12-13 19:02:23.668+0000: 20068: debug : doPeer2PeerMigrate3:2570 : driver=0x7fefd800b440, sconn=0x7fefcc000930, dconn=0x7fefbc0019d0, vm=0x7fefb0000df0, xmlin=(null), dconnuri=qemu+tls://x.x.x.x/system, uri=tcp://y.y.y.y
51, flags=1003, dname=(null), resource=2000
2014-12-13 19:02:23.668+0000: 20068: debug : qemuMigrationBegin:1267 : driver=0x7fefd800b440, vm=0x7fefb0000df0, xmlin=(null), dname=(null), cookieout=0x7fefe38cc880, cookieoutlen=0x7fefe38cc89c, flags=1003
2014-12-13 19:02:23.668+0000: 20068: debug : qemuDomainObjSetJobPhase:727 : Setting 'migration out' phase to 'begin3'
2014-12-13 19:02:23.669+0000: 20068: debug : qemuProcessAutoDestroyActive:4915 : vm=avdbpci01
2014-12-13 19:02:23.669+0000: 20068: debug : qemuDriverCloseCallbackGet:724 : vm=avdbpci01, uuid=cd253a8b-9d5e-4d54-b469-a1baff5f91e8, conn=(nil)
2014-12-13 19:02:23.669+0000: 20068: debug : qemuDriverCloseCallbackGet:730 : cb=(nil)
2014-12-13 19:02:23.669+0000: 20068: debug : qemuMigrationEatCookie:758 : cookielen=0 cookie='(null)'
2014-12-13 19:02:23.672+0000: 20068: debug : qemuMigrationBakeCookie:735 : cookielen=248 cookie=<qemu-migration>
  <name>avdbpci01</name>
  <uuid>cd253a8b-9d5e-4d54-b469-a1baff5f91e8</uuid>
  <hostname>XXX</hostname>
  <hostuuid>445b6a92-9fe2-4d98-b1ad-f1f8976c6e3b</hostuuid>
  <feature name='lockstate'/>
</qemu-migration>

2014-12-13 19:02:23.674+0000: 20068: debug : qemuDomainDefFormatBuf:1339 : Removing default USB controller from domain 'avdbpci01' for migration compatibility
2014-12-13 19:02:23.674+0000: 20068: debug : doPeer2PeerMigrate3:2587 : Prepare3 0x7fefbc0019d0
2014-12-13 19:02:23.820+0000: 20068: error : virNetClientProgramDispatchError:174 : error creating macvtap type of interface: Invalid argument
2014-12-13 19:02:23.820+0000: 20068: debug : virConnectClose:1449 : conn=0x7fefbc0019d0
2014-12-13 19:02:23.822+0000: 20068: debug : qemuDomainObjEndAsyncJob:1010 : Stopping async job: migration out
  • dest host vdsm log;
Thread-60::DEBUG::2014-12-13 17:54:57,671::vm::3764::vm.Vm::(_waitForIncomingMigrationFinish) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::Waiting 21600 seconds for end of migration
Thread-59::DEBUG::2014-12-13 17:54:57,684::utils::642::root::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_migrate_destination/50_directlun' (cwd None)
Thread-59::DEBUG::2014-12-13 17:54:57,746::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-59::INFO::2014-12-13 17:54:57,747::hooks::100::root::(_runHooksDir) 
Thread-59::DEBUG::2014-12-13 17:54:57,747::utils::642::root::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_migrate_destination/50_hugepages' (cwd None)
Thread-59::DEBUG::2014-12-13 17:54:57,815::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-59::INFO::2014-12-13 17:54:57,816::hooks::100::root::(_runHooksDir) 
Thread-59::DEBUG::2014-12-13 17:54:57,816::utils::642::root::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_migrate_destination/50_vmfex' (cwd None)
Thread-59::DEBUG::2014-12-13 17:54:57,969::utils::662::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-59::INFO::2014-12-13 17:54:57,969::hooks::100::root::(_runHooksDir) 
Thread-59::DEBUG::2014-12-13 17:54:57,969::API::535::vds::(migrationCreate) Destination VM creation succeeded
.........
Thread-60::ERROR::2014-12-13 17:54:58,269::vm::2341::vm.Vm::(_startUnderlyingVm) vmId=`cd253a8b-9d5e-4d54-b469-a1baff5f91e8`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 2319, in _startUnderlyingVm
    self._waitForIncomingMigrationFinish()
  File "/usr/share/vdsm/vm.py", line 3771, in _waitForIncomingMigrationFinish
    self._connection.lookupByUUIDString(self.id),
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 92, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2902, in lookupByUUIDString
    if ret is None:raise libvirtError('virDomainLookupByUUIDString() failed', conn=self)
libvirtError: Domain not found: no domain with matching uuid 'cd253a8b-9d5e-4d54-b469-a1baff5f91e8'
  • dest host libvirtd log;
    <interface type='direct'>
      <mac address='00:1a:dd:1e:00:67'/>
      <source dev='eth7' mode='passthrough'/>
      <virtualport type='802.1Qbh'>
        <parameters profileid='U_SVCS_PCI_1725'/>
      </virtualport>
      <target dev='macvtap3'/>
      <model type='virtio'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='direct'>
      <mac address='00:1a:dd:1e:00:68'/>
      <source dev='eth8' mode='passthrough'/>
      <virtualport type='802.1Qbh'>
        <parameters profileid='U_DATA_PCI_1722'/>
      </virtualport>
      <target dev='macvtap4'/>
      <model type='virtio'/>
      <link state='up'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
..........
2014-12-13 22:54:58.101+0000: 21677: debug : virNetDevMacVLanCreateWithVPortProfile:840 : virNetDevMacVLanCreateWithVPortProfile: VM OPERATION: migrate in start
2014-12-13 22:54:58.131+0000: 21677: error : virNetDevMacVLanCreate:180 : error creating macvtap type of interface: Invalid argument
..........
2014-12-13 22:54:58.135+0000: 21677: error : virFileReadAll:462 : Failed to open file '/var/run/libvirt/qemu/eth8': No such file or directory
2014-12-13 22:54:58.208+0000: 21677: debug : qemuDomainObjEndAsyncJob:1010 : Stopping async job: migration in
  • All the VMs use the 'vmfex' hook, two have two properties, one has three, the rest have only one;
# psql -U engine -c "select vm_name,vm_guid,userdefined_properties from vm_static where userdefined_properties ilike '%vmfex%' order by vm_name;" 
    vm_name    |               vm_guid                |                                                 userdefined_properties                                                 
---------------+--------------------------------------+------------------------------------------------------------------------------------------------------------------------
 arcmona01     | c9774e73-9b90-4585-9d5c-c2f591d3c225 | vmfex={'00:1a:dd:1e:01:33':'U_DATA_EXT_471'}
 avdbp08       | 015d2557-fc88-4627-8b12-70809e545358 | vmfex={'00:1a:dd:1e:a6:54':'U_DATA_EXT_557'}
 avdbpci01     | cd253a8b-9d5e-4d54-b469-a1baff5f91e8 | vmfex={'00:1a:dd:1e:00:67':'U_SVCS_PCI_1725','00:1a:dd:1e:00:68':'U_DATA_PCI_1722'}
 avp2vp01      | 35fc8af4-221c-497e-b611-a3198e5390a3 | vmfex={'00:1a:dd:1e:2d:64':'U_DATA_EXT_431','00:1a:dd:1e:2d:65':'U_DATA_EXT_217','00:1a:dd:1e:a6:44':'U_DATA_EXT_557'}

Beside this basic logs check, here is how to verify the specific problem:

Get the following info from one of the hosts:

# ip -d link show
# virsh -r net-dumpxml direct-pool
# vdsClient -s 0 getAllVmStats

Then make sure that all the macvtap devices in ip -d link show output are allocated to a device in the direct-pool with "connection". See example below. Use vdsClient output just to confirm the number of vmfex devices on running VMs at that time.

Example:

$ grep -o 'macvtap.*eth..' iplinkshow.out
macvtap3@eth25
macvtap1@eth27
macvtap5@eth28
macvtap10@eth30   <-- eth30 is in use!
macvtap0@eth24
macvtap2@eth26
macvtap4@eth29

$ cat netdump.output
<network connections='6'>
  <name>direct-pool</name>
  <uuid>e5021770-4c9d-c59d-fb3a-a75f359e30d7</uuid>
  <forward dev='eth24' mode='passthrough'>
    <interface dev='eth24' connections='1'/>
    <interface dev='eth25' connections='1'/>
    <interface dev='eth26' connections='1'/>
    <interface dev='eth27' connections='1'/>
    <interface dev='eth28' connections='1'/>
    <interface dev='eth29' connections='1'/>
    <interface dev='eth30' />          <--- this device is not assigned from the pool

$ grep -o 'macvtap.*.*' vdsclientallvmstats.out | awk -F: '{ print  $1 }'
macvtap10'
macvtap3'
macvtap2'
macvtap5'
macvtap1'
macvtap0'
macvtap4'

In a faulty setup, not all macvtap@ethX would have assigned to direct-pool, and there would not be a connection assigned for that device in virsh output. For example, see device eth30 above.

Additional way is to check the static/config xml. When running dumpxml on a domain, without specifying inactive argument, you get only status xml. Which is not always equal to config xml representing the domain. In the bug, during live migration, libvirt was incorrectly forwarding status xml to migration destination and that's why vmfex definitions where lost.

Below is the example of correct networking definitions per domain, acquired using the inactive argument:

# virsh -r dumpxml vm_name --inactive | grep -A8 'interface type='
    <interface type='network'>
      <mac address='00:1a:4a:9f:01:8a'/>
      <source network='direct-pool'/>
      <virtualport type='802.1Qbh'>
        <parameters profileid='VMFEX_portprofile_1'/>
      </virtualport>
      <target dev='macvtap2'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>

Below is the output of a status xml, without inactive (the incorrect output):

    <interface type='direct'>
       <mac address='00:1a:4a:9f:01:8a'/>
       <source dev='eth6' mode='passthrough'/>
       <virtualport type='802.1Qbh'>
         <parameters profileid='VMFEX_portprofile_1'/>
       </virtualport>
       <target dev='macvtap2'/>
       <model type='virtio'/>
       <filterref filter='vdsm-no-mac-spoofing'/>
       <link state='up'/>
       <alias name='net1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
     </interface>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments