VMs with multiqueue won't boot

Issue

• We have a few instances which require multiqueue to be enabled but currently when enabled in glance image properties, VMs boot up and shutdown itself right away.

• In /var/log/libvirt/qemu/instance.xml, the following error is seen:

2019-11-11 18:33:34.665+0000: starting up libvirt version: 4.5.0, package: 10.el7_6.4 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2019-01-10-06:11:18, x86-017.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-18.el7_6.1, kernel: 3.10.0-957.1.3.el7.x86_64, hostname: overcloud-compute-0.localdomain
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name guest=instance-000001bb,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-51-instance-000001bb/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client-IBRS,ss=on,hypervisor=on,tsc_adjust=on,stibp=on,ssbd=on,pdpe1gb=on,mpx=off,xsavec=off,xgetbv1=off -m 16384 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid b0a0665a-27b9-4f52-b1ed-abe0f1273ede -smbios 'type=1,manufacturer=Red Hat,product=OpenStack Compute,version=17.0.7-5.el7ost,serial=4c4c4544-0057-5410-8052-b5c04f464b32,uuid=b0a0665a-27b9-4f52-b1ed-abe0f1273ede,family=Virtual Machine' -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=84,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/mnt/1ca738a6e5074f0a8aa855f81a8b682d/volume-91f53dd0-1a05-4964-8277-2df5acf833ef,format=raw,if=none,id=drive-virtio-disk0,serial=91f53dd0-1a05-4964-8277-2df5acf833ef,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1,write-cache=on -netdev tap,fds=86:87:88:89,id=hostnet0,vhost=on,vhostfds=90:91:92:93 -device virtio-net-pci,mq=on,vectors=10,rx_queue_size=512,host_mtu=8896,netdev=hostnet0,id=net0,mac=fa:16:ff:ff:ff:ff,bus=pci.0,addr=0x3 -netdev tap,fds=94:95:96:97,id=hostnet1,vhost=on,vhostfds=98:99:100:101 -device virtio-net-pci,mq=on,vectors=10,rx_queue_size=512,host_mtu=8896,netdev=hostnet1,id=net1,mac=fa:16:ef:ff:ff:ff,bus=pci.0,addr=0x4 -add-fd set=17,fd=103 -chardev pty,id=charserial0,logfile=/dev/fdset/17,logappend=on -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 10.10.10.10:6 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2019-11-11T18:33:35.320879Z qemu-kvm: -chardev pty,id=charserial0,logfile=/dev/fdset/17,logappend=on: char device redirected to /dev/pts/6 (label charserial0)
2019-11-11T18:33:35.324961Z qemu-kvm: -drive file=/var/lib/nova/mnt/1ca738a6e5074f0a8aa855f81a8b682d/volume-91f53dd0-1a05-4964-8277-2df5acf833ef,format=raw,if=none,id=drive-virtio-disk0,serial=91f53dd0-1a05-4964-8277-2df5acf833ef,cache=none,aio=native: 'serial' is deprecated, please use the corresponding option of '-device' instead
2019-11-11T18:34:13.821981Z qemu-kvm: could not enable queue
qemu-kvm: /builddir/build/BUILD/qemu-2.12.0/hw/net/virtio-net.c:590: virtio_net_set_queues: Assertion `!r' failed.
2019-11-11 18:34:14.239+0000: shutting down, reason=crashed

• In /var/log/containers/nova/nova-compute.log, a similar message is seen:

2019-11-08 23:43:20.256 1 WARNING nova.compute.manager [req-aefdd6ba-f896-43d0-b2b9-13f21218e385 - - - - -] [instance: b0a0665a-27b9-4f52-b1ed-abe0f1273ede] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4

• One of the property is set either on the flavor or the image:

nova flavor-key m1.small.2cpu set  hw:vif_multiqueue_enabled=true
openstack image set --property hw_vif_multiqueue_enabled=true  rhel