libvirtError: Timed out during operation: cannot acquire state change lock

Issue

• After rebooting KVM host,virtual machine will not start and remains in paused state. When attempting to resume, shutdown or force off the VM, the following error is seen:

  Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/engine.py", line 515, in shutdown_domain
  vm.shutdown()
  File "/usr/share/virt-manager/virtManager/domain.py", line 567, in shutdown
  self.vm.shutdown()
  File "/usr/lib64/python2.4/site-packages/libvirt.py", line 662, in shutdown
  if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
  libvirtError: Timed out during operation: cannot acquire state change lock
  

• This could also happen when trying to live migrate a VM and the following error would be seen in /var/log/nova/nova-compute.log:

2020-06-22 13:58:06.722 88272 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self)
2020-06-22 13:58:06.722 88272 ERROR oslo_messaging.rpc.server libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=qemuProcessReconnect)
2020-06-22 13:58:06.722 88272 ERROR oslo_messaging.rpc.server

• The following error message could be seen in /var/log/messages:

Jun 22 14:30:52 openstack-nova-compute libvirtd: 2020-06-22 21:30:52.697+0000: 89350: error : virProcessKillPainfully:401 : Failed to terminate process 79972 with SIGKILL: Device or resource busy
Jun 22 14:31:15 openstack-nova-compute libvirtd: 2020-06-22 21:31:15.078+0000: 89352: error : virProcessKillPainfully:401 : Failed to terminate process 64013 with SIGKILL: Device or resource busy

• Some qemu-kvm process could be stuck in Zombie state:

# ps aufxg | grep qemu-kvm
qemu       64013  0.0  0.0      0     0 ?        Zl   Feb02  25:56 [qemu-kvm] <defunct>