Live Migration error after update compute nodes
Issue
- After executing a minor update, live migration fails with the following error message:
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: Running command: '/usr/sbin/libvirtd --listen'
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: + exec /usr/sbin/libvirtd --listen
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: info : libvirt version: 4.5.0, package: 33.el7_8.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-03-23-10:21:01, x86
-vm-28.build.eng.bos.redhat.com)
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: info : hostname: overcloud-compute-0.localdomain
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: debug : virLogParseOutputs:1760 : outputs=3:stderr
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: debug : virLogParseOutput:1588 : output=3:stderr
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.624+0000: 560645: info : libvirt version: 4.5.0, package: 33.el7_8.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-03-23-10:21:01, x86
-vm-28.build.eng.bos.redhat.com)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.624+0000: 560645: info : hostname: overcloud-compute-0.localdomain
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.624+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-del': object 'objlibvirt_migrate_t
ls0' not found
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.626+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-add': Unable to access credentials
/etc/pki/qemu/ca-cert.pem: No such file or directory
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.867+0000: 560645: error : virNetClientProgramDispatchError:174 : migration successfully aborted
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.873+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-del': object 'objlibvirt_migrate_t
ls0' not found
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.874+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-del': object 'libvirt_migrate-secr
et0' not found
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: Traceback (most recent call last):
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: timer()
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: cb(*args, **kw)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: waiter.switch(result)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: result = function(*args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: return func(*args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7139, in _live_migration_operation
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: LOG.error("Live Migration failure: %s", e, instance=instance)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: self.force_reraise()
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: six.reraise(self.type_, self.value, self.tb)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7133, in _live_migration_operation
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: bandwidth=CONF.libvirt.live_migration_bandwidth)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 674, in migrate
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: destination, params=params, flags=flags)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: result = proxy_call(self._autowrap, f, *args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: rv = execute(f, *args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: six.reraise(c, e, tb)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: rv = meth(*args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1779, in migrateToURI3
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: libvirtError: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file or directory
Environment
- Red Hat OpenStack Platform 13.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.