Live Migration error after update compute nodes

Solution In Progress - Updated -

Issue

  • After executing a minor update, live migration fails with the following error message:
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: Running command: '/usr/sbin/libvirtd --listen'
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: + exec /usr/sbin/libvirtd --listen
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: info : libvirt version: 4.5.0, package: 33.el7_8.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-03-23-10:21:01, x86
-vm-28.build.eng.bos.redhat.com)
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: info : hostname: overcloud-compute-0.localdomain
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: debug : virLogParseOutputs:1760 : outputs=3:stderr
Oct 16 11:34:58 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:34:58.124+0000: 560626: debug : virLogParseOutput:1588 : output=3:stderr
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.624+0000: 560645: info : libvirt version: 4.5.0, package: 33.el7_8.1 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-03-23-10:21:01, x86
-vm-28.build.eng.bos.redhat.com)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.624+0000: 560645: info : hostname: overcloud-compute-0.localdomain
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.624+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-del': object 'objlibvirt_migrate_t
ls0' not found
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.626+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-add': Unable to access credentials
 /etc/pki/qemu/ca-cert.pem: No such file or directory
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.867+0000: 560645: error : virNetClientProgramDispatchError:174 : migration successfully aborted
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.873+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-del': object 'objlibvirt_migrate_t
ls0' not found
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: 2020-10-16 11:36:46.874+0000: 560645: error : qemuMonitorJSONCheckError:396 : internal error: unable to execute QEMU command 'object-del': object 'libvirt_migrate-secr
et0' not found
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: Traceback (most recent call last):
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     timer()
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     cb(*args, **kw)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     waiter.switch(result)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     result = function(*args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     return func(*args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7139, in _live_migration_operation
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     LOG.error("Live Migration failure: %s", e, instance=instance)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     self.force_reraise()
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     six.reraise(self.type_, self.value, self.tb)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7133, in _live_migration_operation
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     bandwidth=CONF.libvirt.live_migration_bandwidth)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 674, in migrate
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     destination, params=params, flags=flags)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     rv = execute(f, *args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     six.reraise(c, e, tb)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     rv = meth(*args, **kwargs)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1779, in migrateToURI3
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]:     if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Oct 16 11:36:46 overcloud-compute-0.localdomain dockerd-current[433488]: libvirtError: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file or directory

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content