Recieve errors while migrating instances within the Overcloud
Issue
- When attempting to migrate instances errors can be seen when viewing the nova migration-list:
# nova migration-list
+---------------------------------+---------------------------------+---------------------------------+---------------------------------+------------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+
| Source Node | Dest Node | Source Compute | Dest Compute | Dest Host | Status | Instance UUID | Old Flavor | New Flavor | Created At | Updated At |
+---------------------------------+---------------------------------+---------------------------------+---------------------------------+------------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+
| overcloud-compute-1.localdomain | overcloud-compute-6.localdomain | overcloud-compute-1.localdomain | overcloud-compute-6.localdomain | 192.0.2.9 | error | <uuid> | 5 | 8 | 2015-10-23T00:30:03.000000 | 2015-10-23T00:30:04.000000 |
| overcloud-compute-6.localdomain | overcloud-compute-7.localdomain | overcloud-compute-6.localdomain | overcloud-compute-7.localdomain | 192.0.2.8 | error | <uuid> | 5 | 5 | 2015-10-23T19:00:35.000000 | 2015-10-23T19:00:36.000000 |
| overcloud-compute-1.localdomain | overcloud-compute-0.localdomain | overcloud-compute-1.localdomain | overcloud-compute-0.localdomain | 192.0.2.19 | error | <uuid> | 5 | 5 | 2015-10-23T19:00:42.000000 | 2015-10-23T19:00:42.000000 |
+---------------------------------+---------------------------------+---------------------------------+---------------------------------+------------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+
- The following traceback can be seen in nova-compute.log
2015-10-23 15:00:43.055 18166 ERROR oslo_messaging.rpc.dispatcher [req-87cbfae6-5ed3-4517-b843-c51daf163a51 a84d988868ac45f1a8e703cc535e878d 0375e35c77014e97a985bcbf372d3bee - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh 192.0.2.19 mkdir -p /var/lib/nova/instances/8a9b8dd5-b814-427a-b0b0-9152904dc373
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6821, in resize_instance
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher clean_shutdown=clean_shutdown)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher payload)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 335, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 306, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 385, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 294, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher migration.instance_uuid, exc_info=True)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 277, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 363, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 351, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4076, in resize_instance
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher self.instance_events.clear_events_for_instance(instance)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher self.gen.throw(type, value, traceback)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6455, in _error_out_instance_on_exception
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher raise error.inner_exception
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Command: ssh 192.0.2.19 mkdir -p /var/lib/nova/instances/8a9b8dd5-b814-427a-b0b0-9152904dc373
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'
Environment
- Red Hat OpenStack 7.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.