Recieve errors while migrating instances within the Overcloud

Solution Unverified - Updated -

Issue

  • When attempting to migrate instances errors can be seen when viewing the nova migration-list:
# nova migration-list
+---------------------------------+---------------------------------+---------------------------------+---------------------------------+------------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+
| Source Node                     | Dest Node                       | Source Compute                  | Dest Compute                    | Dest Host  | Status | Instance UUID                        | Old Flavor | New Flavor | Created At                 | Updated At                 |
+---------------------------------+---------------------------------+---------------------------------+---------------------------------+------------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+
| overcloud-compute-1.localdomain | overcloud-compute-6.localdomain | overcloud-compute-1.localdomain | overcloud-compute-6.localdomain | 192.0.2.9  | error  | <uuid> | 5          | 8          | 2015-10-23T00:30:03.000000 | 2015-10-23T00:30:04.000000 |
| overcloud-compute-6.localdomain | overcloud-compute-7.localdomain | overcloud-compute-6.localdomain | overcloud-compute-7.localdomain | 192.0.2.8  | error  | <uuid> | 5          | 5          | 2015-10-23T19:00:35.000000 | 2015-10-23T19:00:36.000000 |
| overcloud-compute-1.localdomain | overcloud-compute-0.localdomain | overcloud-compute-1.localdomain | overcloud-compute-0.localdomain | 192.0.2.19 | error  | <uuid> | 5          | 5          | 2015-10-23T19:00:42.000000 | 2015-10-23T19:00:42.000000 |
+---------------------------------+---------------------------------+---------------------------------+---------------------------------+------------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+
  • The following traceback can be seen in nova-compute.log
2015-10-23 15:00:43.055 18166 ERROR oslo_messaging.rpc.dispatcher [req-87cbfae6-5ed3-4517-b843-c51daf163a51 a84d988868ac45f1a8e703cc535e878d 0375e35c77014e97a985bcbf372d3bee - - -] Exception during message handling: Resize error: not able to execute ssh command: Unexpected error while running command.
Command: ssh 192.0.2.19 mkdir -p /var/lib/nova/instances/8a9b8dd5-b814-427a-b0b0-9152904dc373
Exit code: 255
Stdout: u''
Stderr: u'Host key verification failed.\r\n'
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6821, in resize_instance
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     clean_shutdown=clean_shutdown)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 335, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 306, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 385, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 294, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     migration.instance_uuid, exc_info=True)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 277, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 363, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 351, in decorated_function
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4076, in resize_instance
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     self.instance_events.clear_events_for_instance(instance)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     self.gen.throw(type, value, traceback)
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6455, in _error_out_instance_on_exception
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher     raise error.inner_exception
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Command: ssh 192.0.2.19 mkdir -p /var/lib/nova/instances/8a9b8dd5-b814-427a-b0b0-9152904dc373
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Exit code: 255
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Stdout: u''
2015-10-23 15:00:43.055 18166 TRACE oslo_messaging.rpc.dispatcher Stderr: u'Host key verification failed.\r\n'

Environment

  • Red Hat OpenStack 7.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content