During resize instance went into ERROR not able to bring the instance up

Solution Verified - Updated -

Issue

  • During resize instance went into ERROR not able to bring the instance up.
  • While performing resize of instance, it tried to migrate from one compute to another compute node and went into ERROR state, remained on source node because of some issue with attached cinder volumes backed storage. Tried to reset the state of instance, but not able to bring the instance up.
  • Instance went into ERROR state during the resize with following call trace on source compute node.
    2016-12-03 06:06:42.616 5710 ERROR nova.compute.manager [req-763cd501-fe02-466c-88af-f36f9d679130 a393dd7f32df4be495299a66cde0585c 56a3086cce3349dd888b89cf2bba1451 - - -] [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0] Setting instance vm_state to ERROR
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0] Traceback (most recent call last):
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6479, in _error_out_instance_on_exception
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     yield
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4072, in resize_instance
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     self._terminate_volume_connections(context, instance, bdms)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4101, in _terminate_volume_connections
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     connector)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     res = method(self, ctx, volume_id, *args, **kwargs)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 369, in terminate_connection
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     connector)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 448, in terminate_connection
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     {'connector': connector})
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 375, in _action
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     return self.api.client.post(url, body=body)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 118, in post
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     return self._cs_request(url, 'POST', **kwargs)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 112, in _cs_request
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     return self.request(url, method, **kwargs)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 105, in request
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]     raise exceptions.from_response(resp, body)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-e59de5f1-bfb2-460a-ace6-4ac2ec9674ff)
    2016-12-03 06:06:42.616 5710 TRACE nova.compute.manager [instance: 5ca9bddc-5230-4c14-8baf-f052e06195f0]

Environment

  • Red Hat OpenStack Platform 7.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In