Intropection is not starting/finishing correctly and logs say node is locked

Solution Verified - Updated -

Issue

The introspection is not starting/finishing correctly and the follwoing logs can seen in mistral-executor.logs:

017-05-02 08:14:45.990 1464 ERROR mistral.engine.default_executor [-] Failed to run action [action_ex_id=8fff1f31-45bd-4c82-bf42-60c0a4f80e2f, action_cls='<class 'mistral.actions.action_factory.IronicAction'>', attributes='{u'client_method_name': u'node.set_provision_state'}', params='{u'state': u'provide', u'node_uuid': u'ea73efd1-a4db-46f8-a843-469ac52ef55e', u'configdrive': None, u'cleansteps': None}']
 IronicAction.node.set_provision_state failed: <class 'ironicclient.common.apiclient.exceptions.Conflict'>: Node ea73efd1-a4db-46f8-a843-469ac52ef55e is locked by host nManager.testdev.env, please retry after the current operation is completed.

2017-05-02 08:14:45.836 2514 DEBUG wsme.api [req-764a9d87-2360-4475-a93a-d68c99c02838 admin admin - - -] Client-side error: Node 07cfae86-e8b3-4dc8-bf92-22ac77b509da is locked by host nManager.testdev.env, please retry after the current operation is completed.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ironic/conductor/manager.py", line 1032, in do_provisioning_action
    % action) as task:
  File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 158, in acquire
    driver_name=driver_name, purpose=purpose)
  File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 223, in __init__
    self.release_resources()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 210, in __init__
    self._lock()
  File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 255, in _lock
    reserve_node()
  File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python2.7/site-packages/retrying.py", line 229, in call
    raise attempt.get()
  File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 248, in reserve_node
    self.node_id)
  File "/usr/lib/python2.7/site-packages/ironic/objects/node.py", line 270, in reserve
    db_node = cls.dbapi.reserve_node(tag, node_id)
  File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 264, in reserve_node
    host=node['reservation'])
NodeLocked: Node 07cfae86-e8b3-4dc8-bf92-22ac77b509da is locked by host nManager.testdev.env, please retry after the current operation is completed.

Environment

  • Red Hat Openstack Platform 10
  • ssh_pxe ironic driver

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content