Overcloud deploy failing with new role for different configuration hardware than existing hardware

Issue

• We couple nodes which have different hardware configuration and we have created different role for them. While attempting overcloud deployment it fails with following message:

2020-01-17 00:06:00Z [overcloud-ComputeNP-sqmplqdk2aed-0-634ubcy7kyum.NovaComputeNP]: CREATE_IN_PROGRESS  state changed
2020-01-17 00:28:31Z [overcloud-ComputeNP-sqmplqdk2aed-0-634ubcy7kyum.NovaComputeNP]: CREATE_FAILED  ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 90df9bb2-bba0-40de-a48d-f2d960d56947., Code: 500"
2020-01-17 00:28:31Z [overcloud-ComputeNP-sqmplqdk2aed-0-634ubcy7kyum]: UPDATE_FAILED  Resource CREATE failed: ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts availabl2020-01-17 00:28:35.351 4156 INFO osc_lib.shell [  admin] START with options: ['stack', 'failures', 'list', 'overcloud']
2020-01-17 00:28:35.367 4156 INFO osc_lib.shell [  admin] command: stack failures list -> heatclient.osc.v1.stack_failures.ListStackFailures (auth=True)
2020-01-17 00:28:35.369 4156 INFO osc_lib.clientmanager [  admin] Using auth plugin: password
2020-01-17 00:28:41.309 4156 INFO osc_lib.shell [  admin] END return value: 0
2020-01-17 00:28:41.346 4156 ERROR openstack [  admin] Heat Stack update failed.
2020-01-17 00:28:41.348 4156 INFO osc_lib.shell [  admin] END return value: 1
e for retrying build failures for instance aed42aaf-5e07-43a7-ad5a-c359f8d1abce., Code:
2020-01-17 00:28:31Z [overcloud-ComputeNP-sqmplqdk2aed.0]: UPDATE_FAILED  resources[0]: Resource CREATE failed: ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aed42aaf-5e07-43a7-ad5a-c359f8
2020-01-17 00:28:31Z [overcloud-ComputeNP-sqmplqdk2aed]: UPDATE_FAILED  Resource UPDATE failed: resources[0]: Resource CREATE failed: ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aed42a
2020-01-17 00:28:32Z [ComputeNP]: UPDATE_FAILED  resources[0]: resources.ComputeNP.Resource CREATE failed: ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aed42aaf-5
2020-01-17 00:28:32Z [overcloud]: UPDATE_FAILED  Resource UPDATE failed: resources[0]: resources.ComputeNP.Resource CREATE failed: ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures

 Stack overcloud UPDATE_FAILED

overcloud.ComputeNP.0.NovaComputeNP:
  resource_type: OS::TripleO::ComputeNPServer
  physical_resource_id: 90df9bb2-bba0-40de-a48d-f2d960d56947
  status: CREATE_FAILED
  status_reason: |
    ResourceInError: resources.NovaComputeNP: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 90df9bb2-bba0-40de-a48d-f2d960d56947., Code: 500"

• The following logs are also seen in /var/log/ironic/ironic-conductor.log:

2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor [req-2a747bca-6b78-472c-909f-f274d7983b56 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Failed to deploy instance: Failed to start the iSCSI target to deploy the node df140859-dadb-47ae-81e4-e33c63e61ef8. Error: {u'message': u'Error starting iSCSI target: Error starting iSCSI target: [Errno 2] No such file or directory. Failed with exit code 2. stdout: None. stderr: None', u'code': 500, u'type': u'ISCSICommandError', u'details': u'Error starting iSCSI target: [Errno 2] No such file or directory. Failed with exit code 2. stdout: None. stderr: None'} for node df140859-dadb-47ae-81e4-e33c63e61ef8: InstanceDeployFailure: Failed to deploy instance: Failed to start the iSCSI target to deploy the node df140859-dadb-47ae-81e4-e33c63e61ef8. Error: {u'message': u'Error starting iSCSI target: Error starting iSCSI target: [Errno 2] No such file or directory. Failed with exit code 2. stdout: None. stderr: None', u'code': 500, u'type': u'ISCSICommandError', u'details': u'Error starting iSCSI target: [Errno 2] No such file or directory. Failed with exit code 2. stdout: None. stderr: None'}
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 316, in heartbeat
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task)
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 408, in continue_deploy
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor     uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 328, in do_agent_iscsi_deploy
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor     raise exception.InstanceDeployFailure(reason=msg)
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Failed to deploy instance: Failed to start the iSCSI target to deploy the node df140859-dadb-47ae-81e4-e33c63e61ef8. Error: {u'message': u'Error starting iSCSI target: Error starting iSCSI target: [Errno 2] No such file or directory. Failed with exit code 2. stdout: None. stderr: None', u'code': 500, u'type': u'ISCSICommandError', u'details': u'Error starting iSCSI target: 
[Errno 2] No such file or directory. Failed with exit code 2. stdout: None. stderr: None'}
2020-01-20 21:33:27.216 64063 ERROR ironic.drivers.modules.agent_base_vendor