Failures during Overcloud deploy due to "Error: Unable to stat device" or "Error: Unexpected error while running command."

Issue

• After a successful director deploy, we are seeing "No Valid Host" during attempts to deploy the Overcloud.

• Director appears to be working as expected, introspection works as expected, and then deploy fails with "No Valid Host". Repeatedly occurs during deploys.

• The following errors are seen in /var/log/ironic/ironic-conductor.log:

2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor [req-26373bab-fa93-41bf-a65d-b82b1c49c657 - - - - -] Asynchronous exception: Node failed to deploy. Exception: Deploy failed for instance e6f47fcc-00d2-481c-a100-
0cc39feca27d. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf parted -a optimal -s /dev/disk/by-path/ip-10.10.10.10:3260-iscsi-iqn.2008-10.org.openstack:3708950f-ad33-427b-9519-b229e4c676a6-lun-1 -- unit MiB mklabel msdos mkpart primary  1 2 m
kpart primary  2 40962 set 2 boot on
Exit code: 1
Stdout: u''
Stderr: u'Warning: The resulting partition is not properly aligned for best performance.\nError: Partition(s) 1 on /dev/sdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use
.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\n' for node 3708950f-ad33-427b-9519-b229e4c676a6: InstanceDeployFailure: Deploy failed for instance e6f47fcc-00d2-481c-a100-0cc
39feca27d. Error: Unexpected error while running command.
Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf parted -a optimal -s /dev/disk/by-path/ip-10.10.10.10:3260-iscsi-iqn.2008-10.org.openstack:3708950f-ad33-427b-9519-b229e4c676a6-lun-1 -- unit MiB mklabel msdos mkpart primary  1 2 m
kpart primary  2 40962 set 2 boot on
Exit code: 1
Stdout: u''
Stderr: u'Warning: The resulting partition is not properly aligned for best performance.\nError: Partition(s) 1 on /dev/sdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use
.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\n'
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 316, in heartbeat
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 408, in continue_deploy
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 333, in do_agent_iscsi_deploy
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 269, in continue_deploy
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     _fail_deploy(task, e)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 240, in _fail_deploy
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor     raise exception.InstanceDeployFailure(msg)
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Deploy failed for instance e6f47fcc-00d2-481c-a100-0cc39feca27d. Error: Unexpected error while running command.
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor Command: sudo ironic-rootwrap /etc/ironic/rootwrap.conf parted -a optimal -s /dev/disk/by-path/ip-10.10.10.10:3260-iscsi-iqn.2008-10.org.openstack:3708950f-ad33-
427b-9519-b229e4c676a6-lun-1 -- unit MiB mklabel msdos mkpart primary  1 2 mkpart primary  2 40962 set 2 boot on
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor Exit code: 1
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor Stdout: u''
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor Stderr: u'Warning: The resulting partition is not properly aligned for best performance.\nError: Partition(s) 1 on /dev/sdb have been written, but we have been un
able to inform the kernel of the change, probably because it/they are in use.  As a result, the old partition(s) will remain in use.  You should reboot now before making further changes.\n'
2020-09-22 11:12:11.588 1639 ERROR ironic.drivers.modules.agent_base_vendor 

and:

2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor [req-93eb9793-bd4d-4f8a-a9f0-71711ca7bfce - - - - -] Asynchronous exception: Node failed to deploy. Exception: Deploy failed for instance 29451979-d0d1-41ca-ba22-
47d51367863b. Error: Unable to stat device /dev/disk/by-path/ip-10.10.10.10:3260-iscsi-iqn.2008-10.org.openstack:9702b1e0-473c-43c3-9f75-9cc23ffef84a-lun-1-part2 after attempting to verify 3 times. for node 9702b1e0-473c-43c3-9f75-9cc23f
fef84a: InstanceDeployFailure: Deploy failed for instance 29451979-d0d1-41ca-ba22-47d51367863b. Error: Unable to stat device /dev/disk/by-path/ip-10.10.10.10:3260-iscsi-iqn.2008-10.org.openstack:9702b1e0-473c-43c3-9f75-9cc23ffef84a-lun-1
-part2 after attempting to verify 3 times.
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 316, in heartbeat
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     self.continue_deploy(task)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     return f(*args, **kwargs)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 408, in continue_deploy
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 333, in do_agent_iscsi_deploy
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     uuid_dict_returned = continue_deploy(task, iqn=iqn, address=address)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 269, in continue_deploy
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     _fail_deploy(task, e)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python2.7/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 240, in _fail_deploy
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor     raise exception.InstanceDeployFailure(msg)
2020-09-22 11:09:55.096 1639 ERROR ironic.drivers.modules.agent_base_vendor InstanceDeployFailure: Deploy failed for instance 29451979-d0d1-41ca-ba22-47d51367863b. Error: Unable to stat device /dev/disk/by-path/ip-10.10.10.10:3260-iscsi-
iqn.2008-10.org.openstack:9702b1e0-473c-43c3-9f75-9cc23ffef84a-lun-1-part2 after attempting to verify 3 times.