cinder volume attachment fails with "ProcessExecutionError: Unexpected error while running command." in nova-compute.log

Issue

• cinder volumes are not getting attached in nova instance
• compute node has below error in nova-compute.log

2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4685, in attach_volume
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4683, in do_attach_volume
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     bdm.destroy()
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4680, in do_attach_volume
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4703, in _attach_volume
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     self.volume_api.unreserve_volume(context, bdm.volume_id)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4695, in _attach_volume
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     do_check_attach=False, do_driver_attach=True)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 49, in wrapped
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 256, in attach
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     connector = virt_driver.get_volume_connector(instance)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1063, in get_volume_connector
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     host=CONF.host)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connector.py", line 218, in get_connector_properties
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     execute=execute))
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/base.py", line 57, in get_connector_properties
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     execute=kwargs.get('execute')))
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 121, in is_multipath_running
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     run_as_root=True, root_helper=root_helper)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 57, in execute
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     return execute_root(*cmd, **kwargs)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 204, in _wrap
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     return self.channel.remote_call(name, args, kwargs)
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 187, in remote_call
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server     raise exc_type(*result[2])
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server ProcessExecutionError: Unexpected error while running command.
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server Command: multipathd show status
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server Exit code: 1
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server Stdout: u''
2018-11-02 19:26:27.379 45604 ERROR oslo_messaging.rpc.server Stderr: u'ux_socket_connect: Connection refused\n'