cinder volume is down for Hitachi backend with error "Bad or unexpected response from the storage volume backend API: Failed to communicate with the REST API server."

Issue

• cinder volume is down for Hitachi backend with error "Bad or unexpected response from the storage volume backend API: Failed to communicate with the REST API server."

2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager [req-d3e61769-b85b-4bb6-b941-578bb5a939f6 - - - - -] Failed to initialize driver.: HBSDError: Bad or unexpected response from the storage volume backend API: Failed to communicate with the REST API server. (exception: <class 'requests.exceptions.ConnectionError'>, message: HTTPSConnectionPool(host='10.32.6.11', port=23451): Max retries exceeded with url: /ConfigurationManager/configuration/version (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f80bf2ddb50>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',)), method: GET, url: https://10.32.6.11:23451/ConfigurationManager/configuration/version, params: None, body: None)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager Traceback (most recent call last):
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 456, in init_host
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     self.driver.do_setup(ctxt)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 69, in wrap
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ret = func(*args, **kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 104, in wrap
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     return inner(self, *args, **kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 87, in inner
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ret = func(*_args, **_kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_fc.py", line 155, in do_setup
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     self.common.do_setup(context)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_rest.py", line 347, in do_setup
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     super(HBSDREST, self).do_setup(context)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_common.py", line 801, in do_setup
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     _do_setup(self, context)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 52, in inner
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ret = func(*args, **kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_common.py", line 794, in _do_setup
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     self.setup_client()
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 52, in inner
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ret = func(*args, **kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_simple_rest.py", line 65, in setup_client
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     super(HBSDSimpleREST, self).setup_client()
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_rest.py", line 369, in setup_client
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     self.client.check_api_version()
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 52, in inner
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ret = func(*args, **kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_rest_api.py", line 444, in check_api_version
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     version = self.get_api_version()
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_traceutils.py", line 52, in inner
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ret = func(*args, **kwargs)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_rest_api.py", line 439, in get_api_version
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     ver = self._get_object(url, auth=None, no_relogin=True)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_rest_api.py", line 407, in _get_object
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     rsp = self._request("GET", url, params=params, **kwargs)[0]
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hitachi/hbsd/hbsd_rest_api.py", line 300, in _request
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager     raise exception.HBSDError(data=msg)
2019-08-14 09:58:38.776 73 ERROR cinder.volume.manager HBSDError: Bad or unexpected response from the storage volume backend API: Failed to communicate with the REST API server. (exception: <class 'requests.exceptions.ConnectionError'>, message: HTTPSConnectionPool(host='10.32.6.11', port=23451): Max retries exceeded with url: /ConfigurationManager/configuration/version (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f80bf2ddb50>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',)), method: GET, url: https://10.32.6.11:23451/ConfigurationManager/configuration/version, params: None, body: None)

• cinder services shows down :

+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                    | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | overcloud-controller-0  | nova | enabled | up    | 2019-08-14T11:12:16.000000 | -               |
| cinder-scheduler | overcloud-controller-0  | nova | enabled | up    | 2019-08-14T11:12:24.000000 | -               |
| cinder-scheduler | overcloud-controller-1  | nova | enabled | up    | 2019-08-14T11:12:17.000000 | -               |
| cinder-scheduler | overcloud-controller-2  | nova | enabled | up    | 2019-08-14T11:12:23.000000 | -               |
| cinder-volume    | hostgroup@VSPGxx        | nova | enabled | down  | 2019-08-14T09:58:38.000000 | -               |
| cinder-volume    | hostgroup@tripleo_iscsi | nova | enabled | up    | 2019-08-14T11:12:25.000000 | -               |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+