Openstack minor update is failing when updating the gnocchi container
Issue
openstack overcloud update run --nodes Controller
is failing with:
"2019-11-18 11:07:35,063 [22] CRITICAL root: Traceback (most recent call last):",
" File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>",
" sys.exit(upgrade())",
" File \"/usr/lib/python2.7/site-packages/gnocchi/cli/manage.py\", line 73, in upgrade",
" i.upgrade(conf.sacks_number)",
" File \"/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py\", line 71, in upgrade",
" self.set_storage_settings(num_sacks)",
" File \"/usr/lib/python2.7/site-packages/gnocchi/incoming/redis.py\", line 38, in set_storage_settings",
" self._client.hset(self.CFG_PREFIX, self.CFG_SACKS, num_sacks)",
" File \"/usr/lib/python2.7/site-packages/redis/client.py\", line 1853, in hset",
" return self.execute_command('HSET', name, key, value)",
" File \"/usr/lib/python2.7/site-packages/redis/client.py\", line 570, in execute_command",
" connection.send_command(*args)",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 556, in send_command",
" self.send_packed_command(self.pack_command(*args))",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 532, in send_packed_command",
" self.connect()",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 440, in connect",
" self.on_connect()",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 508, in on_connect",
" if nativestr(self.read_response()) != 'OK':",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 569, in read_response",
" response = self._parser.read_response()",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 224, in read_response",
" response = self._buffer.readline()",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 162, in readline",
" self._read_from_socket()",
" File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 137, in _read_from_socket",
- On all 3 controllers the metrics container logs show this repeating:
ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)
2019-11-18 12:07:16,863 [21] ERROR gnocchi.utils: Unable to initialize coordination driver
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
result = fn(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 45, in get_coordinator_and_start
coord.start(start_heart=True)
File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 687, in start
super(CoordinationDriverWithExecutor, self).start(start_heart)
File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 423, in start
self._start()
File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 449, in _start
self._server_info = self._client.info()
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 51, in _translate_failures
cause=e)
File "/usr/lib/python2.7/site-packages/tooz/utils.py", line 225, in raise_with_cause
excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
File "/usr/lib/python2.7/site-packages/six.py", line 718, in raise_from
raise value
ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)
Environment
- Red Hat OpenStack Platform 13.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.