Openstack minor update is failing when updating the gnocchi container

Solution Unverified - Updated -

Issue

  • openstack overcloud update run --nodes Controller is failing with:
        "2019-11-18 11:07:35,063 [22] CRITICAL root: Traceback (most recent call last):",
        "  File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>",
        "    sys.exit(upgrade())",
        "  File \"/usr/lib/python2.7/site-packages/gnocchi/cli/manage.py\", line 73, in upgrade",
        "    i.upgrade(conf.sacks_number)",
        "  File \"/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py\", line 71, in upgrade",
        "    self.set_storage_settings(num_sacks)",
        "  File \"/usr/lib/python2.7/site-packages/gnocchi/incoming/redis.py\", line 38, in set_storage_settings",
        "    self._client.hset(self.CFG_PREFIX, self.CFG_SACKS, num_sacks)",
        "  File \"/usr/lib/python2.7/site-packages/redis/client.py\", line 1853, in hset",
        "    return self.execute_command('HSET', name, key, value)",
        "  File \"/usr/lib/python2.7/site-packages/redis/client.py\", line 570, in execute_command",
        "    connection.send_command(*args)",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 556, in send_command",
        "    self.send_packed_command(self.pack_command(*args))",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 532, in send_packed_command",
        "    self.connect()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 440, in connect",
        "    self.on_connect()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 508, in on_connect",
        "    if nativestr(self.read_response()) != 'OK':",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 569, in read_response",
        "    response = self._parser.read_response()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 224, in read_response",
        "    response = self._buffer.readline()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 162, in readline",
        "    self._read_from_socket()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 137, in _read_from_socket",
  • On all 3 controllers the metrics container logs show this repeating:
ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)
2019-11-18 12:07:16,863 [21] ERROR    gnocchi.utils: Unable to initialize coordination driver
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 45, in get_coordinator_and_start
    coord.start(start_heart=True)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 687, in start
    super(CoordinationDriverWithExecutor, self).start(start_heart)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 423, in start
    self._start()
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 449, in _start
    self._server_info = self._client.info()
  File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 51, in _translate_failures
    cause=e)
  File "/usr/lib/python2.7/site-packages/tooz/utils.py", line 225, in raise_with_cause
    excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
    six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
  File "/usr/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)

Environment

  • Red Hat OpenStack Platform 13.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In