Openstack minor update is failing when updating the gnocchi container

Solution Unverified - Updated -

Issue

  • openstack overcloud update run --nodes Controller is failing with:
        "2019-11-18 11:07:35,063 [22] CRITICAL root: Traceback (most recent call last):",
        "  File \"/usr/bin/gnocchi-upgrade\", line 10, in <module>",
        "    sys.exit(upgrade())",
        "  File \"/usr/lib/python2.7/site-packages/gnocchi/cli/manage.py\", line 73, in upgrade",
        "    i.upgrade(conf.sacks_number)",
        "  File \"/usr/lib/python2.7/site-packages/gnocchi/incoming/__init__.py\", line 71, in upgrade",
        "    self.set_storage_settings(num_sacks)",
        "  File \"/usr/lib/python2.7/site-packages/gnocchi/incoming/redis.py\", line 38, in set_storage_settings",
        "    self._client.hset(self.CFG_PREFIX, self.CFG_SACKS, num_sacks)",
        "  File \"/usr/lib/python2.7/site-packages/redis/client.py\", line 1853, in hset",
        "    return self.execute_command('HSET', name, key, value)",
        "  File \"/usr/lib/python2.7/site-packages/redis/client.py\", line 570, in execute_command",
        "    connection.send_command(*args)",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 556, in send_command",
        "    self.send_packed_command(self.pack_command(*args))",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 532, in send_packed_command",
        "    self.connect()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 440, in connect",
        "    self.on_connect()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 508, in on_connect",
        "    if nativestr(self.read_response()) != 'OK':",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 569, in read_response",
        "    response = self._parser.read_response()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 224, in read_response",
        "    response = self._buffer.readline()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 162, in readline",
        "    self._read_from_socket()",
        "  File \"/usr/lib/python2.7/site-packages/redis/connection.py\", line 137, in _read_from_socket",
  • On all 3 controllers the metrics container logs show this repeating:
ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)
2019-11-18 12:07:16,863 [21] ERROR    gnocchi.utils: Unable to initialize coordination driver
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli/metricd.py", line 45, in get_coordinator_and_start
    coord.start(start_heart=True)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 687, in start
    super(CoordinationDriverWithExecutor, self).start(start_heart)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 423, in start
    self._start()
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 449, in _start
    self._server_info = self._client.info()
  File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 51, in _translate_failures
    cause=e)
  File "/usr/lib/python2.7/site-packages/tooz/utils.py", line 225, in raise_with_cause
    excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
    six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
  File "/usr/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
ToozConnectionError: Error while reading from socket: ('Connection closed by server.',)

Environment

  • Red Hat OpenStack Platform 13.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content