gnocchi-statsd service does not start properly

Solution In Progress - Updated -

Issue

  • The gnocchi-statsd service is not starting properly after we rebooted one of the controllers.

  • The service's container keeps restarting intermittently and the following logs are shown in /var/log/containers/gnocchi/gnocchi-statsd.log:

2019-09-12 13:59:45,632 [1] CRITICAL root: Traceback (most recent call last):
  File "/usr/bin/gnocchi-statsd", line 10, in 
    sys.exit(statsd())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 106, in statsd
    statsd_service.start()
  File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 171, in start
    stats = Stats(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 38, in __init__
    self.storage = storage.get_driver(self.conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 163, in get_driver
    conf.storage, incoming, coord)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 47, in __init__
    super(CephStorage, self).__init__(conf, incoming, coord)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 72, in __init__
    utils.get_coordinator_and_start(conf.coordination_url))
  File "/usr/lib/python2.7/site-packages/gnocchi/utils.py", line 76, in get_coordinator_and_start
    coord.start(start_heart=True)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 687, in start
    super(CoordinationDriverWithExecutor, self).start(start_heart)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 423, in start
    self._start()
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 447, in _start
    self._server_info = self._client.info()
  File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 51, in _translate_failures
    cause=e)
  File "/usr/lib/python2.7/site-packages/tooz/utils.py", line 225, in raise_with_cause
    excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
    six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
  File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
    raise value
ToozConnectionError: Timeout reading from socket
  • /var/log/containers/gnocchi/gnocchi-metricd.log show errors similar to this one:
2019-09-13 12:44:45,260 [141555] ERROR    futurist.periodics: Failed to call periodic 'gnocchi.cli.run_watchers' (it runs every 30.00 seconds)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 290, in run
    work()
  File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 64, in __call__
    return self.callback(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 178, in decorator
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 215, in run_watchers
    self.coord.run_watchers()
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 747, in run_watchers
    result = super(RedisDriver, self).run_watchers(timeout=timeout)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in run_watchers
    MemberLeftGroup(group_id, member_id)))
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 120, in run
    return list(map(lambda cb: cb(*args, **kwargs), self))
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 120, in <lambda>
    return list(map(lambda cb: cb(*args, **kwargs), self))
  File "/usr/lib/python2.7/site-packages/tooz/partitioner.py", line 50, in _on_member_leave
    self.ring.remove_node(event.member_id)
  File "/usr/lib/python2.7/site-packages/tooz/hashring.py", line 92, in remove_node
    raise UnknownNode(node)
UnknownNode: Unknown node `84d0c43f-aa67-4667-a317-7e2392be4d92'

Environment

  • Red Hat OpenStack Platform 12.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content