gnocchi-statsd service does not start properly

Issue

• The gnocchi-statsd service is not starting properly after we rebooted one of the controllers.

• The service's container keeps restarting intermittently and the following logs are shown in /var/log/containers/gnocchi/gnocchi-statsd.log:

2019-09-12 13:59:45,632 [1] CRITICAL root: Traceback (most recent call last):
  File "/usr/bin/gnocchi-statsd", line 10, in 
    sys.exit(statsd())
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 106, in statsd
    statsd_service.start()
  File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 171, in start
    stats = Stats(conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 38, in __init__
    self.storage = storage.get_driver(self.conf)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 163, in get_driver
    conf.storage, incoming, coord)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/ceph.py", line 47, in __init__
    super(CephStorage, self).__init__(conf, incoming, coord)
  File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 72, in __init__
    utils.get_coordinator_and_start(conf.coordination_url))
  File "/usr/lib/python2.7/site-packages/gnocchi/utils.py", line 76, in get_coordinator_and_start
    coord.start(start_heart=True)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 687, in start
    super(CoordinationDriverWithExecutor, self).start(start_heart)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 423, in start
    self._start()
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 447, in _start
    self._server_info = self._client.info()
  File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 51, in _translate_failures
    cause=e)
  File "/usr/lib/python2.7/site-packages/tooz/utils.py", line 225, in raise_with_cause
    excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
    six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
  File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
    raise value
ToozConnectionError: Timeout reading from socket

• /var/log/containers/gnocchi/gnocchi-metricd.log show errors similar to this one:

2019-09-13 12:44:45,260 [141555] ERROR    futurist.periodics: Failed to call periodic 'gnocchi.cli.run_watchers' (it runs every 30.00 seconds)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 290, in run
    work()
  File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 64, in __call__
    return self.callback(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/futurist/periodics.py", line 178, in decorator
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 215, in run_watchers
    self.coord.run_watchers()
  File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 747, in run_watchers
    result = super(RedisDriver, self).run_watchers(timeout=timeout)
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in run_watchers
    MemberLeftGroup(group_id, member_id)))
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 120, in run
    return list(map(lambda cb: cb(*args, **kwargs), self))
  File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 120, in <lambda>
    return list(map(lambda cb: cb(*args, **kwargs), self))
  File "/usr/lib/python2.7/site-packages/tooz/partitioner.py", line 50, in _on_member_leave
    self.ring.remove_node(event.member_id)
  File "/usr/lib/python2.7/site-packages/tooz/hashring.py", line 92, in remove_node
    raise UnknownNode(node)
UnknownNode: Unknown node `84d0c43f-aa67-4667-a317-7e2392be4d92'