redis.exceptions.ConnectionError in gnocchi-metricd.

Issue

• We see a lot of errors in the gnocchi-metricd.log, like below:

2021-05-06 11:05:17,018 [125] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/gnocchi/cli/metricd.py", line 186, in _fill_sacks_to_process
    for sack in self.incoming.iter_on_sacks_to_process():
  File "/usr/lib/python3.6/site-packages/gnocchi/incoming/redis.py", line 186, in iter_on_sacks_to_process
    for message in p.listen():
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3123, in listen
    response = self.handle_message(self.parse_response(block=True))
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/lib/python3.6/site-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)
2021-05-06 11:05:17,018 [99] ERROR    gnocchi.cli.metricd: Error while listening for new measures notification, retrying
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

• We see the redis containers running for a long time, so this cannot be associated with a redis restart:

[root@overcloud-ctl01 ~]# podman ps |grep redis
74e39ed51e22  atl1-satellite.mgmt.geix.cloud.ge.com:5000/geix-atl1_production-osp16_containers-osp16_1_containers-redis:16.1.4-10                       /bin/bash /usr/lo...  40 hours ago  Up 40 hours ago         redis-bundle-podman-0
252034352d3b  atl1-satellite.mgmt.geix.cloud.ge.com:5000/geix-atl1_production-osp16_containers-osp16_1_containers-redis:16.1.4-10                       kolla_start           3 weeks ago   Up 40 hours ago         redis_tls_proxy

-We have found closed bug RHBZ #1791243 for that, with no extra explanation.