gnocchi fails with "ToozError: Cannot extend an unlocked lock" in the log
Issue
gnochi logs files are full of the following errors :
2017-06-28 13:48:35.684 1002572 WARNING tooz.drivers.redis [-] Unable to heartbeat lock '<tooz.drivers.redis.RedisLock object at 0x7f196c1cf3d0>'
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis Traceback (most recent call last):
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 506, in heartbeat
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis lock.heartbeat()
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 102, in heartbeat
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis self._lock.extend(self._lock.timeout)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis self.gen.throw(type, value, traceback)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 54, in _translate_failures
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis cause=e)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis raise value
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis ToozError: Cannot extend an unlocked lock
The impact is that gnocchi meters are not proceed. For example :
$ gnocchi status
+-----------------------------------------------------+--------+
| Field | Value |
+-----------------------------------------------------+--------+
| storage/number of metric having measures to process | 675 |
| storage/total number of measures to process | 267470 |
Restarting gnocchi services, temporary solve this issue.
Environment
- Red Hat OpenStack Platform 10.0
python-tooz
<1.43.1-1.el7ost
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.