gnocchi fails with "ToozError: Cannot extend an unlocked lock" in the log
Issue
gnochi logs files are full of the following errors :
2017-06-28 13:48:35.684 1002572 WARNING tooz.drivers.redis [-] Unable to heartbeat lock '<tooz.drivers.redis.RedisLock object at 0x7f196c1cf3d0>'
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis Traceback (most recent call last):
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 506, in heartbeat
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis lock.heartbeat()
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 102, in heartbeat
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis self._lock.extend(self._lock.timeout)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis self.gen.throw(type, value, traceback)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 54, in _translate_failures
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis cause=e)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis raise value
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis ToozError: Cannot extend an unlocked lock
The impact is that gnocchi meters are not proceed. For example :
$ gnocchi status
+-----------------------------------------------------+--------+
| Field | Value |
+-----------------------------------------------------+--------+
| storage/number of metric having measures to process | 675 |
| storage/total number of measures to process | 267470 |
Restarting gnocchi services, temporary solve this issue.
Environment
- Red Hat OpenStack Platform 10.0
python-tooz<1.43.1-1.el7ost
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
