gnocchi fails with "ToozError: Cannot extend an unlocked lock" in the log

Solution Verified - Updated -

Issue

gnochi logs files are full of the following errors :

2017-06-28 13:48:35.684 1002572 WARNING tooz.drivers.redis [-] Unable to heartbeat lock '<tooz.drivers.redis.RedisLock object at 0x7f196c1cf3d0>'
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis Traceback (most recent call last):
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 506, in heartbeat
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     lock.heartbeat()
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 102, in heartbeat
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     self._lock.extend(self._lock.timeout)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     self.gen.throw(type, value, traceback)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib/python2.7/site-packages/tooz/drivers/redis.py", line 54, in _translate_failures
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     cause=e)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib/python2.7/site-packages/tooz/coordination.py", line 763, in raise_with_cause
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     excutils.raise_with_cause(exc_cls, message, *args, **kwargs)
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 143, in raise_with_cause
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     six.raise_from(exc_cls(message, *args, **kwargs), kwargs.get('cause'))
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis   File "/usr/lib/python2.7/site-packages/six.py", line 692, in raise_from
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis     raise value
2017-06-28 13:48:35.684 1002572 ERROR tooz.drivers.redis ToozError: Cannot extend an unlocked lock

The impact is that gnocchi meters are not proceed. For example :

$ gnocchi status
+-----------------------------------------------------+--------+
| Field                                               | Value  |
+-----------------------------------------------------+--------+
| storage/number of metric having measures to process | 675    |
| storage/total number of measures to process         | 267470 |

Restarting gnocchi services, temporary solve this issue.

Environment

  • Red Hat OpenStack Platform 10.0
  • python-tooz < 1.43.1-1.el7ost

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content