Cinder Service Unavailable (HTTP 503) LockCreationFailed: Coordinator uninitialized

Issue

• All Cinder volume creation/deletion started failing with error Service Unavailable (HTTP 503).

• We also found the following error message:

2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server [req-fe125062-19cd-48b1-bfea-74931118fb53 31613c8dcf014fb8be7c047dfc9d73c2 7bcafc9722684dad950897d1a7ff469a - default default] Exception during message handling: cinder.exception.LockCreationFailed: Coordinator uninitialized.
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server   File "<decorator-gen-236>", line 2, in delete_volume
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/coordination.py", line 140, in _synchronized
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server     lock = coordinator.get_lock(lock_name.format(**call_args))
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/coordination.py", line 88, in get_lock
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server     raise exception.LockCreationFailed(_('Coordinator uninitialized.'))
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server cinder.exception.LockCreationFailed: Coordinator uninitialized.
2024-02-21 16:05:18.128 75 ERROR oslo_messaging.rpc.server 

• Issue was resolved after running pcs resource restart openstack-cinder-volume and waiting for about 15min.