Cinder volume stay in `creating` or `deleting` status and `TimeoutError: QueuePool limit` in cinder logs

Solution In Progress - Updated -

Issue

When OpenStack is under load, some cinder volumes stay indefinitely in a creating or deleting state.

[user@controller02 keystone]$ cinder list --all | grep <volume-uuid>
| <volume-uuid> | <tenant-uuid> |  creating |        -         |                 volume-name                 | 100  |      -      |  false   |    False    |                                      |

Cinder volume log contains the following error message

2016-09-06 12:07:59.526 155017 WARNING cinder.volume.manager [req-df8afe19-967b-474b-a3fb-6f5313ec3a7b d668c75d6d364dd1b7faa29d26cf0e6f <tenant-uuid> - - -] Task 'cinder.volume.flows.ma
nager.create_volume.ExtractVolumeRefTask;volume:create' (6155357d-5001-454b-934a-aa8dbf8c900e) transitioned into state 'FAILURE' from state 'RUNNING'
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager Traceback (most recent call last):
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     result = task.execute(**arguments)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 209, in execute
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     volume_ref = self.db.volume_get(context, volume_id)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 205, in volume_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     return IMPL.volume_get(context, volume_id)
...
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     fairy = _ConnectionRecord.checkout(pool)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     rec = pool._do_get()
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1053, in _do_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     (self.size(), self.overflow(), self._timeout))
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager TimeoutError: QueuePool limit of size 5 overflow 15 reached, connection timed out, timeout 30

Environment

Red Hat OpenStack Platform 8.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In