Cinder volume stay in `creating` or `deleting` status and `TimeoutError: QueuePool limit` in cinder logs

Solution In Progress - Updated -

Issue

When OpenStack is under load, some cinder volumes stay indefinitely in a creating or deleting state.

[user@controller02 keystone]$ cinder list --all | grep <volume-uuid>
| <volume-uuid> | <tenant-uuid> |  creating |        -         |                 volume-name                 | 100  |      -      |  false   |    False    |                                      |

Cinder volume log contains the following error message

2016-09-06 12:07:59.526 155017 WARNING cinder.volume.manager [req-df8afe19-967b-474b-a3fb-6f5313ec3a7b d668c75d6d364dd1b7faa29d26cf0e6f <tenant-uuid> - - -] Task 'cinder.volume.flows.ma
nager.create_volume.ExtractVolumeRefTask;volume:create' (6155357d-5001-454b-934a-aa8dbf8c900e) transitioned into state 'FAILURE' from state 'RUNNING'
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager Traceback (most recent call last):
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     result = task.execute(**arguments)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 209, in execute
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     volume_ref = self.db.volume_get(context, volume_id)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 205, in volume_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     return IMPL.volume_get(context, volume_id)
...
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     fairy = _ConnectionRecord.checkout(pool)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     rec = pool._do_get()
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1053, in _do_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager     (self.size(), self.overflow(), self._timeout))
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager TimeoutError: QueuePool limit of size 5 overflow 15 reached, connection timed out, timeout 30

Environment

Red Hat OpenStack Platform 8.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content