Cinder volume stay in `creating` or `deleting` status and `TimeoutError: QueuePool limit` in cinder logs
Issue
When OpenStack is under load, some cinder volumes stay indefinitely in a creating
or deleting
state.
[user@controller02 keystone]$ cinder list --all | grep <volume-uuid>
| <volume-uuid> | <tenant-uuid> | creating | - | volume-name | 100 | - | false | False | |
Cinder volume log contains the following error message
2016-09-06 12:07:59.526 155017 WARNING cinder.volume.manager [req-df8afe19-967b-474b-a3fb-6f5313ec3a7b d668c75d6d364dd1b7faa29d26cf0e6f <tenant-uuid> - - -] Task 'cinder.volume.flows.ma
nager.create_volume.ExtractVolumeRefTask;volume:create' (6155357d-5001-454b-934a-aa8dbf8c900e) transitioned into state 'FAILURE' from state 'RUNNING'
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager Traceback (most recent call last):
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager result = task.execute(**arguments)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 209, in execute
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager volume_ref = self.db.volume_get(context, volume_id)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 205, in volume_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager return IMPL.volume_get(context, volume_id)
...
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager fairy = _ConnectionRecord.checkout(pool)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager rec = pool._do_get()
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1053, in _do_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager (self.size(), self.overflow(), self._timeout))
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager TimeoutError: QueuePool limit of size 5 overflow 15 reached, connection timed out, timeout 30
Environment
Red Hat OpenStack Platform 8.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.