Cinder volume stay in `creating` or `deleting` status and `TimeoutError: QueuePool limit` in cinder logs
Issue
When OpenStack is under load, some cinder volumes stay indefinitely in a creating or deleting state.
[user@controller02 keystone]$ cinder list --all | grep <volume-uuid>
| <volume-uuid> | <tenant-uuid> | creating | - | volume-name | 100 | - | false | False | |
Cinder volume log contains the following error message
2016-09-06 12:07:59.526 155017 WARNING cinder.volume.manager [req-df8afe19-967b-474b-a3fb-6f5313ec3a7b d668c75d6d364dd1b7faa29d26cf0e6f <tenant-uuid> - - -] Task 'cinder.volume.flows.ma
nager.create_volume.ExtractVolumeRefTask;volume:create' (6155357d-5001-454b-934a-aa8dbf8c900e) transitioned into state 'FAILURE' from state 'RUNNING'
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager Traceback (most recent call last):
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager result = task.execute(**arguments)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 209, in execute
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager volume_ref = self.db.volume_get(context, volume_id)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 205, in volume_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager return IMPL.volume_get(context, volume_id)
...
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager fairy = _ConnectionRecord.checkout(pool)
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager rec = pool._do_get()
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1053, in _do_get
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager (self.size(), self.overflow(), self._timeout))
2016-09-06 12:07:59.526 155017 ERROR cinder.volume.manager TimeoutError: QueuePool limit of size 5 overflow 15 reached, connection timed out, timeout 30
Environment
Red Hat OpenStack Platform 8.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
