Cinder volume creation failing
Issue
- Cinder volumes fail to create on all of our cinder mounts.
- Errors seen in cinder logs:
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup x.wait()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup return self.thread.wait()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup return self._exit_event.wait()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup return hubs.get_hub().switch()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup return self.greenlet.switch()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup result = function(*args, **kwargs)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/service.py", line 486, in run_service
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup service.start()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/service.py", line 103, in start
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup self.manager.init_host()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 308, in init_host
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup self.publish_service_capabilities(ctxt)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1160, in publish_service_capabilities
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup self._report_driver_status(context)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1149, in _report_driver_status
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup volume_stats = self.driver.get_volume_stats(refresh=True)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 342, in get_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup self._update_volume_stats()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 933, in _update_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup super(NetAppDirectCmodeNfsDriver, self)._update_volume_stats()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 222, in _update_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup super(NetAppNFSDriver, self)._update_volume_stats()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 361, in _update_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup capacity, free, used = self._get_capacity_info(share)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 565, in _get_capacity_info
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup run_as_root=True)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 136, in execute
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup return processutils.execute(*cmd, **kwargs)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py", line 179, in execute
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup cmd=sanitized_cmd)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Exit code: 1
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Stdout: u''
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Stderr: u'sudo: effective uid is not 0, is sudo installed setuid root?\n'
2016-04-23 17:22:29.416 11345 ERROR cinder.openstack.common.threadgroup [-] Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>
Exit code: 127
Stdout: u''
Stderr: u'\nRejected. That command is not permitted by the PowerBroker policy\nfor this account (Evaluated 2 records).\n\nFor assistance:\t\tpbrun -p mypolicy | more\n\nUser:\t\tcin der\nRunhost:\<ip>\nPRC Group:\tADS.cinder\nDate and Time:\t2016/04/23 17:22:29\nCommand:\tcinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/l ib/cinder/mnt/<uuid>\n\n'
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup x.wait()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup return self.thread.wait()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup return self._exit_event.wait()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup return hubs.get_hub().switch()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup return self.greenlet.switch()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup result = function(*args, **kwargs)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/service.py", line 486, in run_service
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup service.start()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/service.py", line 103, in start
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup self.manager.init_host()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 308, in init_host
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup self.publish_service_capabilities(ctxt)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1160, in publish_service_capabilities
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup self._report_driver_status(context)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1149, in _report_driver_status
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup volume_stats = self.driver.get_volume_stats(refresh=True)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 342, in get_volume_stats
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup self._update_volume_stats()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 933, in _update_volume_sta ts
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup super(NetAppDirectCmodeNfsDriver, self)._update_volume_stats()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 222, in _update_volume_sta ts
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup super(NetAppNFSDriver, self)._update_volume_stats()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 361, in _update_volume_stats
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup capacity, free, used = self._get_capacity_info(share)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 565, in _get_capacity_info
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup run_as_root=True)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 136, in execute
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup return processutils.execute(*cmd, **kwargs)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py", line 179, in execute
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup cmd=sanitized_cmd)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Exit code: 127
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Stdout: u''
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Stderr: u'\nRejected. That command is not permitted by the PowerBroker policy\nfor this account (Evaluated 2 records).\n\nFor assistance:\t\tpbrun -p mypolicy | more\n\nUser:\t\tcinder\nRunhost:\<ip>\nPRC Group:\tADS.cinder\nDate and Time:\t2016/04/23 17:22:29\nComman d:\tcinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>\n\n'
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup
Environment
- Red Hat OpenStack Platform 5
- Red Hat Enterprise Linux 7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
