Cinder volume creation failing

Issue

• Cinder volumes fail to create on all of our cinder mounts.
• Errors seen in cinder logs:

2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     x.wait()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     return self.thread.wait()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     return self._exit_event.wait()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     return hubs.get_hub().switch()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     return self.greenlet.switch()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     result = function(*args, **kwargs)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/service.py", line 486, in run_service
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     service.start()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 103, in start
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     self.manager.init_host()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 308, in init_host
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     self.publish_service_capabilities(ctxt)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1160, in publish_service_capabilities
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     self._report_driver_status(context)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1149, in _report_driver_status
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     volume_stats = self.driver.get_volume_stats(refresh=True)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 342, in get_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     self._update_volume_stats()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 933, in _update_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     super(NetAppDirectCmodeNfsDriver, self)._update_volume_stats()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 222, in _update_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     super(NetAppNFSDriver, self)._update_volume_stats()
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 361, in _update_volume_stats
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     capacity, free, used = self._get_capacity_info(share)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 565, in _get_capacity_info
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     run_as_root=True)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 136, in execute
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     return processutils.execute(*cmd, **kwargs)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py", line 179, in execute
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup     cmd=sanitized_cmd)
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Exit code: 1
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Stdout: u''
2016-04-23 00:34:35.816 26792 TRACE cinder.openstack.common.threadgroup Stderr: u'sudo: effective uid is not 0, is sudo installed setuid root?\n'

2016-04-23 17:22:29.416 11345 ERROR cinder.openstack.common.threadgroup [-] Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>
Exit code: 127
Stdout: u''
Stderr: u'\nRejected.  That command is not permitted by the PowerBroker policy\nfor this account (Evaluated 2 records).\n\nFor assistance:\t\tpbrun -p mypolicy | more\n\nUser:\t\tcin    der\nRunhost:\<ip>\nPRC Group:\tADS.cinder\nDate and Time:\t2016/04/23 17:22:29\nCommand:\tcinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/l    ib/cinder/mnt/<uuid>\n\n'
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     x.wait()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     return self.thread.wait()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     return self._exit_event.wait()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     return hubs.get_hub().switch()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     return self.greenlet.switch()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     result = function(*args, **kwargs)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/service.py", line 486, in run_service
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     service.start()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 103, in start
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     self.manager.init_host()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 308, in init_host
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     self.publish_service_capabilities(ctxt)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1160, in publish_service_capabilities
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     self._report_driver_status(context)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1149, in _report_driver_status
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     volume_stats = self.driver.get_volume_stats(refresh=True)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 342, in get_volume_stats
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     self._update_volume_stats()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 933, in _update_volume_sta    ts
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     super(NetAppDirectCmodeNfsDriver, self)._update_volume_stats()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/nfs.py", line 222, in _update_volume_sta    ts
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     super(NetAppNFSDriver, self)._update_volume_stats()
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 361, in _update_volume_stats
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     capacity, free, used = self._get_capacity_info(share)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line 565, in _get_capacity_info
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     run_as_root=True)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 136, in execute
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     return processutils.execute(*cmd, **kwargs)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/cinder/openstack/common/processutils.py", line 179, in execute
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup     cmd=sanitized_cmd)
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup ProcessExecutionError: Unexpected error while running command.
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Exit code: 127
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Stdout: u''
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup Stderr: u'\nRejected.  That command is not permitted by the PowerBroker policy\nfor this account (Evaluated 2     records).\n\nFor assistance:\t\tpbrun -p mypolicy | more\n\nUser:\t\tcinder\nRunhost:\<ip>\nPRC Group:\tADS.cinder\nDate and Time:\t2016/04/23 17:22:29\nComman    d:\tcinder-rootwrap /etc/cinder/rootwrap.conf stat -f -c %S %b %a /var/lib/cinder/mnt/<uuid>\n\n'
2016-04-23 17:22:29.416 11345 TRACE cinder.openstack.common.threadgroup