Permission denied issue occurs when creating an instance with creating a volume or attaching a volume to a running instance under Cinder NFS backend.

Issue

• Permission denied issue occurs when creating an instance with creating a volume or attaching a volume to a running instance under Cinder NFS backend.

  • Sample log in /var/log/nova/nova-compute.log

    INFO nova.compute.manager [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Attaching volume e230e8c4-a5d7-4370-80bc-b6557afccc4
    
    ERROR nova.virt.libvirt.driver [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Failed to attach volume at mountpoint: /dev/vdb
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Traceback (most recent call last):
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     guest.attach_device(conf, persistent=True, live=live)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = execute(f, *args, **kwargs)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(c, e, tb)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = meth(*args, **kwargs)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] libvirtError: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
    ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] 
    ERROR nova.virt.block_device [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Driver failed to attach volume e230e8c4-a5d7-4370
    
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Traceback (most recent call last):
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 279, in attach
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     device_type=self['device_type'], encryption=encryption)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1358, in attach_volume
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._disconnect_volume(connection_info, disk_dev)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self.force_reraise()
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     guest.attach_device(conf, persistent=True, live=live)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = execute(f, *args, **kwargs)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(c, e, tb)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = meth(*args, **kwargs)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] libvirtError: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
    ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] 
    ERROR nova.compute.manager [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Failed to attach e230e8c4-a5d7-4370-80bc-b6557afccc
    
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Traceback (most recent call last):
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4756, in _attach_volume
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     do_check_attach=False, do_driver_attach=True)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 54, in wrapped
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     ret_val = method(obj, context, *args, **kwargs)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 288, in attach
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     connector)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self.force_reraise()
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 279, in attach
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     device_type=self['device_type'], encryption=encryption)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1358, in attach_volume
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._disconnect_volume(connection_info, disk_dev)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self.force_reraise()
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     guest.attach_device(conf, persistent=True, live=live)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = execute(f, *args, **kwargs)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(c, e, tb)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = meth(*args, **kwargs)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] libvirtError: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
    ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] 
    ERROR oslo_messaging.rpc.dispatcher [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] Exception during message handling: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied