Permission denied issue occurs when creating an instance with creating a volume or attaching a volume to a running instance under Cinder NFS backend.

Solution Verified - Updated -

Issue

  • Permission denied issue occurs when creating an instance with creating a volume or attaching a volume to a running instance under Cinder NFS backend.

    • Sample log in /var/log/nova/nova-compute.log

      INFO nova.compute.manager [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Attaching volume e230e8c4-a5d7-4370-80bc-b6557afccc4
      
      ERROR nova.virt.libvirt.driver [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Failed to attach volume at mountpoint: /dev/vdb
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Traceback (most recent call last):
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     guest.attach_device(conf, persistent=True, live=live)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     result = proxy_call(self._autowrap, f, *args, **kwargs)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = execute(f, *args, **kwargs)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(c, e, tb)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = meth(*args, **kwargs)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] libvirtError: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
      ERROR nova.virt.libvirt.driver [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] 
      ERROR nova.virt.block_device [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Driver failed to attach volume e230e8c4-a5d7-4370
      
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Traceback (most recent call last):
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 279, in attach
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     device_type=self['device_type'], encryption=encryption)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1358, in attach_volume
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._disconnect_volume(connection_info, disk_dev)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self.force_reraise()
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(self.type_, self.value, self.tb)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     guest.attach_device(conf, persistent=True, live=live)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     result = proxy_call(self._autowrap, f, *args, **kwargs)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = execute(f, *args, **kwargs)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(c, e, tb)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = meth(*args, **kwargs)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] libvirtError: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
      ERROR nova.virt.block_device [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] 
      ERROR nova.compute.manager [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Failed to attach e230e8c4-a5d7-4370-80bc-b6557afccc
      
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] Traceback (most recent call last):
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4756, in _attach_volume
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     do_check_attach=False, do_driver_attach=True)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 54, in wrapped
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     ret_val = method(obj, context, *args, **kwargs)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 288, in attach
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     connector)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self.force_reraise()
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(self.type_, self.value, self.tb)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 279, in attach
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     device_type=self['device_type'], encryption=encryption)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1358, in attach_volume
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._disconnect_volume(connection_info, disk_dev)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self.force_reraise()
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(self.type_, self.value, self.tb)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1347, in attach_volume
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     guest.attach_device(conf, persistent=True, live=live)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 250, in attach_device
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     result = proxy_call(self._autowrap, f, *args, **kwargs)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = execute(f, *args, **kwargs)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     six.reraise(c, e, tb)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     rv = meth(*args, **kwargs)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] libvirtError: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
      ERROR nova.compute.manager [instance: 5779ea20-089c-497d-a2b6-b64028f919fb] 
      ERROR oslo_messaging.rpc.dispatcher [req-e7304c39-599a-4d5b-aa08-cb3e9e516718 93f142cf119845f78275557fb93af581 c695ba08f9bd436ca13dc1aa61c24701 - - -] Exception during message handling: Failed to open file '/var/lib/nova/mnt/e2ea589d93e0d379e67b4c381c8d5e0e/volume-e230e8c4-a5d7-4370-80bc-b6557afccc40': Permission denied
      

Environment

  • Red Hat OpenStack Platform 9
  • Red Hat OpenStack Platform 10
  • openstack-cinder
  • Cinder NFS backend
  • no_root_squash enabled NFS server.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In