When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.

Solution In Progress - Updated -

Issue

  • When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.

    Sample log: /var/log/nova/nova-compute.log

    ERROR nova.compute.manager [req-6c0276ac-8f3d-465b-b797-85896a454161 31be7e8e6b13419cb3dc9407ce2e2658 6eeec2e8359a49d1b85fb89641f9718f - - -] [instance: fc2305a1-0b1a-4869-8041-153928ec341b] Failed to attach d9978da2-0941-4aa7-9ce0-56cffd1ab5
    
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b] Traceback (most recent call last):
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4670, in _attach_volume
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     do_check_attach=False, do_driver_attach=True)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 52, in wrapped
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     ret_val = method(obj, context, *args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 265, in attach
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     connector)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 256, in attach
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     device_type=self['device_type'], encryption=encryption)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1147, in attach_volume
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     self._disconnect_volume(connection_info, disk_dev)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1136, in attach_volume
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     guest.attach_device(conf, persistent=True, live=live)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 234, in attach_device
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     rv = execute(f, *args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     six.reraise(c, e, tb)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     rv = meth(*args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b] libvirtError: Cannot access storage file '/var/lib/nova/mnt/f0467099cfa1a0eaa33f8cb69056d5d7/volume-d9978da2-0941-4aa7-9ce0-56cffd1ab5ee' (as uid:107, gid:107): No such file or directory
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b] 
    
  • After update to RHEL7.3, the issue occurs.

Environment

  • Red Hat OpenStack Platform 8
  • Red Hat OpenStack Platform 9
  • Red Hat OpenStack Platform 10
  • openstack-nova
  • util-linux-2.23.2-33.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content