When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.

Solution In Progress - Updated -

Issue

  • When trying to create an instance with creating volume or attaching existing cinder volume to an instance, these operations fails under cinder NFS backend.

    Sample log: /var/log/nova/nova-compute.log

    ERROR nova.compute.manager [req-6c0276ac-8f3d-465b-b797-85896a454161 31be7e8e6b13419cb3dc9407ce2e2658 6eeec2e8359a49d1b85fb89641f9718f - - -] [instance: fc2305a1-0b1a-4869-8041-153928ec341b] Failed to attach d9978da2-0941-4aa7-9ce0-56cffd1ab5
    
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b] Traceback (most recent call last):
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4670, in _attach_volume
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     do_check_attach=False, do_driver_attach=True)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 52, in wrapped
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     ret_val = method(obj, context, *args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 265, in attach
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     connector)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 256, in attach
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     device_type=self['device_type'], encryption=encryption)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1147, in attach_volume
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     self._disconnect_volume(connection_info, disk_dev)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     six.reraise(self.type_, self.value, self.tb)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1136, in attach_volume
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     guest.attach_device(conf, persistent=True, live=live)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 234, in attach_device
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     rv = execute(f, *args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     six.reraise(c, e, tb)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     rv = meth(*args, **kwargs)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in attachDeviceFlags
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b] libvirtError: Cannot access storage file '/var/lib/nova/mnt/f0467099cfa1a0eaa33f8cb69056d5d7/volume-d9978da2-0941-4aa7-9ce0-56cffd1ab5ee' (as uid:107, gid:107): No such file or directory
    ERROR nova.compute.manager [instance: fc2305a1-0b1a-4869-8041-153928ec341b] 
    
  • After update to RHEL7.3, the issue occurs.

Environment

  • Red Hat OpenStack Platform 8
  • Red Hat OpenStack Platform 9
  • Red Hat OpenStack Platform 10
  • openstack-nova
  • util-linux-2.23.2-33.el7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In