"File lock acquisition" error causes an instance to fail to boot when NFSv3 share is mounted on /var/lib/nova/instances

Solution Verified - Updated -

Issue

  • "File lock acquisition" error causes an instance to fail to boot when NFSv3 share is mounted on /var/lib/nova/instances.

    Sample log

    INFO nova.virt.libvirt.driver [req-YYY - - - - -] [instance: XXX] Creating image
    ERROR nova.compute.manager [req-YYY - - - - -] [instance: XXX] Instance failed to spawn
    TRACE nova.compute.manager [instance: XXX] Traceback (most recent call last):
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2455, in _build_resources
    TRACE nova.compute.manager [instance: XXX]     yield resources
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2327, in _build_and_run_instance
    TRACE nova.compute.manager [instance: XXX]     block_device_info=block_device_info)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2424, in spawn
    TRACE nova.compute.manager [instance: XXX]     admin_pass=admin_password)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2835, in _create_image
    TRACE nova.compute.manager [instance: XXX]     instance, size, fallback_from_host)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6015, in _try_fetch_image_cache
    TRACE nova.compute.manager [instance: XXX]     size=size)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
    TRACE nova.compute.manager [instance: XXX]     *args, **kwargs)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 537, in create_image
    TRACE nova.compute.manager [instance: XXX]     copy_qcow2_image(base, self.path, size)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 439, in inner
    TRACE nova.compute.manager [instance: XXX]     do_log=False, semaphores=semaphores, delay=delay):
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    TRACE nova.compute.manager [instance: XXX]     return self.gen.next()
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 395, in lock
    TRACE nova.compute.manager [instance: XXX]     ext_lock.acquire(delay=delay)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 209, in acquire
    TRACE nova.compute.manager [instance: XXX]     do_acquire()
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 158, in wrapper
    TRACE nova.compute.manager [instance: XXX]     return r.call(func, *args, **kwargs)
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/retrying.py", line 222, in call
    TRACE nova.compute.manager [instance: XXX]     if not self.should_reject(attempt):
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/retrying.py", line 206, in should_reject
    TRACE nova.compute.manager [instance: XXX]     reject |= self._retry_on_exception(attempt.value[1])
    TRACE nova.compute.manager [instance: XXX]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 130, in retry_on_exception
    TRACE nova.compute.manager [instance: XXX]     'exception': e,
    TRACE nova.compute.manager [instance: XXX] error: Unable to acquire lock on `/var/lib/nova/instances/locks/nova-ZZZ` due to [Errno 5] Input/output error
    TRACE nova.compute.manager [instance: XXX] 
    

Environment

  • Red Hat OpenStack Platform 13.0
  • openstack-nova
  • NFSv3 storage without NLMv4 support is mounted on /var/lib/nova/instances

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In