instance snapshot failing with ERROR nova.api.openstack.wsgi nova.exception.InternalError: Error from libvirt while quiescing

Issue

• While taking snapshot of any instance failed with an exception thrown in the api logs

/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi [req-7994b88a-e684-48ce-8a31-e63fc8d82dcb f1d54e5c145117af8d5fc395ca4fdf438e002f98d51db43cfe8e3d5f4c4d9f68 97ab3f88e6c64ed4a43be29679443575 - d85d189837b5494ea1108ad291534
1db d85d189837b5494ea1108ad2915341db] Unexpected exception in API method: nova.exception_Remote.InternalError_Remote: Error from libvirt while quiescing instance-00000c9c: [Error Code 1] internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': f
ailed to open /var/lib/docker/overlay2: Permission denied
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 1238, in _action_create_image
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     metadata)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 152, in inner
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return f(self, context, instance, *args, **kw)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 3151, in snapshot_volume_backed
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     self.compute_rpcapi.quiesce_instance(context, instance)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/compute/rpcapi.py", line 1217, in quiesce_instance
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return cctxt.call(ctxt, 'quiesce_instance', instance=instance)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 181, in call
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     transport_options=self.transport_options)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 129, in _send
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     transport_options=transport_options)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 674, in send
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     transport_options=transport_options)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 664, in _send
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     raise result
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi nova.exception_Remote.InternalError_Remote: Error from libvirt while quiescing instance-00000c9c: [Error Code 1] internal error: unable to execute QEMU agent command 'gues
t-fsfreeze-freeze': failed to open /var/lib/docker/overlay2: Permission denied
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2692, in _set_quiesced
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     guest.freeze_filesystems()
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 580, in freeze_filesystems
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     self._domain.fsFreeze()
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     result = proxy_call(self._autowrap, f, *args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     rv = execute(f, *args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     six.reraise(c, e, tb)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     raise value
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     rv = meth(*args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3069, in fsFreeze
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     if ret == -1: raise libvirtError ('virDomainFSFreeze() failed', dom=self)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi libvirt.libvirtError: internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /var/lib/docker/overlay2: Permission denied
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred:
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 235, in inner
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     function_name, call_dict, binary, tb)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     self.force_reraise()
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     six.reraise(self.type_, self.value, self.tb)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     raise value
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     return f(self, context, *args, **kw)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9476, in quiesce_instance
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     self.driver.quiesce(context, instance, image_meta)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2709, in quiesce
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     self._set_quiesced(context, instance, image_meta, True)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2702, in _set_quiesced
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi     raise exception.InternalError(msg)
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi nova.exception.InternalError: Error from libvirt while quiescing instance-00000c9c: [Error Code 1] internal error: unable to execute QEMU agent command 'guest-fsfreeze-fre
eze': failed to open /var/lib/docker/overlay2: Permission denied
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.884 28 ERROR nova.api.openstack.wsgi
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.889 28 INFO nova.api.openstack.wsgi [req-7994b88a-e684-48ce-8a31-e63fc8d82dcb f1d54e5c145117af8d5fc395ca4fdf438e002f98d51db43cfe8e3d5f4c4d9f68 97ab3f88e6c64ed4a43be29679443575 - d85d189837b5494ea1108ad2915341
db d85d189837b5494ea1108ad2915341db] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
/var/log/containers/nova/nova-api.log:2023-12-07 19:01:08.892 28 INFO nova.api.openstack.requestlog [req-7994b88a-e684-48ce-8a31-e63fc8d82dcb f1d54e5c145117af8d5fc395ca4fdf438e002f98d51db43cfe8e3d5f4c4d9f68 97ab3f88e6c64ed4a43be29679443575 - d85d189837b5494ea1108ad2
915341db d85d189837b5494ea1108ad2915341db] 10.133.64.19 "POST /v2.1/servers/8548cd67-c9de-4c30-87d0-6119a1da4949/action" status: 500 len: 216 microversion: 2.1 time: 0.308069