Failed to delete a volume snapshot due to 'cinder.exception.RemoteFSInvalidBackingFile'

Issue

• Failed to delete a volume snapshot even if --force option was added.
• cinder.exception.RemoteFSInvalidBackingFile exception can be seen in /var/log/containers/cinder/cinder-volume.log on any of Controller nodes.

2024-12-02 15:59:59.607 66 INFO cinder.message.api [req-d7825e4c-b2f2-483a-b571-353eac872253 c0187f8021ea497bb1a8279ecf742baa 29260b2ba025427185e045a184e7d3c2 - - -] Creating message record for request_id = req-d7825e4c-b2f2-483a-b571-353eac872253
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server [req-d7825e4c-b2f2-483a-b571-353eac872253 c0187f8021ea497bb1a8279ecf742baa 29260b2ba025427185e045a184e7d3c2 - - -] Exception during message handling: cinder.exception.RemoteFSInvalidBackingFile: File /var/lib/cinder/mnt/fb5ff814-40f8-4a69-87fa-af00cb82271b/volume-3798f25e-5703-473e-900a-fcd7fa99c31c.d229f50e-d938-43c6-885f-17e49068b2b0 has invalid backing file /var/lib/nova/mnt/fb5ff814-40f8-4a69-87fa-af00cb82271b/volume-3798f25e-5703-473e-900a-fcd7fa99c31c.
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "<decorator-gen-11>", line 2, in delete_snapshot
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/coordination.py", line 151, in _synchronized
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 1276, in delete_snapshot
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     self.message_api.create(
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     self.force_reraise()
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     raise self.value
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 1259, in delete_snapshot
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     self.driver.delete_snapshot(snapshot)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "<decorator-gen-39>", line 2, in delete_snapshot
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/coordination.py", line 151, in _synchronized
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/nfs.py", line 597, in delete_snapshot
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     return self._delete_snapshot(snapshot)
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/remotefs.py", line 1257, in _delete_snapshot
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     snapshot_path_img_info = self._qemu_img_info(
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/nfs.py", line 562, in _qemu_img_info
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     return super(NfsDriver, self)._qemu_img_info_base(
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/remotefs.py", line 868, in _qemu_img_info_base
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server     raise exception.RemoteFSInvalidBackingFile(
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server cinder.exception.RemoteFSInvalidBackingFile: File /var/lib/cinder/mnt/fb5ff814-40f8-4a69-87fa-af00cb82271b/volume-3798f25e-5703-473e-900a-fcd7fa99c31c.d229f50e-d938-43c6-885f-17e49068b2b0 has invalid backing file /var/lib/nova/mnt/fb5ff814-40f8-4a69-87fa-af00cb82271b/volume-3798f25e-5703-473e-900a-fcd7fa99c31c.
2024-12-02 15:59:59.610 66 ERROR oslo_messaging.rpc.server