Create Image from volume snapshot failed

Issue

• In OSP 16 production environment, we are trying to create golden image from existing VM (CentOS 8.2) by following steps:
  1) Stop VM
  2) Create Volume snapshot from Volume attached to VM
  3) Create Volume from Volume snapshot
  4) Upload glance image (both RAW and QCOW2) from Volume (created in previous step)
  5) Create instance (testVM) from the image RAW
  6) Create instance (testVM1) from the image QCOW2

• VMs creation fail with:

    Build of instance 5e82eb2f-fb4b-44fa-8c18-62c56698dec7 aborted: Volume 44a4138c-48ff-49ce-92d7-3175db3a96f5 did not finish being created even after we waited 288 seconds or 95 attempts. And its status is error.
Code
    500
Details
    Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1905, in _prep_block_device wait_func=self._await_block_device_map_created) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 892, in attach_block_devices _log_and_attach(device) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 889, in _log_and_attach bdm.attach(*attach_args, **attach_kwargs) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 789, in attach wait_func=wait_func, image_id=self.image_id) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 384, in _create_volume self._call_wait_func(context, wait_func, volume_api, vol['id']) File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 749, in _call_wait_func {'volume_id': volume_id, 'exc': exc}) File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise raise value File "/usr/lib/python3.6/site-packages/nova/virt/block_device.py", line 739, in _call_wait_func wait_func(context, volume_id) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1564, in _await_block_device_map_created volume_status=volume_status) nova.exception.VolumeNotCreated: Volume 44a4138c-48ff-49ce-92d7-3175db3a96f5 did not finish being created even after we waited 288 seconds or 95 attempts. And its status is error. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2628, in _build_resources block_device_mapping) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 1923, in _prep_block_device raise exception.InvalidBDM(six.text_type(ex)) nova.exception.InvalidBDM: Volume 44a4138c-48ff-49ce-92d7-3175db3a96f5 did not finish being created even after we waited 288 seconds or 95 attempts. And its status is error. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2161, in _do_build_and_run_instance filter_properties, request_spec) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2469, in _build_and_run_instance bdms=block_device_mapping, tb=tb) File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise raise value File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2419, in _build_and_run_instance request_group_resource_providers_mapping) as resources: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2647, in _build_resources reason=e.format_message()) nova.exception.BuildAbortException: Build of instance 5e82eb2f-fb4b-44fa-8c18-62c56698dec7 aborted: Volume 44a4138c-48ff-49ce-92d7-3175db3a96f5 did not finish being created even after we waited 288 seconds or 95 attempts. And its status is error. 
Created
    July 12, 2021, 4:41 p.m.

• We see the following error in /var/log/containers/cinder/cinder-volume.log:

2021-07-12 16:41:52.211 111 DEBUG cinder.volume.flows.manager.create_volume [req-df41f2c1-8fb2-4f6a-b523-5c4c7139d0fa 7f4f95dba91047679f85bd8dd11f97b4 3ba65e1ca425440badc895158646f928 - default default] Copying metadata from image 8f03e27
4-bb41-489a-83c7-82b48e5e8cf3 to 44a4138c-48ff-49ce-92d7-3175db3a96f5. _handle_bootable_volume_glance_meta /usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py:441
2021-07-12 16:41:52.211 111 DEBUG cinder.volume.flows.manager.create_volume [req-df41f2c1-8fb2-4f6a-b523-5c4c7139d0fa 7f4f95dba91047679f85bd8dd11f97b4 3ba65e1ca425440badc895158646f928 - default default] Creating volume glance metadata for
 volume 44a4138c-48ff-49ce-92d7-3175db3a96f5 backed by image 8f03e274-bb41-489a-83c7-82b48e5e8cf3 with: {'description': 'CentOS 8 (Core) GenericCloud 8.2-2004 Cloud Image', 'signature_verified': 'False', 'image_id': '8f03e274-bb41-489a-83
c7-82b48e5e8cf3', 'image_name': 'supportcase-test-image', 'checksum': '5203f0b069ca717a9f722e59a52618e3', 'container_format': 'bare', 'disk_format': 'raw', 'min_disk': 0, 'min_ram': 0, 'size': 32212254720}. _capture_volume_image_metadata 
/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py:656
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume [req-df41f2c1-8fb2-4f6a-b523-5c4c7139d0fa 7f4f95dba91047679f85bd8dd11f97b4 3ba65e1ca425440badc895158646f928 - default default] Failed updating volume 44a4138c-48f
f-49ce-92d7-3175db3a96f5 metadata using the provided image 8f03e274-bb41-489a-83c7-82b48e5e8cf3 metadata: cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 44a4138c-48ff-
49ce-92d7-3175db3a96f5
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 443, in _handle_bootable_volume_glance_meta
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume     image_id, image_meta)
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 658, in _capture_volume_image_metadata
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume     volume_metadata)
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 975, in volume_glance_metadata_bulk_create
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume     metadata)
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume     return f(*args, **kwargs)
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 204, in wrapper
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume     return f(context, volume_id, *args, **kwargs)
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 5046, in volume_glance_metadata_bulk_create
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume     volume_id=volume_id)
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 44a4138c-48ff-49ce-92d7-3175db3a96f5
2021-07-12 16:41:52.219 111 ERROR cinder.volume.flows.manager.create_volume 
2021-07-12 16:41:52.224 111 WARNING cinder.volume.manager [req-df41f2c1-8fb2-4f6a-b523-5c4c7139d0fa 7f4f95dba91047679f85bd8dd11f97b4 3ba65e1ca425440badc895158646f928 - default default] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (51e17d24-9cc1-4fc3-a8d0-cbfedc4eb983) transitioned into state 'FAILURE' from state 'RUNNING'