Cannot create a volume

Issue

• In the past we have created volumes and have few instances running but now we are unable to create any kind of volumes.

• The following error is present in /var/log/containers/cinder/cinder-volume.log :

cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server [req-4db4b8ca-a938-4581-bcde-9366991f72cd b08c2cefd6be455b902cb75aa3433aae 7260d5aa7a024399adcd6cca60b96e71 - default default] Exception during message handling: ImageCopyFailure: Failed to copy image to volume: qemu-img: error while writing sector 14680064: Input/output error
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in create_volume
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 690, in create_volume
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     _run_flow()
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 682, in _run_flow
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     flow_engine.run()
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     failures[0].reraise()
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1020, in execute
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     **volume_spec)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 929, in _create_from_image
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     image_service)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 830, in _create_from_image_cache_or_download
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     image_service
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 671, in _create_from_image_download
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     image_location, image_service)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 533, in _copy_image_to_volume
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server     raise exception.ImageCopyFailure(reason=ex.stderr)
cinder-volume.log.1:2019-05-13 18:52:47.259 69 ERROR oslo_messaging.rpc.server ImageCopyFailure: Failed to copy image to volume: qemu-img: error while writing sector 14680064: Input/output error

• Creating volumes fails immediately:

(undercloud) [stack@dir ~]$ cinder create --name volume 1
+--------------------------------+--------------------------------------+
| Property                       | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2019-05-15T13:39:15.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 87c266a9-65c5-46fa-9786-e49e1f8a8f13 |
| metadata                       | {}                                   |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | volume                               |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 61025d03106c464685acdf7242b8b7b0     |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | error                                |
| updated_at                     | 2019-05-15T13:39:15.000000           |
| user_id                        | ea268f8711bb4354a4f745b93a83adc7     |
| volume_type                    | None                                 |
+--------------------------------+--------------------------------------+

• All newly created volumes are in error state:

(undercloud) [stack@dir ~]$ cinder list
+--------------------------------------+--------+----------------+------+-------------+----------+-------------+
| ID                                   | Status | Name           | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+----------------+------+-------------+----------+-------------+
| 87c266a9-65c5-46fa-9786-e49e1f8a8f13 | error  | volume         | 1    | -           | false    |             |
| ad8e244a-da48-4373-bf7b-b0223d8031d5 | error  | my-new-volume1 | 100  | -           | false    |             |
| b8c5228a-c87c-400f-b6c0-af07b0e7aa26 | error  | my-new-volume  | 80   | -           | false    |             |
+--------------------------------------+--------+----------------+------+-------------+----------+-------------+