Not able to create more than 10 images

Solution In Progress - Updated -

Environment

  • Red Hat OpenStack Platform 17.x (RHOSP)
  • Red Hat OpenStack Platform 16.x (RHOSP)

Issue

  • When attempting to create an image, the status of the image becomes 'queued', and after a while, it disappears from the image list.
(overcloud) $ openstack image create --disk-format qcow2 --volume <VOLNAME> --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi xxxxxxxxxxxxxx-20240319-xxxxx
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| container_format    | bare                                 |
| disk_format         | qcow2                                |
| display_description | None                                 |
| id                  | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx |
| image_id            | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx |
| image_name          | xxxxxxxxxxxxxx-20240319-xxxxx        |
| protected           | False                                |
| size                | 145                                  |
| status              | uploading                            |
| updated_at          | 2024-03-19T02:35:57.000000           |
| visibility          | shared                               |
| volume_type         | xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
+---------------------+--------------------------------------+
(overcloud) $ openstack image list
+--------------------------------------+-------------------------------+--------+
| ID                                   | Name                          | Status |
+--------------------------------------+-------------------------------+--------+
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240227-01-xx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240227-02-xx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240228-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240301-02-xx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240301-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240306-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240319-xxxxx | queued | <--- This image is gone in the next "openstack image list".
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240229-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240226-xxxxx | active |
+--------------------------------------+-------------------------------+--------+
(overcloud) $ openstack image list
+--------------------------------------+-------------------------------+--------+
| ID                                   | Name                          | Status |
+--------------------------------------+-------------------------------+--------+
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240227-01-xx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240227-02-xx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240228-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240301-02-xx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240301-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240306-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240229-xxxxx | active |
| xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | xxxxxxxxxxxxxx-20240226-xxxxx | active |
+--------------------------------------+-------------------------------+--------+
  • Our deployment uses cinder as the backend for glance:
[root@overcloud-controller-0 ~]# grep ^stores /var/lib/config-data/puppet-generated/glance_api/etc/glance/glance-api.conf
stores=http,cinder
  • We use Dell Unity as the backend for cinder:
[root@overcloud-controller-0 ~]# grep volume_driver /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
#volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_driver=cinder.volume.drivers.dell_emc.unity.Driver
  • We are not able to create more than 10 images with glance. It ends up with the error:
2020-04-22 13:33:28.604 24 DEBUG glance_store._drivers.cinder [req-5614de0d-e751-4ef7-986c-db73b400b44e 0dbe1fca33904038a29a3d94dd998b33 6996f85c06be4d648ebf0eabde35e119 - default default] Creating a new volume: image_size=0 size_gb=1 type=None add /usr/lib/python3.6/site-packages/glance_store/_drivers/cinder.py:695
2020-04-22 13:33:28.604 24 INFO glance_store._drivers.cinder [req-5614de0d-e751-4ef7-986c-db73b400b44e 0dbe1fca33904038a29a3d94dd998b33 6996f85c06be4d648ebf0eabde35e119 - default default] Since image size is zero, we will be doing resize-before-write for each GB which will be considerably slower than normal.
2020-04-22 13:33:29.676 24 ERROR glance.api.v2.image_data [req-5614de0d-e751-4ef7-986c-db73b400b44e 0dbe1fca33904038a29a3d94dd998b33 6996f85c06be4d648ebf0eabde35e119 - default default] Failed to upload image data due to internal error: cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5b0cd390-5a1c-4680-82de-134bbb3e7b96)
2020-04-22 13:33:29.708 24 ERROR glance.common.wsgi [req-5614de0d-e751-4ef7-986c-db73b400b44e 0dbe1fca33904038a29a3d94dd998b33 6996f85c06be4d648ebf0eabde35e119 - default default] Caught error: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5b0cd390-5a1c-4680-82de-134bbb3e7b96): cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5b0cd390-5a1c-4680-82de-134bbb3e7b96)
  • We updated volume quotas to 100, but this did not help.

Resolution

  • We noticed that quota-defaults for volumes was set to 10. Increasing the quota fixed the issue:
(overcloud) [stack@undercloud ~]$ cinder quota-defaults Openstack-DEV
+-----------------------+-------+
| Property              | Value |
+-----------------------+-------+
| backup_gigabytes      | 1000  |
| backups               | 10    |
| gigabytes             | 1000  |
| gigabytes___DEFAULT__ | -1    |
| gigabytes_tripleo     | -1    |
| groups                | 10    |
| per_volume_gigabytes  | -1    |
| snapshots             | 10    |
| snapshots___DEFAULT__ | -1    |
| snapshots_tripleo     | -1    |
| volumes               | 10    |
| volumes___DEFAULT__   | -1    |
| volumes_tripleo       | -1    |
+-----------------------+-------+
  • An alternative workaround would be to set the volumes quota of the services tenant / project to unlimited with -1 or a higher value.
1. Confirm the maximum number of volumes.
  $ openstack quota show service -c volumes

2. Change the maximum number of volumes.
  $ openstack quota set --volumes <new number of volumes> service

3. Confirm that the maximum number of volumes has been changed.
  $ openstack quota show service -c volumes

Root Cause

  • Volume limit was reached for the services tenant / project.

Diagnostic Steps

  • You are using glance image stored cinder backend

  • When trying to create a new image, the following error is shown in glance-api.log:

2020-04-22 13:33:29.708 24 ERROR glance.common.wsgi [req-5614de0d-e751-4ef7-986c-db73b400b44e 0dbe1fca33904038a29a3d94dd998b33 6996f85c06be4d648ebf0eabde35e119 - default default] Caught error: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5b0cd390-5a1c-4680-82de-134bbb3e7b96): cinderclient.exceptions.OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-5b0cd390-5a1c-4680-82de-134bbb3e7b96)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments