Creating a manila share fails and gets stuck in "creating" status

Issue

• We have deployed our nodes with RHOSP 16.1 with ceph and when we are trying to create a manila share, it is getting stuck at creating[bug] state forever and when we try to force delete the same share, it is going into deleting state forever. It is neither getting created nor deleted.

  • We created a new type manila type-create default false:

+----------------------+--------------------------------------+
| Property             | Value                                |
+----------------------+--------------------------------------+
| ID                   | f9b1ee6e-ef9d-4ba8-a6fa-3245636f5011 |
| Name                 | default                              |
| Visibility           | public                               |
| is_default           | YES                                  |
| required_extra_specs | driver_handles_share_servers : False |
| optional_extra_specs |                                      |
| Description          | None                                 |
+----------------------+--------------------------------------+

• We created a new share with manila create --name share-01 nfs 10:

+---------------------------------------+--------------------------------------+
| Property                              | Value                                |
+---------------------------------------+--------------------------------------+
| id                                    | b0c35891-1f64-499a-846c-a3243a3350f4 |
| size                                  | 10                                   |
| availability_zone                     | None                                 |
| created_at                            | 2021-07-08T11:30:26.000000           |
| status                                | creating                             |
| name                                  | share-01                             |
| description                           | None                                 |
| project_id                            | f023c93840cf431b960d3e52e80eefa2     |
| snapshot_id                           | None                                 |
| share_network_id                      | None                                 |
| share_proto                           | NFS                                  |
| metadata                              | {}                                   |
| share_type                            | f9b1ee6e-ef9d-4ba8-a6fa-3245636f5011 |
| is_public                             | False                                |
| snapshot_support                      | False                                |
| task_state                            | None                                 |
| share_type_name                       | default                              |
| access_rules_status                   | active                               |
| replication_type                      | None                                 |
| has_replicas                          | False                                |
| user_id                               | f4c3ec7d513c43658c37f8efaefb2d18     |
| create_share_from_snapshot_support    | False                                |
| revert_to_snapshot_support            | False                                |
| share_group_id                        | None                                 |
| source_share_group_snapshot_member_id | None                                 |
| mount_snapshot_support                | False                                |
| share_server_id                       | None                                 |
| host                                  |                                      |
+---------------------------------------+--------------------------------------+

• The existing manila types returned with manila type-list:

+--------------------------------------+---------+------------+------------+--------------------------------------+----------------------+-------------+
| ID                                   | Name    | visibility | is_default | required_extra_specs                 | optional_extra_specs | Description |
+--------------------------------------+---------+------------+------------+--------------------------------------+----------------------+-------------+
| f9b1ee6e-ef9d-4ba8-a6fa-3245636f5011 | default | public     | YES        | driver_handles_share_servers : False |                      | None        |
+--------------------------------------+---------+------------+------------+--------------------------------------+----------------------+-------------+

• We can see the share is stuck in CREATING with manila list:

+--------------------------------------+----------+------+-------------+----------+-----------+-----------------+-------------------------+-------------------+
| ID                                   | Name     | Size | Share Proto | Status   | Is Public | Share Type Name | Host                    | Availability Zone |
+--------------------------------------+----------+------+-------------+----------+-----------+-----------------+-------------------------+-------------------+
| b0c35891-1f64-499a-846c-a3243a3350f4 | share-01 | 10   | NFS         | creating | False     | default         | hostgroup@cephfs#cephfs | nova              |
+--------------------------------------+----------+------+-------------+----------+-----------+-----------------+-------------------------+-------------------+

• We see the following error in the manila logs:

2021-06-27 10:22:00.278 290 DEBUG manila.share.drivers.cephfs.driver
[req-d9e9bf29-7254-4ae8-8739-6256298b768d
345024b1d79c4a1780d88f796575097c d0d3edbb67754c01a4b71f0b379ca120 - -
-] create_share cephfs name=8dc0239c-1aef-4dc5-8647-f7bde9db4f66
size=1 share_group_id=None create_share
/usr/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py:262
2021-06-27 10:22:00.288 290 INFO ceph_volume_client
[req-d9e9bf29-7254-4ae8-8739-6256298b768d
345024b1d79c4a1780d88f796575097c d0d3edbb67754c01a4b71f0b379ca120 - -
-] create_volume:
/volumes/_nogroup/8dc0239c-1aef-4dc5-8647-f7bde9db4f66
2021-06-27 10:22:01.017 7 INFO oslo_service.service
[req-54e69491-1385-4065-9f86-a77c606d65c6 - - - - -] Child 290 killed
by signal 6
2021-06-27 10:22:01.023 7 DEBUG oslo_service.service
[req-54e69491-1385-4065-9f86-a77c606d65c6 - - - - -] Started child 328
_start_child
/usr/lib/python3.6/site-packages/oslo_service/service.py:579