Cannot Create Volumes - Error

Issue

• When we create a volume via the openstack dashboard or cli the volume returns a status of "Error". We have edited nova.conf in all of my compute nodes and changed a few variables and restarted the docker service per this documentation, but still no luck

• We thought we were hitting the same issue as this article.

• Volumes are never created:

(undercloud) [stack@undercloud ~]$ source overcloudrc 
(overcloud) [stack@undercloud ~]$ openstack volume create --size 100 yy
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2021-01-26T19:06:05.000000           |
| description         | None                                 |
| encrypted           | False                                |
| id                  | c6310045-9920-4373-9a51-3ed32eada09c |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | yy                                   |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 100                                  |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | tripleo                              |
| updated_at          | None                                 |
| user_id             | 3da287fd5a03484188e49f122d066b1a     |
+---------------------+--------------------------------------+
(overcloud) [stack@undercloud ~]$ openstack volume list
+--------------------------------------+---------------------+--------+------+-------------+
| ID                                   | Name                | Status | Size | Attached to |
+--------------------------------------+---------------------+--------+------+-------------+
| c6310045-9920-4373-9a51-3ed32eada09c | yy                  | error  |  100 |             |
| c6e33880-1547-4f29-a356-fd60cd78b1ba | Centos 7 ISO (2020) | error  |    5 |             |
| b9002319-38d5-4e30-ab46-7784832af85d |                     | error  |   50 |             |
+--------------------------------------+---------------------+--------+------+-------------+
(overcloud) [stack@undercloud ~]$ openstack volume show yy
+--------------------------------+--------------------------------------+
| Field                          | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2021-01-26T19:06:05.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | c6310045-9920-4373-9a51-3ed32eada09c |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | yy                                   |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 0cac70ef49df4434ba072102ae12d1c1     |
| properties                     |                                      |
| replication_status             | None                                 |
| size                           | 100                                  |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | error                                |
| type                           | tripleo                              |
| updated_at                     | 2021-01-26T19:06:06.000000           |
| user_id                        | 3da287fd5a03484188e49f122d066b1a     |
+--------------------------------+--------------------------------------+

• Many resources are failed and/or blocked as pcs status shows:

[...]
Docker container: openstack-cinder-volume [192.168.88.70:8787/rhosp13/openstack-cinder-volume:pcmklatest]
      openstack-cinder-volume-docker-0  (ocf::heartbeat:docker):    FAILED overcloud-controller-1 (blocked)
[...]

Failed Resource Actions:
* rabbitmq-bundle-docker-2_stop_0 on overcloud-controller-1 'unknown error' (1): call=98, status=Timed Out, exitreason='',
    last-rc-change='Mon Jan 25 14:12:28 2021', queued=0ms, exec=20003ms
* galera-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=100, status=Timed Out, exitreason='',
    last-rc-change='Mon Jan 25 14:12:51 2021', queued=0ms, exec=20002ms
* redis-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=106, status=Timed Out, exitreason='',
    last-rc-change='Mon Jan 25 14:13:11 2021', queued=0ms, exec=20001ms
* haproxy-bundle-docker-1_stop_0 on overcloud-controller-1 'unknown error' (1): call=102, status=Timed Out, exitreason='',
    last-rc-change='Mon Jan 25 14:13:11 2021', queued=0ms, exec=20003ms
* openstack-cinder-volume-docker-0_stop_0 on overcloud-controller-1 'unknown error' (1): call=104, status=Timed Out, exitreason='',
    last-rc-change='Mon Jan 25 14:13:11 2021', queued=0ms, exec=20001ms
[...]