After 13-16.1 ffu upgrade, openstack overcloud deploy always fails because cinder endpoints

Solution In Progress - Updated -

Issue

After OSP13->OSP16.1 upgrade, openstack overcloud deploy always fails because of the error in "Check Keystone public endpoint status" task.
By removing no_logs: true in /usr/share/ansible/roles/tripleo-keystone-resources/tasks/endpoints.yml, I captured the following error in tripleo-keystone-resources.

...

TASK [tripleo-keystone-resources : Check Keystone public endpoint status] ******
Sunday 13 September 2020  11:14:39 +0000 (0:00:05.000)       0:25:13.826 ****** 
...
failed: [undercloud] (item={'started': 1, 'finished': 0, 'ansible_job_id': '668760418547.976504', 'results_file': '/root/.ansible_async/668760418547.976504', 'changed': True, 'failed': False, 'tripleo_keystone_resources_data': {'key': 'cinderv3', 'value': {'endpoints': {'admin': 'http://172.17.1.25:8776/v3/%(tenant_id)s', 'internal': 'http://172.17.1.25:8776/v3/%(tenant_id)s', 'public': 'http://10.0.0.145:8776/v3/%(tenant_id)s'}, 'region': 'regionOne', 'service': 'volumev3', 'users': {'cinderv3': {'password': 'NAR9UexZ7u28rCGRuEEArtt7v', 'roles': ['admin', 'service']}}}}, 'ansible_loop_var': 'tripleo_keystone_resources_data'}) => {"ansible_job_id": "668760418547.976504", "ansible_loop_var": "tripleo_keystone_resources_endpoint_async_result_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Multiple matches found for cinderv3", "tripleo_keystone_resources_endpoint_async_result_item": {"ansible_job_id": "668760418547.976504", "ansible_loop_var": "tripleo_keystone_resources_data", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/668760418547.976504", "started": 1, "tripleo_keystone_resources_data": {"key": "cinderv3", "value": {"endpoints": {"admin": "http://172.17.1.25:8776/v3/%(tenant_id)s", "internal": "http://172.17.1.25:8776/v3/%(tenant_id)s", "public": "http://10.0.0.145:8776/v3/%(tenant_id)s"}, "region": "regionOne", "service": "volumev3", "users": {"cinderv3": {"password": "NAR9UexZ7u28rCGRuEEArtt7v", "roles": ["admin", "service"]}}}}}}
...

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
compute-0                  : ok=256  changed=103  unreachable=0    failed=0    skipped=118  rescued=0    ignored=0   
compute-1                  : ok=252  changed=103  unreachable=0    failed=0    skipped=118  rescued=0    ignored=0   
controller-0               : ok=317  changed=138  unreachable=0    failed=0    skipped=127  rescued=0    ignored=0   
controller-1               : ok=306  changed=138  unreachable=0    failed=0    skipped=128  rescued=0    ignored=0   
controller-2               : ok=306  changed=138  unreachable=0    failed=0    skipped=128  rescued=0    ignored=0   
undercloud                 : ok=51   changed=16   unreachable=0    failed=1    skipped=44   rescued=0    ignored=0   

Sunday 13 September 2020  11:14:42 +0000 (0:00:02.533)       0:25:16.360 ****** 
=============================================================================== 

Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.
Overcloud configuration failed.

Cinder v1 endpoint created during deployment of OSP13, while it is no longer created in OSP16.1 deployment.
The problem is that it is not purged during upgrade and left in OSP16.1 deployment after upgrade,
then results in the deployment failure.

Fresh RHOSP13 deployment

(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list --service cinderv3
Multiple service matches found for 'cinderv3', use an ID to be more specific.
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep cinderv3
| 0232351c41fc41eb8dc281a01f27e20b | regionOne | cinderv3     | volumev3       | True    | internal  | http://172.17.1.103:8776/v3/%(tenant_id)s     |
| 1cfa53ca39184f94a55a91ed6d3ae8af | regionOne | cinderv3     | volume         | True    | admin     | http://172.17.1.103:8776/v3/%(tenant_id)s     |
| 5a6b1149cc4142d08e411e10743b8356 | regionOne | cinderv3     | volume         | True    | public    | http://10.0.0.113:8776/v3/%(tenant_id)s       |
| 7aea0836ca984a6fb4dd2c55318dd8af | regionOne | cinderv3     | volumev3       | True    | public    | http://10.0.0.113:8776/v3/%(tenant_id)s       |
| c04ae1bae20b45e0b127749e75911907 | regionOne | cinderv3     | volume         | True    | internal  | http://172.17.1.103:8776/v3/%(tenant_id)s     |
| e097455080374c5fad4ab96725e8ec8c | regionOne | cinderv3     | volumev3       | True    | admin     | http://172.17.1.103:8776/v3/%(tenant_id)s     |

Fresh RHOSP16.1 deployment

(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list --service cinderv3
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| ID                               | Region    | Service Name | Service Type | Enabled | Interface | URL                                       |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| 691ce1bdc9b343ebb8fee891bd2b9733 | regionOne | cinderv3     | volumev3     | True    | admin     | http://172.17.1.49:8776/v3/%(tenant_id)s  |
| 905b33cf92454c7881c8c222b5c30059 | regionOne | cinderv3     | volumev3     | True    | public    | https://10.0.0.101:13776/v3/%(tenant_id)s |
| e133d65395874e5fb2dd0ae53abd268e | regionOne | cinderv3     | volumev3     | True    | internal  | http://172.17.1.49:8776/v3/%(tenant_id)s  |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+

RHOSP16.1 upgraded from RHOSP13

(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list --service cinderv3
Multiple service matches found for 'cinderv3', use an ID to be more specific.
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep cinderv3
| 596146ea98914c69ab77b023ff3ade3e | regionOne | cinderv3     | volume         | True    | public    | http://10.0.0.145:8776/v3/%(tenant_id)s       |
| 5e2429696b2143c8a9dbe7553fde9e1e | regionOne | cinderv3     | volume         | True    | admin     | http://172.17.1.25:8776/v3/%(tenant_id)s      |
| 8eeb1f1780ba46769cd1112ac1899a46 | regionOne | cinderv3     | volume         | True    | internal  | http://172.17.1.25:8776/v3/%(tenant_id)s      |
| 95dbaf739c3348cf8016ee50b53eea9a | regionOne | cinderv3     | volumev3       | True    | public    | http://10.0.0.145:8776/v3/%(tenant_id)s       |
| a5ca4c12a3f04ac5b11bd1c7334fc094 | regionOne | cinderv3     | volumev3       | True    | internal  | http://172.17.1.25:8776/v3/%(tenant_id)s      |
| d98b138eba27410097ce333d6fb5a7cb | regionOne | cinderv3     | volumev3       | True    | admin     | http://172.17.1.25:8776/v3/%(tenant_id)s      |

Environment

  • Red Hat OpenStack Platform 16.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content