After 13-16.1 ffu upgrade, openstack overcloud deploy always fails because cinder endpoints
Issue
After OSP13->OSP16.1 upgrade, openstack overcloud deploy always fails because of the error in "Check Keystone public endpoint status" task.
By removing no_logs: true in /usr/share/ansible/roles/tripleo-keystone-resources/tasks/endpoints.yml, I captured the following error in tripleo-keystone-resources.
...
TASK [tripleo-keystone-resources : Check Keystone public endpoint status] ******
Sunday 13 September 2020 11:14:39 +0000 (0:00:05.000) 0:25:13.826 ******
...
failed: [undercloud] (item={'started': 1, 'finished': 0, 'ansible_job_id': '668760418547.976504', 'results_file': '/root/.ansible_async/668760418547.976504', 'changed': True, 'failed': False, 'tripleo_keystone_resources_data': {'key': 'cinderv3', 'value': {'endpoints': {'admin': 'http://172.17.1.25:8776/v3/%(tenant_id)s', 'internal': 'http://172.17.1.25:8776/v3/%(tenant_id)s', 'public': 'http://10.0.0.145:8776/v3/%(tenant_id)s'}, 'region': 'regionOne', 'service': 'volumev3', 'users': {'cinderv3': {'password': 'NAR9UexZ7u28rCGRuEEArtt7v', 'roles': ['admin', 'service']}}}}, 'ansible_loop_var': 'tripleo_keystone_resources_data'}) => {"ansible_job_id": "668760418547.976504", "ansible_loop_var": "tripleo_keystone_resources_endpoint_async_result_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Multiple matches found for cinderv3", "tripleo_keystone_resources_endpoint_async_result_item": {"ansible_job_id": "668760418547.976504", "ansible_loop_var": "tripleo_keystone_resources_data", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/668760418547.976504", "started": 1, "tripleo_keystone_resources_data": {"key": "cinderv3", "value": {"endpoints": {"admin": "http://172.17.1.25:8776/v3/%(tenant_id)s", "internal": "http://172.17.1.25:8776/v3/%(tenant_id)s", "public": "http://10.0.0.145:8776/v3/%(tenant_id)s"}, "region": "regionOne", "service": "volumev3", "users": {"cinderv3": {"password": "NAR9UexZ7u28rCGRuEEArtt7v", "roles": ["admin", "service"]}}}}}}
...
NO MORE HOSTS LEFT *************************************************************
PLAY RECAP *********************************************************************
compute-0 : ok=256 changed=103 unreachable=0 failed=0 skipped=118 rescued=0 ignored=0
compute-1 : ok=252 changed=103 unreachable=0 failed=0 skipped=118 rescued=0 ignored=0
controller-0 : ok=317 changed=138 unreachable=0 failed=0 skipped=127 rescued=0 ignored=0
controller-1 : ok=306 changed=138 unreachable=0 failed=0 skipped=128 rescued=0 ignored=0
controller-2 : ok=306 changed=138 unreachable=0 failed=0 skipped=128 rescued=0 ignored=0
undercloud : ok=51 changed=16 unreachable=0 failed=1 skipped=44 rescued=0 ignored=0
Sunday 13 September 2020 11:14:42 +0000 (0:00:02.533) 0:25:16.360 ******
===============================================================================
Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log.
Overcloud configuration failed.
Cinder v1 endpoint created during deployment of OSP13, while it is no longer created in OSP16.1 deployment.
The problem is that it is not purged during upgrade and left in OSP16.1 deployment after upgrade,
then results in the deployment failure.
Fresh RHOSP13 deployment
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list --service cinderv3
Multiple service matches found for 'cinderv3', use an ID to be more specific.
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep cinderv3
| 0232351c41fc41eb8dc281a01f27e20b | regionOne | cinderv3 | volumev3 | True | internal | http://172.17.1.103:8776/v3/%(tenant_id)s |
| 1cfa53ca39184f94a55a91ed6d3ae8af | regionOne | cinderv3 | volume | True | admin | http://172.17.1.103:8776/v3/%(tenant_id)s |
| 5a6b1149cc4142d08e411e10743b8356 | regionOne | cinderv3 | volume | True | public | http://10.0.0.113:8776/v3/%(tenant_id)s |
| 7aea0836ca984a6fb4dd2c55318dd8af | regionOne | cinderv3 | volumev3 | True | public | http://10.0.0.113:8776/v3/%(tenant_id)s |
| c04ae1bae20b45e0b127749e75911907 | regionOne | cinderv3 | volume | True | internal | http://172.17.1.103:8776/v3/%(tenant_id)s |
| e097455080374c5fad4ab96725e8ec8c | regionOne | cinderv3 | volumev3 | True | admin | http://172.17.1.103:8776/v3/%(tenant_id)s |
Fresh RHOSP16.1 deployment
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list --service cinderv3
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
| 691ce1bdc9b343ebb8fee891bd2b9733 | regionOne | cinderv3 | volumev3 | True | admin | http://172.17.1.49:8776/v3/%(tenant_id)s |
| 905b33cf92454c7881c8c222b5c30059 | regionOne | cinderv3 | volumev3 | True | public | https://10.0.0.101:13776/v3/%(tenant_id)s |
| e133d65395874e5fb2dd0ae53abd268e | regionOne | cinderv3 | volumev3 | True | internal | http://172.17.1.49:8776/v3/%(tenant_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-------------------------------------------+
RHOSP16.1 upgraded from RHOSP13
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list --service cinderv3
Multiple service matches found for 'cinderv3', use an ID to be more specific.
(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list | grep cinderv3
| 596146ea98914c69ab77b023ff3ade3e | regionOne | cinderv3 | volume | True | public | http://10.0.0.145:8776/v3/%(tenant_id)s |
| 5e2429696b2143c8a9dbe7553fde9e1e | regionOne | cinderv3 | volume | True | admin | http://172.17.1.25:8776/v3/%(tenant_id)s |
| 8eeb1f1780ba46769cd1112ac1899a46 | regionOne | cinderv3 | volume | True | internal | http://172.17.1.25:8776/v3/%(tenant_id)s |
| 95dbaf739c3348cf8016ee50b53eea9a | regionOne | cinderv3 | volumev3 | True | public | http://10.0.0.145:8776/v3/%(tenant_id)s |
| a5ca4c12a3f04ac5b11bd1c7334fc094 | regionOne | cinderv3 | volumev3 | True | internal | http://172.17.1.25:8776/v3/%(tenant_id)s |
| d98b138eba27410097ce333d6fb5a7cb | regionOne | cinderv3 | volumev3 | True | admin | http://172.17.1.25:8776/v3/%(tenant_id)s |
Environment
- Red Hat OpenStack Platform 16.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.