overcloud deploy fails with unable to push cib errors
Issue
note only 1 of the 3 controllers had the unable to push cib errors
- journalctl -u os-collect-config
nError: unable to push cib\nCall cib_apply_diff failed (-203): Update does not conform to the configured schema\n<cib crm_feature_set=\"3.0.14\" validate-with=\"pacemaker-2.10\" epoch=\"7\" num_updates=\"0\" admin_epoch=\"0\" cib-last-written=\"Fri Jul 13 18:13:04 2018\" update-origin=\"overcloud-controller-2\" update-client=\"cibadmin\" update-user=\"root\" have-quorum=\"1\" dc-uuid=\"2\">\n <configuration>\n <crm_config>
\n <cluster_property_set id=\"cib-bootstrap-options\">\n <nvpair id=\"cib-bootstrap-options-have-watchdog\" name=\"have-watchdog\" value=\"false\"
></nvpair>\n <nvpair id=\"cib-bootstrap-options-dc-version\" name=\"dc-version\" value=\"1.1.18-11.el7_5.3-2b07d5c5a9\"></nvpair>\n <nvpair id=\"cib-bootstrap-o
ptions-cluster-infrastructure\" name=\"cluster-infrastructure\" value=\"corosync\"></nvpair>\n <nvpair id=\"cib-bootstrap-options-cluster-name\" name=\"cluster
-name\" value=\"tripleo_cluster\"></nvpair>\n <nvpair id=\"cib-bootstrap-options-stonith-enabled\" name=\"stonith-enabled\" value=\"false\"></nvpair>\n <nvpair
id=\"cib-bootstrap-options-stonith-enabled\" name=\"stonith-enabled\" value=\"false\"></nvpair>\n </cluster_property_set>\n </crm_config>\n <nodes>\n
<node id=\"1\" uname=\"overcloud-controller-0\"></node>\n <node id=\"2\" uname=\"overcloud-controller-1\"></node>\n <node id=\"3\" uname=\"overcloud-controller-
2\"></node>\n </nodes>\n <resources></resources>\n <constraints></constraints>\n </configuration>\n <status>\n <node_state id=\"2\" uname=\"overcloud-controller-1\" in_ccm=\"true\" crmd=\"online\" crm-debug-origin=\"do_state_transition\" join=\"member\" expected=\"member\">\n <lrm id=\"2\">\n <lrm_resources></lrm_resources>\n </lrm>\n </node_state>\n <node_state id=\"3\" uname=\"overcloud-controller-2\" in_ccm=\"true\" crmd=\"online\" crm-debug-origin=\"do_state_transition\" join=\"member\" expected=\"member\">\n <lrm id=\"3\">\n <lrm_resources></lrm_resources>\n </lrm>\n </node_state>\n <node_state id=\"1\" uname=\"overcloud-controller-0\" in_ccm=\"true\" crmd=\"online\" crm-debug-origin=\"do_state_transition\" join=\"member\" expected=\"member\">\n <lrm id=\"1\">\n <lrm_resources></lrm_resources>\n </lrm>\n </node_state>\n </status>\n</cib>\n\n", "deploy_status_code": 0}
- heat stack-list --show-nested|grep -iv compl
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+
| id | stack_name | stack_status | creation_time | updated_time | parent |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+
| c7213b12-efc9-4783-8b14-00a2dc689b90 | overcloud | UPDATE_FAILED | 2018-07-13T19:37:09Z | 2018-07-13T19:41:09Z | None |
| 48115d3b-9947-4f48-92c9-f5a5a7fb61c7 | overcloud-AllNodesDeploySteps-3os4k5m2cdtq | CREATE_FAILED | 2018-07-13T20:01:20Z | None | c7213b12-efc9-4783-8b14-00a2dc689b90 |
| ecfdb082-8177-47ae-b3d4-77f3e7903e3a | overcloud-AllNodesDeploySteps-3os4k5m2cdtq-ControllerDeployment_Step2-fve2trl2ru2e | CREATE_FAILED | 2018-07-13T20:04:45Z | None | 48115d3b-9947-4f48-92c9-f5a5a7fb61c7 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+-----------------+----------------------+----------------------+--------------------------------------+
[stack@osp10-director- ~]$ heat resource-list ecfdb082-8177-47ae-b3d4-77f3e7903e3a
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+
| resource_name | physical_resource_id | resource_type | resource_status | updated_time |
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+
| 0 | ecaa85af-6c0f-49fa-8bcf-13207044934f | OS::Heat::StructuredDeployment | CREATE_FAILED | 2018-07-13T20:04:45Z |
| 1 | db28fbc6-6c3e-46ba-a264-af8d0a17a3f5 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2018-07-13T20:04:45Z |
| 2 | d2d5b52f-0bab-4a42-9280-2d73e9f1b84a | OS::Heat::StructuredDeployment | CREATE_FAILED | 2018-07-13T20:04:45Z |
+---------------+--------------------------------------+--------------------------------+-----------------+----------------------+
[stack@osp10-director- ~]$
heat deployment-show d2d5b52f-0bab-4a42-9280-2d73e9f1b84a
"deploy_stderr": "exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!\u001b[0m\n",
- corosync.log
IPaddr2(ip-10.12.143.201)[52045]: ERROR: Unable to find nic or netmask.
Environment
- Red Hat OpenStack Platform 10.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.