Overcloud update failed on step_2 because of clustercheck container.
Issue
-
After FFU the stack update return with issue.
-
The following deploymen fails at step2:
2021-01-20 15:32:09,630 p=195345 u=mistral n=ansible | fatal: [overcloud-controller-1]: FAILED! => {"ansible_job_id": "513475148366.3257313", "attempts": 42, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2"
[...]
2021-01-20 15:34:18,527 p=195345 u=mistral n=ansible | PLAY RECAP *********************************************************************
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-controller-0 : ok=285 changed=116 unreachable=0 failed=0 skipped=216 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-controller-1 : ok=274 changed=116 unreachable=0 failed=1 skipped=214 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-controller-2 : ok=275 changed=116 unreachable=0 failed=0 skipped=214 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-5 : ok=212 changed=65 unreachable=0 failed=0 skipped=256 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-6 : ok=210 changed=65 unreachable=0 failed=0 skipped=256 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-7 : ok=210 changed=65 unreachable=0 failed=0 skipped=256 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-8 : ok=210 changed=65 unreachable=0 failed=0 skipped=256 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-9 : ok=181 changed=53 unreachable=0 failed=0 skipped=281 rescued=0 ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-0 : ok=176 changed=53 unreachable=0 failed=0 skipped=281 rescued=0 ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | overcloud-compute-1 : ok=176 changed=53 unreachable=0 failed=0 skipped=281 rescued=0 ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | overcloud-compute-2 : ok=176 changed=53 unreachable=0 failed=0 skipped=281 rescued=0 ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | overcloud-compute-3 : ok=176 changed=53 unreachable=0 failed=0 skipped=281 rescued=0 ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | undercloud : ok=99 changed=15 unreachable=0 failed=0 skipped=37 rescued=0 ignored=0
- Why the node overcloud-controller-1 has 2 containers from galera-bundle-podman?
(undercloud) [stack@overcloud-compute-1 ~]$ for I in $(openstack server list | grep control | awk '{print $8}' | cut -d '=' -f 2 ); do echo '########### NODE ###########' ; ssh -q heat-admin@$I "hostname -f ; sudo podman ps -a | grep maria"; done
########### NODE ###########
overcloud-controller-2.localdomain
91a424506e72 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /container_puppet... 2 hours ago Exited (0) 2 hours ago mysql_init_bundle
a40f05484974 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /bin/bash /usr/lo... 12 hours ago Up 12 hours ago galera-bundle-podman-1
4e71c5846566 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /pacemaker_restar... 4 days ago Exited (0) 4 days ago mysql_restart_bundle
abc67fd3cd78 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 kolla_start 4 days ago Up 15 hours ago clustercheck
0d440ff09b00 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 bash -ec if [ -e ... 4 days ago Exited (0) 4 days ago mysql_bootstrap
f7a091f9072e satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 chown -R mysql: /... 4 days ago Exited (0) 4 days ago mysql_data_ownership
########### NODE ###########
overcloud-controller-0.localdomain
1b85399234a4 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /container_puppet... 2 hours ago Exited (0) 2 hours ago mysql_init_bundle
80f0e8819959 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /bin/bash /usr/lo... 12 hours ago Up 12 hours ago galera-bundle-podman-2
5e4e110fa8a4 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /pacemaker_restar... 4 days ago Exited (0) 4 days ago mysql_restart_bundle
f9765811a35a satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 bash -ec if [ -e ... 4 days ago Exited (0) 4 days ago mysql_bootstrap
649d80c15a45 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 kolla_start 4 days ago Up 15 hours ago clustercheck
6c1a6bb7b68b satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 chown -R mysql: /... 4 days ago Exited (0) 4 days ago mysql_data_ownership
########### NODE ###########
overcloud-controller-1.localdomain
d5d43d528441 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /container_puppet... 2 hours ago Exited (0) 2 hours ago mysql_init_bundle
fa55529ec256 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /bin/bash /usr/lo... 11 hours ago Up 11 hours ago galera-bundle-podman-0
025ed15fd33b satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 kolla_start 13 hours ago Up 11 hours ago clustercheck
7e6d0ec38f03 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /bin/bash /usr/lo... 3 days ago Created galera-bundle-podman-1
a4559fd42e7a satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 /pacemaker_restar... 4 days ago Exited (0) 4 days ago mysql_restart_bundle
31dd4978de2d satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 bash -ec if [ -e ... 4 days ago Exited (0) 4 days ago mysql_bootstrap
910394a49b86 satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1 chown -R mysql: /... 4 days ago Exited (0) 4 days ago mysql_data_ownership
Environment
- Red Hat OpenStack Platform 16.1 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
