Overcloud update failed on step_2 because of clustercheck container.

Solution In Progress - Updated -

Issue

  • After FFU the stack update return with issue.

  • The following deploymen fails at step2:

2021-01-20 15:32:09,630 p=195345 u=mistral n=ansible | fatal: [overcloud-controller-1]: FAILED! => {"ansible_job_id": "513475148366.3257313", "attempts": 42, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2"
[...]
2021-01-20 15:34:18,527 p=195345 u=mistral n=ansible | PLAY RECAP *********************************************************************
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-controller-0                     : ok=285  changed=116  unreachable=0    failed=0    skipped=216  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-controller-1                     : ok=274  changed=116  unreachable=0    failed=1    skipped=214  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-controller-2                     : ok=275  changed=116  unreachable=0    failed=0    skipped=214  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-5                     : ok=212  changed=65   unreachable=0    failed=0    skipped=256  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-6                     : ok=210  changed=65   unreachable=0    failed=0    skipped=256  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-7                     : ok=210  changed=65   unreachable=0    failed=0    skipped=256  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-8                     : ok=210  changed=65   unreachable=0    failed=0    skipped=256  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-9                     : ok=181  changed=53   unreachable=0    failed=0    skipped=281  rescued=0    ignored=0
2021-01-20 15:34:18,528 p=195345 u=mistral n=ansible | overcloud-compute-0                     : ok=176  changed=53   unreachable=0    failed=0    skipped=281  rescued=0    ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | overcloud-compute-1                     : ok=176  changed=53   unreachable=0    failed=0    skipped=281  rescued=0    ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | overcloud-compute-2                     : ok=176  changed=53   unreachable=0    failed=0    skipped=281  rescued=0    ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | overcloud-compute-3                     : ok=176  changed=53   unreachable=0    failed=0    skipped=281  rescued=0    ignored=0
2021-01-20 15:34:18,529 p=195345 u=mistral n=ansible | undercloud                 : ok=99   changed=15   unreachable=0    failed=0    skipped=37   rescued=0    ignored=0
  • Why the node overcloud-controller-1 has 2 containers from galera-bundle-podman?
(undercloud) [stack@overcloud-compute-1 ~]$  for I in $(openstack server list | grep control | awk '{print $8}' | cut -d '=' -f 2 ); do  echo '########### NODE ###########' ; ssh -q heat-admin@$I "hostname -f ; sudo podman ps -a | grep maria"; done
########### NODE ###########
overcloud-controller-2.localdomain
91a424506e72  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /container_puppet...  2 hours ago   Exited (0) 2 hours ago          mysql_init_bundle
a40f05484974  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /bin/bash /usr/lo...  12 hours ago  Up 12 hours ago                 galera-bundle-podman-1
4e71c5846566  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /pacemaker_restar...  4 days ago    Exited (0) 4 days ago           mysql_restart_bundle
abc67fd3cd78  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     kolla_start           4 days ago    Up 15 hours ago                 clustercheck
0d440ff09b00  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     bash -ec if [ -e ...  4 days ago    Exited (0) 4 days ago           mysql_bootstrap
f7a091f9072e  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     chown -R mysql: /...  4 days ago    Exited (0) 4 days ago           mysql_data_ownership
########### NODE ###########
overcloud-controller-0.localdomain
1b85399234a4  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /container_puppet...  2 hours ago   Exited (0) 2 hours ago          mysql_init_bundle
80f0e8819959  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /bin/bash /usr/lo...  12 hours ago  Up 12 hours ago                 galera-bundle-podman-2
5e4e110fa8a4  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /pacemaker_restar...  4 days ago    Exited (0) 4 days ago           mysql_restart_bundle
f9765811a35a  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     bash -ec if [ -e ...  4 days ago    Exited (0) 4 days ago           mysql_bootstrap
649d80c15a45  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     kolla_start           4 days ago    Up 15 hours ago                 clustercheck
6c1a6bb7b68b  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     chown -R mysql: /...  4 days ago    Exited (0) 4 days ago           mysql_data_ownership
########### NODE ###########
overcloud-controller-1.localdomain
d5d43d528441  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /container_puppet...  2 hours ago   Exited (0) 2 hours ago          mysql_init_bundle
fa55529ec256  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /bin/bash /usr/lo...  11 hours ago  Up 11 hours ago                 galera-bundle-podman-0
025ed15fd33b  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     kolla_start           13 hours ago  Up 11 hours ago                 clustercheck
7e6d0ec38f03  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /bin/bash /usr/lo...  3 days ago    Created                         galera-bundle-podman-1
a4559fd42e7a  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     /pacemaker_restar...  4 days ago    Exited (0) 4 days ago           mysql_restart_bundle
31dd4978de2d  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     bash -ec if [ -e ...  4 days ago    Exited (0) 4 days ago           mysql_bootstrap
910394a49b86  satellite.localdomain:5000/rtm-osp16_1-mariadb:16.1                     chown -R mysql: /...  4 days ago    Exited (0) 4 days ago           mysql_data_ownership

Environment

  • Red Hat OpenStack Platform 16.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In