all-in-one OpenStack cloud deploy failed

Issue

• Due to COVID-19 caused server shortage so we adapted our deployment strategy and decided to start deploying all-in-ones.

• We followed the following documentation to create all-in-one cloud to support customer application services.

• The deployment is blocked at Step2:

2020-04-08 16:03:48,746 p=5037 u=root |  fatal: [allinone-rhosp16]: FAILED! => {"ansible_job_id": "523757146131.28717", "attempts": 640, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2", "rc": 6, "stderr": "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_api_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_api_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_scheduler_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_scheduler_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=horizon_fix_perms', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=horizon_fix_perms', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_restart_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_restart_bundle', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_compute_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_compute_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_conductor_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_conductor_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_metadata_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_metadata_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_restart_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_restart_bundle', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=clustercheck', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=clustercheck', '--format', '{{.Names}}']\"\nCreated symlink /etc/systemd/system/multi-user.target.wants/tripleo_clustercheck.service → /etc/systemd/system/tripleo_clustercheck.service.\nCreated symlink /etc/systemd/system/timers.target.wants/tripleo_clustercheck_healthcheck.timer → /etc/systemd/system/tripleo_clustercheck_healthcheck.timer.\nCreated symlink /etc/systemd/system/tripleo_clustercheck.service.requires/tripleo_clustercheck_healthcheck.timer → /etc/systemd/system/tripleo_clustercheck_healthcheck.timer.\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_haproxy_wrapper', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_haproxy_wrapper', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_init_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_init_bundle', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_init_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_init_bundle', '--format', '{{.Names}}']\"\nError running ['podman', 'run', '--name', 'rabbitmq_init_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=rabbitmq_init_bundle', '--label', 'managed_by=tripleo-Standalone', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"2\", \"file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,rabbitmq_policy,rabbitmq_user,rabbitmq_ready\", \"include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::rabbitmq_bundle\", \"\"], \"detach\": false, \"environment\": {\"KOLLA_BOOTSTRAP\": true, \"KOLLA_CONFIG_STRATEGY\": \"COPY_ALWAYS\", \"RABBITMQ_CLUSTER_COOKIE\": \"lv6cP2JSiXJfcxl9eNnk\", \"TRIPLEO_DEPLOY_IDENTIFIER\": \"1586376311\"}, \"image\": \"registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/bin/true:/bin/epmd\"]}', '--conmon-pidfile=/var/run/rabbitmq_init_bundle.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/rabbitmq_init_bundle.log', '--env=KOLLA_BOOTSTRAP=True', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=RABBITMQ_CLUSTER_COOKIE=lv6cP2JSiXJfcxl9eNnk', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1586376311', '--net=host', '--ipc=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/bin/true:/bin/epmd', '--cpuset-cpus=0,1,2,3,4,5,6,7,8,9,10,11,:

• Deployment failed with the following error:

  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1322, in _standalone_deploy
    raise exceptions.DeploymentError('Deployment failed')
tripleoclient.exceptions.DeploymentError: Deployment failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1340, in _standalone_deploy
    raise exceptions.DeploymentError(six.text_type(e))
tripleoclient.exceptions.DeploymentError: Deployment failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1443, in take_action
    if self._standalone_deploy(parsed_args) != 0:
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1377, in _standalone_deploy
    raise exceptions.DeploymentError('Deployment failed.')
tripleoclient.exceptions.DeploymentError: Deployment failed.

END return value: 1

• Only the following containers were started:

[stack@allinone-rhosp16 ~]$ sudo podman ps
CONTAINER ID  IMAGE                                                       COMMAND               CREATED      STATUS          PORTS  NAMES
7c6cd391559d  registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101  /bin/bash /usr/lo...  5 hours ago  Up 5 hours ago         rabbitmq-bundle-podman-0
28e05394753f  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101   /bin/bash /usr/lo...  5 hours ago  Up 5 hours ago         galera-bundle-podman-0
29c5c41aea44  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101   kolla_start           5 hours ago  Up 5 hours ago         clustercheck
8ca19c67c2fb  registry.redhat.io/rhosp-rhel8/openstack-memcached:16.0-99  /bin/bash -c sour...  5 hours ago  Up 5 hours ago         memcached

• rabbitmq_init_bundle exited with 6:

[stack@allinone-rhosp16 ~]$ sudo podman ps -a
CONTAINER ID  IMAGE                                                                        COMMAND               CREATED      STATUS                  PORTS  NAMES
7c6cd391559d  registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101                   /bin/bash /usr/lo...  5 hours ago  Up 5 hours ago                 rabbitmq-bundle-podman-0
e8ac02e92ecf  registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101                   /container_puppet...  5 hours ago  Exited (6) 4 hours ago         rabbitmq_init_bundle
33f74e4121d6  registry.redhat.io/rhosp-rhel8/openstack-placement-api:16.0-95               /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         placement_init_log
28e05394753f  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101                    /bin/bash /usr/lo...  5 hours ago  Up 5 hours ago                 galera-bundle-podman-0
5f6617aaa6d9  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101                    /container_puppet...  5 hours ago  Exited (0) 5 hours ago         mysql_init_bundle
7b6e23f18389  registry.redhat.io/rhosp-rhel8/openstack-keystone:16.0-93                    /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         keystone_init_log
ccb99eaaefbf  registry.redhat.io/rhosp-rhel8/openstack-neutron-metadata-agent-ovn:16.0-94  /container_puppet...  5 hours ago  Exited (0) 5 hours ago         create_haproxy_wrapper
29c5c41aea44  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101                    kolla_start           5 hours ago  Up 5 hours ago                 clustercheck
433356575a55  registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101                   /pacemaker_restar...  5 hours ago  Exited (0) 5 hours ago         rabbitmq_restart_bundle
bccb1e0f9c1b  registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.0-94                    /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         nova_metadata_init_logs
51df904cdfce  registry.redhat.io/rhosp-rhel8/openstack-nova-conductor:16.0-93              /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         nova_conductor_init_log
bc7be01a68aa  registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.0-94                /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         nova_compute_init_log
01845dc13021  registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.0-94                    /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         nova_api_init_logs
00befbb5d44f  registry.redhat.io/rhosp-rhel8/openstack-neutron-server-ovn:16.0-90          /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         neutron_init_logs
b3527d0ca173  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101                    /pacemaker_restar...  5 hours ago  Exited (0) 5 hours ago         mysql_restart_bundle
1459c3e8ddc7  registry.redhat.io/rhosp-rhel8/openstack-horizon:16.0-96                     /bin/bash -c touc...  5 hours ago  Exited (0) 5 hours ago         horizon_fix_perms
92e9c576e8e1  registry.redhat.io/rhosp-rhel8/openstack-glance-api:16.0-91                  /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         glance_init_logs
286d087f291f  registry.redhat.io/rhosp-rhel8/openstack-cinder-scheduler:16.0-92            /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         cinder_scheduler_init_logs
80ae0965efd3  registry.redhat.io/rhosp-rhel8/openstack-cinder-api:16.0-90                  /bin/bash -c chow...  5 hours ago  Exited (0) 5 hours ago         cinder_api_init_logs
7376a9fb2cc1  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101                    bash -ec if [ -e ...  5 hours ago  Exited (0) 5 hours ago         mysql_bootstrap
4e86e02df2e2  registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101                   kolla_start           5 hours ago  Exited (0) 5 hours ago         rabbitmq_bootstrap
ad9e6a37feb9  registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101                    chown -R mysql: /...  5 hours ago  Exited (0) 5 hours ago         mysql_data_ownership
8ca19c67c2fb  registry.redhat.io/rhosp-rhel8/openstack-memcached:16.0-99                   /bin/bash -c sour...  5 hours ago  Up 5 hours ago                 memcached