all-in-one OpenStack cloud deploy failed
Issue
-
Due to COVID-19 caused server shortage so we adapted our deployment strategy and decided to start deploying all-in-ones.
-
We followed the following documentation to create all-in-one cloud to support customer application services.
-
The deployment is blocked at Step2:
2020-04-08 16:03:48,746 p=5037 u=root | fatal: [allinone-rhosp16]: FAILED! => {"ansible_job_id": "523757146131.28717", "attempts": 640, "changed": false, "finished": 1, "msg": "Paunch failed with config_id tripleo_step2", "rc": 6, "stderr": "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_api_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_api_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_scheduler_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=cinder_scheduler_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=glance_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=horizon_fix_perms', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=horizon_fix_perms', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_restart_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_restart_bundle', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=neutron_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_api_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_compute_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_compute_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_conductor_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_conductor_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_metadata_init_logs', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_metadata_init_logs', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_restart_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_restart_bundle', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=clustercheck', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=clustercheck', '--format', '{{.Names}}']\"\nCreated symlink /etc/systemd/system/multi-user.target.wants/tripleo_clustercheck.service → /etc/systemd/system/tripleo_clustercheck.service.\nCreated symlink /etc/systemd/system/timers.target.wants/tripleo_clustercheck_healthcheck.timer → /etc/systemd/system/tripleo_clustercheck_healthcheck.timer.\nCreated symlink /etc/systemd/system/tripleo_clustercheck.service.requires/tripleo_clustercheck_healthcheck.timer → /etc/systemd/system/tripleo_clustercheck_healthcheck.timer.\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_haproxy_wrapper', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_haproxy_wrapper', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=keystone_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_init_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=mysql_init_bundle', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=placement_init_log', '--format', '{{.Names}}']\"\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_init_bundle', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id\nDid not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=rabbitmq_init_bundle', '--format', '{{.Names}}']\"\nError running ['podman', 'run', '--name', 'rabbitmq_init_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=rabbitmq_init_bundle', '--label', 'managed_by=tripleo-Standalone', '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"2\", \"file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation,rabbitmq_policy,rabbitmq_user,rabbitmq_ready\", \"include ::tripleo::profile::base::pacemaker;include ::tripleo::profile::pacemaker::rabbitmq_bundle\", \"\"], \"detach\": false, \"environment\": {\"KOLLA_BOOTSTRAP\": true, \"KOLLA_CONFIG_STRATEGY\": \"COPY_ALWAYS\", \"RABBITMQ_CLUSTER_COOKIE\": \"lv6cP2JSiXJfcxl9eNnk\", \"TRIPLEO_DEPLOY_IDENTIFIER\": \"1586376311\"}, \"image\": \"registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101\", \"ipc\": \"host\", \"net\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/bin/true:/bin/epmd\"]}', '--conmon-pidfile=/var/run/rabbitmq_init_bundle.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/containers/stdouts/rabbitmq_init_bundle.log', '--env=KOLLA_BOOTSTRAP=True', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=RABBITMQ_CLUSTER_COOKIE=lv6cP2JSiXJfcxl9eNnk', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1586376311', '--net=host', '--ipc=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/bin/true:/bin/epmd', '--cpuset-cpus=0,1,2,3,4,5,6,7,8,9,10,11,:
- Deployment failed with the following error:
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1322, in _standalone_deploy
raise exceptions.DeploymentError('Deployment failed')
tripleoclient.exceptions.DeploymentError: Deployment failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1340, in _standalone_deploy
raise exceptions.DeploymentError(six.text_type(e))
tripleoclient.exceptions.DeploymentError: Deployment failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1443, in take_action
if self._standalone_deploy(parsed_args) != 0:
File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1377, in _standalone_deploy
raise exceptions.DeploymentError('Deployment failed.')
tripleoclient.exceptions.DeploymentError: Deployment failed.
END return value: 1
- Only the following containers were started:
[stack@allinone-rhosp16 ~]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c6cd391559d registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101 /bin/bash /usr/lo... 5 hours ago Up 5 hours ago rabbitmq-bundle-podman-0
28e05394753f registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 /bin/bash /usr/lo... 5 hours ago Up 5 hours ago galera-bundle-podman-0
29c5c41aea44 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 kolla_start 5 hours ago Up 5 hours ago clustercheck
8ca19c67c2fb registry.redhat.io/rhosp-rhel8/openstack-memcached:16.0-99 /bin/bash -c sour... 5 hours ago Up 5 hours ago memcached
rabbitmq_init_bundle
exited with 6:
[stack@allinone-rhosp16 ~]$ sudo podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c6cd391559d registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101 /bin/bash /usr/lo... 5 hours ago Up 5 hours ago rabbitmq-bundle-podman-0
e8ac02e92ecf registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101 /container_puppet... 5 hours ago Exited (6) 4 hours ago rabbitmq_init_bundle
33f74e4121d6 registry.redhat.io/rhosp-rhel8/openstack-placement-api:16.0-95 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago placement_init_log
28e05394753f registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 /bin/bash /usr/lo... 5 hours ago Up 5 hours ago galera-bundle-podman-0
5f6617aaa6d9 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 /container_puppet... 5 hours ago Exited (0) 5 hours ago mysql_init_bundle
7b6e23f18389 registry.redhat.io/rhosp-rhel8/openstack-keystone:16.0-93 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago keystone_init_log
ccb99eaaefbf registry.redhat.io/rhosp-rhel8/openstack-neutron-metadata-agent-ovn:16.0-94 /container_puppet... 5 hours ago Exited (0) 5 hours ago create_haproxy_wrapper
29c5c41aea44 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 kolla_start 5 hours ago Up 5 hours ago clustercheck
433356575a55 registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101 /pacemaker_restar... 5 hours ago Exited (0) 5 hours ago rabbitmq_restart_bundle
bccb1e0f9c1b registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.0-94 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago nova_metadata_init_logs
51df904cdfce registry.redhat.io/rhosp-rhel8/openstack-nova-conductor:16.0-93 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago nova_conductor_init_log
bc7be01a68aa registry.redhat.io/rhosp-rhel8/openstack-nova-compute:16.0-94 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago nova_compute_init_log
01845dc13021 registry.redhat.io/rhosp-rhel8/openstack-nova-api:16.0-94 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago nova_api_init_logs
00befbb5d44f registry.redhat.io/rhosp-rhel8/openstack-neutron-server-ovn:16.0-90 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago neutron_init_logs
b3527d0ca173 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 /pacemaker_restar... 5 hours ago Exited (0) 5 hours ago mysql_restart_bundle
1459c3e8ddc7 registry.redhat.io/rhosp-rhel8/openstack-horizon:16.0-96 /bin/bash -c touc... 5 hours ago Exited (0) 5 hours ago horizon_fix_perms
92e9c576e8e1 registry.redhat.io/rhosp-rhel8/openstack-glance-api:16.0-91 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago glance_init_logs
286d087f291f registry.redhat.io/rhosp-rhel8/openstack-cinder-scheduler:16.0-92 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago cinder_scheduler_init_logs
80ae0965efd3 registry.redhat.io/rhosp-rhel8/openstack-cinder-api:16.0-90 /bin/bash -c chow... 5 hours ago Exited (0) 5 hours ago cinder_api_init_logs
7376a9fb2cc1 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 bash -ec if [ -e ... 5 hours ago Exited (0) 5 hours ago mysql_bootstrap
4e86e02df2e2 registry.redhat.io/rhosp-rhel8/openstack-rabbitmq:16.0-101 kolla_start 5 hours ago Exited (0) 5 hours ago rabbitmq_bootstrap
ad9e6a37feb9 registry.redhat.io/rhosp-rhel8/openstack-mariadb:16.0-101 chown -R mysql: /... 5 hours ago Exited (0) 5 hours ago mysql_data_ownership
8ca19c67c2fb registry.redhat.io/rhosp-rhel8/openstack-memcached:16.0-99 /bin/bash -c sour... 5 hours ago Up 5 hours ago memcached
Environment
- Red Hat OpenStack Platform 16.0 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.