ERROR configuring keystone_init_tasks - RHOSP16

RHOSP 16 deployment all virtual environment failing where details are as below:

Description of problem:

overcloud deployment failed with subject error at step_3

Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates.noarch 11.3.2-0.20200211065546.d3d6dc3.el8ost @openstack-16-for-rhel-8-x86_64-rpms

How reproducible:

fresh overcloud deployment in all virtual nodes set on KVM

Steps to Reproduce:
1. followed director installation steps [1]
2. enable templates for deployment
3. run overcloud deployment which fails at step_3 as mentioned above for HA or non-HA both attempts

Actual results:

where the ansible.log shows following repeated failures/ERRORs:

ERROR: 51441 -- ERROR configuring keystone_init_tasks", "stdout_lines": ["2020-04-03 21:58:10,548 INFO: 51441 -- Running container-puppet", "2020-04-03 21:58:10,548 IN
FO: 51441 -- Service compilation completed.", "2020-04-03 21:58:10,549 INFO: 51441 -- Starting multiprocess configuration steps. Using 32 processes.", "2020-04-03 21:58:10,611 INFO: 51459 --
Starting configuration of keystone_init_tasks using image undercloud.ctlplane.local.domain:8787/rhosp-rhel8/openstack-keystone:16.0-82", "2020-04-03 21:58:10,708 INFO: 51459 -- Removing containe
r: container-puppet-keystone_init_tasks", "2020-04-03 21:58:11,039 INFO: 51459 -- Image already exists: undercloud.ctlplane.local.domain:8787/rhosp-rhel8/openstack-keystone:16.0-82", "2020-04-03
22:04:11,281 ERROR: 51459 -- ['/usr/bin/podman', 'run', '--user', '0', '--name', 'container-puppet-keystone_init_tasks', '--env', 'PUPPET_TAGS=file,file_line,concat,augeas,cron,keystone_confi
g,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain', '--env', 'NAME=keystone
_init_tasks', '--env', 'HOSTNAME=devpod-controller-00', '--env', 'NO_ARCHIVE=true', '--env', 'STEP=3', '--env', 'NET_HOST=true', '--env', 'DEBUG=False', '--volume', '/etc/localtime:/etc/local
time:ro', '--volume', '/tmp/tmpmqkkacfm:/etc/config.pp:ro', '--volume', '/etc/puppet/:/tmp/puppet-etc/:ro', '--volume', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume
', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume', '/etc/pk
i/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume', '/var/lib/config-data:/var/lib/config-data/:rw', '--volume', '/var/lib/container-puppet/puppetlabs/facter.conf:/etc/puppetlabs/facter/fac
ter.conf:ro', '--volume', '/var/lib/container-puppet/puppetlabs/:/opt/puppetlabs/:ro', '--volume', '/dev/log:/dev/log:rw', '--rm', '--log-driver', 'k8s-file', '--log-opt', 'path=/var/log/cont
ainers/stdouts/container-puppet-keystone_init_tasks.log', '--security-opt', 'label=disable', '--volume', '/usr/share/openstack-puppet/modules/:/usr/share/openstack-puppet/modules/:ro', '--ent
rypoint', '/var/lib/container-puppet/container-puppet.sh', '--net', 'host', '--volume', '/etc/hosts:/etc/hosts:ro', '--volume', '/var/lib/container-puppet/container-puppet.sh:/var/lib/contain
er-puppet/container-puppet.sh:ro', 'undercloud.ctlplane.local.domain:8787/rhosp-rhel8/openstack-keystone:16.0-82'] run failed after + mkdir -p /etc/puppet", "+ cp -dR /tmp/puppet-etc/auth.conf /t
mp/puppet-etc/hieradata /tmp/puppet-etc/hiera.yaml /tmp/puppet-etc/modules /tmp/puppet-etc/puppet.conf /tmp/puppet-etc/ssl /etc/puppet", "+ rm -Rf /etc/puppet/ssl", "+ echo '{\"step\": 3}'",
"+ TAGS=", "+ '[' -n file,file_line,concat,augeas,cron,keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_role,keystone_service,keystone_tenant,keyst
one_user,keystone_user_role,keystone_domain ']'", "+ TAGS='--tags \"file,file_line,concat,augeas,cron,keystone_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keyst
one_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain\"'", "+ CHECK_MODE=", "+ '[' -d /tmp/puppet-check-mode ']'", "+ conf_data_path=/var/lib/config-data/
keystone_init_tasks", "+ origin_of_time=/var/lib/config-data/keystone_init_tasks.origin_of_time", "+ touch /var/lib/config-data/keystone_init_tasks.origin_of_time", "+ sync", "+ export NET_HO
ST=true", "+ NET_HOST=true", "+ set +e", "+ '[' true == false ']'", "+ export FACTER_deployment_type=containers", "+ FACTER_deployment_type=containers", "++ cat /sys/class/dmi/id/product_uuid", "++ tr '[:upper:]' '[:lower:]'", "+ export FACTER_uuid=2845f2fb-36fa-4721-bfc9-c21d6a823a3e", "+ FACTER_uuid=2845f2fb-36fa-4721-bfc9-c21d6a823a3e", "+ echo 'Running puppet'", "+ set -x", "
+ /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '\"file,file_line,concat,augeas,cron,keyston
e_config,keystone_domain_config,keystone_endpoint,keystone_identity_provider,keystone_role,keystone_service,keystone_tenant,keystone_user,keystone_user_role,keystone_domain\"' /etc/config.pp"
, "+ logger -s -t puppet-user", "<13>Apr 3 21:58:11 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5", "<13>Apr
3 21:58:19 puppet-user: (file: /etc/puppet/hiera.yaml)", "<13>Apr 3 21:58:19 puppet-user: Warning: Undefined variable '::deploy_config_name'; \
(file & line not available)", "<13>Apr 3 21:58:19 puppet-user: Warning: ModuleLoader: module 'tripleo' has unresolved dependencies - it will only see those that are resolved. Use 'puppet
module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:19 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/5.5/deprecated_la
nguage.html\
(file & line not available)", "<13>Apr 3 21:58:19 puppet-user: Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'pu
ppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:19 puppet-user: Warning: ModuleLoader: module 'keystone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet
module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:20 puppet-user: Warning: Scope(Class[Keystone]): keystone::admin_port is deprecated, has not effect and will be removed in a later relase",
"<13>Apr 3 21:58:20 puppet-user: Notice: Accepting previously invalid value for target type 'Enum['sql', 'template']'", "<13>Apr 3 21:58:20 puppet-user: Warning: ModuleLoader: module 'oslo'
has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:20 puppet-user: Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at
https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at [\"/etc/puppet/modules/keystone/manifests/init.pp\", 873]:[\"/etc/config.pp\", 2]", "<13>Apr 3 21:58
:20 puppet-user: (location: /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:34:in `deprecation')", "<13>Apr 3 21:58:20 puppet-user: Warning: ModuleLoader: module 'concat' h
as unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:20 puppet-user: Warning: ModuleLoader: module 'cinder' has unresolved dependencies - it will only see those that are resolved. Use 'puppet m
odule list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:20 puppet-user: Warning: ModuleLoader: module 'glance' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:20 puppet-user: Warning: ModuleLoader: module 'heat' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:20 puppet-user: Warning: ModuleLoader: module 'neutron' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:21 puppet-user: Warning: ModuleLoader: module 'nova' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:21 puppet-user: Warning: ModuleLoader: module 'placement' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:21 puppet-user: Warning: ModuleLoader: module 'swift' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)", "<13>Apr 3 21:58:21 puppet-user: Warning: ModuleLoader: module 'mysql' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules\
(file & line not available)"

Expected results:

should be a successful deployment.

Additional info:

following answer file for templates called:

templates: /home/stack/templates
environments:
- /home/stack/templates/node-info.yaml
- /home/stack/templates/containers-prepare-parameter.yaml
- /home/stack/templates/environments/network-isolation.yaml
- /home/stack/templates/environments/enable-swap.yaml
- /home/stack/templates/network-environment.yaml
- /home/stack/templates/environments/ceph-ansible/ceph-ansible.yaml
- /home/stack/templates/domain.yaml

whereas the deployment command is:

time openstack overcloud deploy --answers-file /home/stack/templates/answers.yaml -r /home/stack/templates/roles_data.yaml --stack devstack

parameter_defaults:
ContainerImagePrepare:
- push_destination: true
set:
ceph_alertmanager_image: ose-prometheus-alertmanager
ceph_alertmanager_namespace: registry.redhat.io/openshift4
ceph_alertmanager_tag: 4.1
ceph_grafana_image: rhceph-3-dashboard-rhel7
ceph_grafana_namespace: registry.redhat.io/rhceph
ceph_grafana_tag: 3
ceph_image: rhceph-4-rhel8
ceph_namespace: registry.redhat.io/rhceph
ceph_node_exporter_image: ose-prometheus-node-exporter
ceph_node_exporter_namespace: registry.redhat.io/openshift4
ceph_node_exporter_tag: v4.1
ceph_prometheus_image: ose-prometheus
ceph_prometheus_namespace: registry.redhat.io/openshift4
ceph_prometheus_tag: 4.1
ceph_tag: latest
name_prefix: openstack-
name_suffix: ''
namespace: registry.redhat.io/rhosp-rhel8
neutron_driver: ovn
rhel_containers: false
tag: '16.0'
tag_from_label: '{version}-{release}'

Additional controller logs showing errors:

/var/log/containers/stdouts/container-puppet-rabbitmq.log:29:2020-04-03T21:46:58.464479203+03:00 stderr F <13>Apr 3 21:46:52 puppet-user: Error: Facter: error while resolving custom fact "rabbitmq_nodename": undefined method `[]' for nil:NilClass
/var/log/containers/stdouts/container-puppet-swift.log:80:2020-04-03T21:47:26.420690858+03:00 stderr F <13>Apr 3 21:47:26 puppet-user: Notice: /Stage[main]/Swift::Proxy/Swift_proxy_config[pipeline:main/pipeline]/value: value changed catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server to catch_errors healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken s3api s3token keystone staticweb copy container_quotas account_quotas slo dlo versioned_writes proxy-logging proxy-server
/var/log/containers/stdouts/container-puppet-keystone_init_tasks.log:154:2020-04-03T22:04:07.592994717+03:00 stderr F <13>Apr 3 22:04:07 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://10.101.40.10:5000/v3/domains?: HTTPConnectionPool(host='10.101.40.10', port=5000): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f68642dd748>: Failed to establish a new connection: [Errno 113] No route to host',)) (tried 23, for a total of 170 seconds)
/var/log/containers/stdouts/container-puppet-keystone_init_tasks.log:336:2020-04-03T22:10:08.709676095+03:00 stderr F <13>Apr 3 22:10:08 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://10.101.40.10:5000/v3/domains?: HTTPConnectionPool(host='10.101.40.10', port=5000): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8b8a4d67f0>: Failed to establish a new connection: [Errno 113] No route to host',)) (tried 23, for a total of 170 seconds)
/var/log/containers/stdouts/container-puppet-keystone_init_tasks.log:518:2020-04-03T22:16:09.344888077+03:00 stderr F <13>Apr 3 22:16:09 puppet-user: Error: Failed to apply catalog: Execution of '/usr/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://10.101.40.10:5000/v3/domains?: HTTPConnectionPool(host='10.101.40.10', port=5000): Max retries exceeded with url: /v3/domains (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcfc8e437f0>: Failed to establish a new connection: [Errno 113] No route to host',)) (tried 23, for a total of 170 seconds)

Where as no container is running on that system.

Details can be viewed here; https://bugzilla.redhat.com/show_bug.cgi?id=1820942 as well