overcloud deploy exit for Permission denied

Latest response

Environment
Red Hat OpenStack Platform 7.2

Issue
1)Heat stack-list stack_status is CREATE_COMPLETE
2)Nova/Neutron/Glance endpoints are not created
3)Overcloud deploys has only keystone endpoint
4)Overcloud endpoints not created on deploy
5)Overcloud deploy fails
6)Undercloud ping keystone publicURL success
7)Controller-0 mysql service start failes, but it can start manually

I deployed three controller nodes and three compute nodes, and I tryed four times always failed in the "Permission denied".
I think it is different with [1].

[1] (https://access.redhat.com/solutions/2444561)

[root@undercloud stack]# time openstack overcloud deploy \
> --templates templates/openstack-tripleo-heat-templates/ \
> -e templates/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
> -e templates/network-environment.yaml \
> --control-flavor control --compute-flavor compute \
> --control-scale 3 \
> --compute-scale 3 \
> --ceph-storage-scale 0 \
> --block-storage-scale 0 \
> --swift-storage-scale 0 \
> --ntp-server 192.168.191.253 \
> --neutron-network-type vlan \
> --neutron-bridge-mappings datacentre:br-ex \
> --neutron-network-vlan-ranges datacentre:200:1200 \
> --neutron-disable-tunneling \
> --timeout 90
Deploying templates in the directory /home/stack/templates/openstack-tripleo-heat-templates
/root/.ssh/known_hosts updated.
Original contents retained as /root/.ssh/known_hosts.old
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.168.191.57' (ECDSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
ERROR: openstack Command '['ssh', '-oStrictHostKeyChecking=no', '-t', '-l', 'heat-admin', u'192.168.191.57', 'sudo', 'keystone-manage', 'pki_setup', '--keystone-user', "$(getent passwd | grep '^keystone' | cut -d: -f1)", '--keystone-group', "$(getent group | grep '^keystone' | cut -d: -f1)"]' returned non-zero exit status 255

real    34m29.371s
user    0m2.375s
sys 0m0.287s
[root@undercloud stack]# heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 663a6941-f497-4542-92e7-2ffa625daff6 | overcloud  | CREATE_COMPLETE | 2016-12-12T12:35:33Z |
+--------------------------------------+------------+-----------------+----------------------+
[root@undercloud stack]# openstack catalog list
+----------+----------+-------------------------------------------------+
| Name     | Type     | Endpoints                                       |
+----------+----------+-------------------------------------------------+
| keystone | identity | regionOne                                       |
|          |          |   publicURL: http://192.168.192.151:5000/v2.0   |
|          |          |   internalURL: http://192.168.192.151:5000/v2.0 |
|          |          |   adminURL: http://192.168.192.151:5000/v2.0    |
|          |          |                                                 |
+----------+----------+-------------------------------------------------+
[root@undercloud stack]# ping 192.168.192.151
PING 192.168.192.151 (192.168.192.151) 56(84) bytes of data.
64 bytes from 192.168.192.151: icmp_seq=1 ttl=64 time=0.174 ms
64 bytes from 192.168.192.151: icmp_seq=2 ttl=64 time=0.166 ms
64 bytes from 192.168.192.151: icmp_seq=3 ttl=64 time=0.160 ms
64 bytes from 192.168.192.151: icmp_seq=4 ttl=64 time=0.156 ms
[root@undercloud stack]# nova list
ERROR (EndpointNotFound): publicURL endpoint for compute service not found
[root@undercloud stack]# glance image-list
public endpoint for image service not found
[root@undercloud stack]# neutron net-list
publicURL endpoint for network service not found

The undercloud node network:

[root@undercloud stack]# neutron net-list
+--------------------------------------+--------------+-------------------------------------------------------+
| id                                   | name         | subnets                                               |
+--------------------------------------+--------------+-------------------------------------------------------+
| 29921fa5-9c05-44d8-8f3d-c8c302c3cb63 | external     | 2898def8-2030-496b-80f1-9a1c11ece3a4 192.168.192.0/24 |
| 02f781ea-b38c-4bc6-91d2-388bcd0c8181 | storage_mgmt | 7763b44d-67a7-4a54-8e59-ff53a630556e 192.168.200.0/24 |
| 33f4d8a7-2995-454e-a7ca-0d9a04d746e5 | storage      | 322a964e-8dbe-4e65-a25b-0ca98f6b8c72 172.100.0.0/16   |
| 931564f1-2d41-4493-a898-95274a4b6c44 | tenant       | 8e5216a0-7cab-4410-8e47-cede6c104ecc 192.168.193.0/24 |
| 96dc121c-a22c-4b29-bc8c-70f2d68f9b66 | internal_api | 9ae7355b-e029-450d-a79d-662e60af5cb0 192.168.190.0/24 |
| f46a025c-6698-40c1-be08-64bc1944a2bd | ctlplane     | 920e01c1-ae01-4fd3-b093-892b3804b536 192.168.191.0/24 |
+--------------------------------------+--------------+-------------------------------------------------------+

The controller-0 os-collect-config is the ERRORs:

Dec 13 10:31:13 overcloud-controller-0.localdomain os-collect-config[4698]: haproxy/haproxy.cfg]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy::Config/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]: Triggered 'refresh' from 41 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy::Config/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: content changed '{md5}1f337186b0e1ba5ee82760cb437fb810' to '{md5}94f27aa67c234585323bf6b6746093bf'\u001b[0m\n\u001b[mNotice: /File[/etc/haproxy/haproxy.cfg]/seluser: seluser changed 'unconfined_u' to 'system_u'\u001b[0m\n\u001b[mNotice: Finished catalog run in 41.79 seconds\u001b[0m\n", "deploy_stderr": "Device \"br_ex\" does not exist.\nDevice \"ovs_system\" does not exist.\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0}
Dec 13 10:31:13 overcloud-controller-0.localdomain os-collect-config[4698]: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Dec 13 10:31:13 overcloud-controller-0.localdomain os-collect-config[4698]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Dec 13 10:33:07 overcloud-controller-0.localdomain os-collect-config[4698]: lic_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[redis_vip-then-haproxy]/Exec[Creating order constraint redis_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Base[storage_vip-then-haproxy]/Exec[Creating order constraint storage_vip-then-haproxy]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Pacemaker::Constraint::Colocation[public_vip-with-haproxy]/Pcmk_constraint[colo-ip-192.168.192.151-haproxy-clone]/ensure: created\u001b[0m\n\u001b[mNotice: Finished catalog run in 52.40 seconds\u001b[0m\n", "deploy_stderr": "Device \"br_ex\" does not exist.\nDevice \"ovs_system\" does not exist.\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0}
Dec 13 10:33:08 overcloud-controller-0.localdomain os-collect-config[4698]: Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Dec 13 10:33:08 overcloud-controller-0.localdomain os-collect-config[4698]: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

Responses

It seems you do the overcloud deployment task as the root user on the undercloud. All overcloud actions need to be done as the stack user on the undercloud. Please do the same as the stack user and verify if it works then.