rabbitmq will not start after major upgrade from Red Hat OpenStack Platform 9 to Red Hat OpenStack Platform 10

Issue

rabbitmq will not start after major upgrade from Red Hat OpenStack Platform 9 to Red Hat OpenStack Platform 10

Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-1 ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0

Failed Actions:
* rabbitmq_start_0 on overcloud-controller-1 'unknown error' (1): call=288, status=complete, exitreason='none',
    last-rc-change='Wed Sep 13 07:51:26 2017', queued=0ms, exec=90484ms

From /var/log/messages:

Sep 13 07:52:37 overcloud-controller-1 rabbitmq-cluster(rabbitmq)[515936]: INFO: Attempting to join cluster with target node rabbit@overcloud-controller-0
Sep 13 07:52:37 overcloud-controller-1 su: (to rabbitmq) root on none
Sep 13 07:52:37 overcloud-controller-1 systemd: Started Session c3257 of user rabbitmq.
Sep 13 07:52:37 overcloud-controller-1 systemd: Starting Session c3257 of user rabbitmq.
Sep 13 07:52:39 overcloud-controller-1 kernel: IN=vlan200 OUT= MAC=01:00:5e:00:00:01:00:10:db:ff:00:00:00:01 SRC=192.168.0.254 DST=224.0.0.1 LEN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=9110 PROTO=2 
Sep 13 07:52:57 overcloud-controller-1 rabbitmq-cluster(rabbitmq)[515936]: INFO: Join process incomplete, shutting down.
Sep 13 07:52:57 overcloud-controller-1 rabbitmq-cluster(rabbitmq)[515936]: INFO: node failed to join even after reseting local data. Check SELINUX policy
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-1': nodedown ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ DIAGNOSTICS ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ =========== ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ attempted to contact: ['rabbit@overcloud-controller-1'] ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ rabbit@overcloud-controller-1: ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * connected to epmd (port 4369) on overcloud-controller-1 ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * epmd reports: node 'rabbit' not running at all ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [                   no other nodes on overcloud-controller-1 ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * suggestion: start the node ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ current node details: ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ - node name: 'rabbitmq-cli-22@overcloud-controller-1' ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ - home dir: /var/lib/rabbitmq ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ - cookie hash: yJFqtNjpAtppBLUgeE7Fcg== ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-2': nodedown ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ DIAGNOSTICS ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ =========== ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ attempted to contact: ['rabbit@overcloud-controller-2'] ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ rabbit@overcloud-controller-2: ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * connected to epmd (port 4369) on overcloud-controller-2 ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * epmd reports node 'rabbit' running on port 35672 ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * can't establish TCP connection, reason: timeout (timed out) ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [   * suggestion: blocked by firewall? ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ current node details: ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ - node name: 'rabbitmq-cli-42@overcloud-controller-1' ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ - home dir: /var/lib/rabbitmq ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ - cookie hash: yJFqtNjpAtppBLUgeE7Fcg== ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown ]
Sep 13 07:52:57 overcloud-controller-1 lrmd[3616]:  notice: rabbitmq_start_0:515936:stderr [  ]

Verification with ping and telnet yields:

Here are the results.  Unable to telnet to telnet overcloud-controller-2 35672

[heat-admin@overcloud-controller-1 ~]$ ping overcloud-controller-2
PING overcloud-controller-2.localdomain (192.168.1.18) 56(84) bytes of data.
64 bytes from overcloud-controller-2.localdomain (192.168.1.18): icmp_seq=1 ttl=64 time=0.074 ms
64 bytes from overcloud-controller-2.localdomain (192.168.1.18): icmp_seq=2 ttl=64 time=0.081 ms
64 bytes from overcloud-controller-2.localdomain (192.168.1.18): icmp_seq=3 ttl=64 time=0.097 ms
64 bytes from overcloud-controller-2.localdomain (192.168.1.18): icmp_seq=4 ttl=64 time=0.086 ms
^C
--- overcloud-controller-2.localdomain ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.074/0.084/0.097/0.012 ms
[heat-admin@overcloud-controller-1 ~]$ telnet overcloud-controller-2 4369
Trying 192.168.1.18...
Connected to overcloud-controller-2.
Escape character is '^]'.
^]
telnet> q
Connection closed.
[heat-admin@overcloud-controller-1 ~]$ telnet overcloud-controller-2 35672
Trying 192.168.1.18...
^C