After a fresh deployment seen issue on controllers : resources rabbitmq / galera / redis fails on one node at a time,
Issue
-
After a fresh deployment seen issue on controllers : resources rabbitmq / galera / redis fails on one node at a time,
Stack: corosync Current DC: ctr1-fr-east-1 (version 1.1.19-8.el7_6.2-c3c624ea3d) - partition with quorum Last updated: Mon Feb 4 11:30:07 2019 Last change: Mon Feb 4 10:37:29 2019 by hacluster via crmd on ctr1-fr-east-1 15 nodes configured 47 resources configured Online: [ ctr0-fr-east-1 ctr1-fr-east-1 ctr2-fr-east-1 ] GuestOnline: [ galera-bundle-0@ctr0-fr-east-1 galera-bundle-1@ctr1-fr-east-1 galera-bundle-2@ctr2-fr-east-1 ovn-dbs-bundle-0@ctr1-fr-east-1 ovn-dbs-bundle-1@ctr2-fr-east-1 ovn-dbs-bundle-2@ctr0-fr-east-1 rabbitmq-bundle-0@ctr0-fr-eas t-1 rabbitmq-bundle-1@ctr1-fr-east-1 rabbitmq-bundle-2@ctr2-fr-east-1 redis-bundle-0@ctr0-fr-east-1 redis-bundle-1@ctr1-fr-east-1 redis-bundle-2@ctr2-fr-east-1 ] Active resources: Docker container set: rabbitmq-bundle [xx.xx.0.1:8787/rhosp13/openstack-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started ctr0-fr-east-1 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): FAILED ctr1-fr-east-1 rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Started ctr2-fr-east-1 Docker container set: galera-bundle [xx.xx.0.1:8787/rhosp13/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master ctr0-fr-east-1 galera-bundle-1 (ocf::heartbeat:galera): Master ctr1-fr-east-1 galera-bundle-2 (ocf::heartbeat:galera): FAILED Master ctr2-fr-east-1 Docker container set: redis-bundle [xx.xx.0.1:8787/rhosp13/openstack-redis:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master ctr0-fr-east-1 redis-bundle-1 (ocf::heartbeat:redis): Slave ctr1-fr-east-1 redis-bundle-2 (ocf::heartbeat:redis): FAILED ctr2-fr-east-1 ip-xx.xx.0.10 (ocf::heartbeat:IPaddr2): Started ctr0-fr-east-1 ip-xx.xx.105.1 (ocf::heartbeat:IPaddr2): Started ctr1-fr-east-1 ip-xx.xx.255.251 (ocf::heartbeat:IPaddr2): Started ctr2-fr-east-1 ip-2a06.7d80..1.0.40 (ocf::heartbeat:IPaddr2): Started ctr0-fr-east-1 ip-xx.xx0.1 (ocf::heartbeat:IPaddr2): Started ctr1-fr-east-1 ip-xx.xx.0.1 (ocf::heartbeat:IPaddr2): Started ctr2-fr-east-1 ip-xx.xx.0.1 (ocf::heartbeat:IPaddr2): Started ctr0-fr-east-1 Docker container set: haproxy-bundle [xx.xx.0.1:8787/rhosp13/openstack-haproxy:pcmklatest] haproxy-bundle-docker-0 (ocf::heartbeat:docker): Started ctr1-fr-east-1 haproxy-bundle-docker-1 (ocf::heartbeat:docker): Started ctr2-fr-east-1 haproxy-bundle-docker-2 (ocf::heartbeat:docker): Started ctr0-fr-east-1 Docker container set: ovn-dbs-bundle [xx.xx.0.1:8787/rhosp13/openstack-ovn-northd:pcmklatest] ovn-dbs-bundle-0 (ocf::ovn:ovndb-servers): Master ctr1-fr-east-1 ovn-dbs-bundle-1 (ocf::ovn:ovndb-servers): FAILED ctr2-fr-east-1 ovn-dbs-bundle-2 (ocf::ovn:ovndb-servers): Slave ctr0-fr-east-1 Docker container: openstack-cinder-volume [xx.xx.0.1:8787/rhosp13/openstack-cinder-volume:pcmklatest] openstack-cinder-volume-docker-0 (ocf::heartbeat:docker): Started ctr1-fr-east-1 Failed Actions: * rabbitmq_monitor_10000 on rabbitmq-bundle-0 'unknown error' (1): call=182786, status=complete, exitreason='', last-rc-change='Mon Feb 4 11:29:46 2019', queued=0ms, exec=0ms * rabbitmq_monitor_10000 on rabbitmq-bundle-1 'unknown error' (1): call=182511, status=complete, exitreason='', last-rc-change='Mon Feb 4 11:30:05 2019', queued=0ms, exec=0ms * rabbitmq_monitor_0 on rabbitmq-bundle-2 'unknown error' (1): call=182769, status=complete, exitreason='', last-rc-change='Mon Feb 4 10:38:19 2019', queued=0ms, exec=4415ms * rabbitmq_monitor_10000 on rabbitmq-bundle-2 'unknown error' (1): call=182778, status=complete, exitreason='', last-rc-change='Mon Feb 4 11:29:25 2019', queued=0ms, exec=0ms * galera_monitor_10000 on galera-bundle-2 'unknown error' (1): call=-1, status=Timed Out, exitreason='', last-rc-change='Mon Feb 4 11:06:16 2019', queued=0ms, exec=0ms * redis_monitor_45000 on redis-bundle-2 'unknown error' (1): call=-1, status=Timed Out, exitreason='', last-rc-change='Mon Feb 4 11:06:46 2019', queued=0ms, exec=0ms * ovndb_servers_monitor_30000 on ovn-dbs-bundle-1 'unknown error' (1): call=-1, status=Timed Out, exitreason='', last-rc-change='Mon Feb 4 11:06:06 2019', queued=0ms, exec=0ms
Environment
- Red Hat OpenStack Platform Version 13.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.