After a fresh deployment seen issue on controllers : resources rabbitmq / galera / redis fails on one node at a time,

Solution In Progress - Updated -

Issue

  • After a fresh deployment seen issue on controllers : resources rabbitmq / galera / redis fails on one node at a time,

    Stack: corosync
    Current DC: ctr1-fr-east-1 (version 1.1.19-8.el7_6.2-c3c624ea3d) - partition with quorum
    Last updated: Mon Feb  4 11:30:07 2019
    Last change: Mon Feb  4 10:37:29 2019 by hacluster via crmd on ctr1-fr-east-1
    
    15 nodes configured
    47 resources configured
    
    Online: [ ctr0-fr-east-1 ctr1-fr-east-1 ctr2-fr-east-1 ]
    GuestOnline: [ galera-bundle-0@ctr0-fr-east-1 galera-bundle-1@ctr1-fr-east-1 galera-bundle-2@ctr2-fr-east-1 ovn-dbs-bundle-0@ctr1-fr-east-1 ovn-dbs-bundle-1@ctr2-fr-east-1 ovn-dbs-bundle-2@ctr0-fr-east-1 rabbitmq-bundle-0@ctr0-fr-eas
    t-1 rabbitmq-bundle-1@ctr1-fr-east-1 rabbitmq-bundle-2@ctr2-fr-east-1 redis-bundle-0@ctr0-fr-east-1 redis-bundle-1@ctr1-fr-east-1 redis-bundle-2@ctr2-fr-east-1 ]
    
    Active resources:
    
    Docker container set: rabbitmq-bundle [xx.xx.0.1:8787/rhosp13/openstack-rabbitmq:pcmklatest]
     rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started ctr0-fr-east-1
     rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      FAILED ctr1-fr-east-1
     rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started ctr2-fr-east-1
    Docker container set: galera-bundle [xx.xx.0.1:8787/rhosp13/openstack-mariadb:pcmklatest]
     galera-bundle-0      (ocf::heartbeat:galera):        Master ctr0-fr-east-1
     galera-bundle-1      (ocf::heartbeat:galera):        Master ctr1-fr-east-1
     galera-bundle-2      (ocf::heartbeat:galera):        FAILED Master ctr2-fr-east-1
    Docker container set: redis-bundle [xx.xx.0.1:8787/rhosp13/openstack-redis:pcmklatest]
     redis-bundle-0       (ocf::heartbeat:redis): Master ctr0-fr-east-1
     redis-bundle-1       (ocf::heartbeat:redis): Slave ctr1-fr-east-1
     redis-bundle-2       (ocf::heartbeat:redis): FAILED ctr2-fr-east-1
    ip-xx.xx.0.10  (ocf::heartbeat:IPaddr2):       Started ctr0-fr-east-1
    ip-xx.xx.105.1        (ocf::heartbeat:IPaddr2):       Started ctr1-fr-east-1
    ip-xx.xx.255.251        (ocf::heartbeat:IPaddr2):       Started ctr2-fr-east-1
    ip-2a06.7d80..1.0.40    (ocf::heartbeat:IPaddr2):       Started ctr0-fr-east-1
    ip-xx.xx0.1    (ocf::heartbeat:IPaddr2):       Started ctr1-fr-east-1
    ip-xx.xx.0.1    (ocf::heartbeat:IPaddr2):       Started ctr2-fr-east-1
    ip-xx.xx.0.1    (ocf::heartbeat:IPaddr2):       Started ctr0-fr-east-1
    Docker container set: haproxy-bundle [xx.xx.0.1:8787/rhosp13/openstack-haproxy:pcmklatest]
     haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started ctr1-fr-east-1
     haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started ctr2-fr-east-1
     haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started ctr0-fr-east-1
    Docker container set: ovn-dbs-bundle [xx.xx.0.1:8787/rhosp13/openstack-ovn-northd:pcmklatest]
     ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Master ctr1-fr-east-1
     ovn-dbs-bundle-1     (ocf::ovn:ovndb-servers):       FAILED ctr2-fr-east-1
     ovn-dbs-bundle-2     (ocf::ovn:ovndb-servers):       Slave ctr0-fr-east-1
    Docker container: openstack-cinder-volume [xx.xx.0.1:8787/rhosp13/openstack-cinder-volume:pcmklatest]
     openstack-cinder-volume-docker-0     (ocf::heartbeat:docker):        Started ctr1-fr-east-1
    
    Failed Actions:
    * rabbitmq_monitor_10000 on rabbitmq-bundle-0 'unknown error' (1): call=182786, status=complete, exitreason='',
      last-rc-change='Mon Feb  4 11:29:46 2019', queued=0ms, exec=0ms
    * rabbitmq_monitor_10000 on rabbitmq-bundle-1 'unknown error' (1): call=182511, status=complete, exitreason='',
      last-rc-change='Mon Feb  4 11:30:05 2019', queued=0ms, exec=0ms
    * rabbitmq_monitor_0 on rabbitmq-bundle-2 'unknown error' (1): call=182769, status=complete, exitreason='',
      last-rc-change='Mon Feb  4 10:38:19 2019', queued=0ms, exec=4415ms
    * rabbitmq_monitor_10000 on rabbitmq-bundle-2 'unknown error' (1): call=182778, status=complete, exitreason='',
      last-rc-change='Mon Feb  4 11:29:25 2019', queued=0ms, exec=0ms
    * galera_monitor_10000 on galera-bundle-2 'unknown error' (1): call=-1, status=Timed Out, exitreason='',
      last-rc-change='Mon Feb  4 11:06:16 2019', queued=0ms, exec=0ms
    * redis_monitor_45000 on redis-bundle-2 'unknown error' (1): call=-1, status=Timed Out, exitreason='',
      last-rc-change='Mon Feb  4 11:06:46 2019', queued=0ms, exec=0ms
    * ovndb_servers_monitor_30000 on ovn-dbs-bundle-1 'unknown error' (1): call=-1, status=Timed Out, exitreason='',
      last-rc-change='Mon Feb  4 11:06:06 2019', queued=0ms, exec=0ms
    

Environment

  • Red Hat OpenStack Platform Version 13.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content