OpenStack Galera will not start with a failed Controller node and galera-bundle resources in Slave

Solution Verified - Updated -

Issue

  • One controller is down due to H/W failure or File Corruption
  • This causes Galera not to start on the other two controller nodes and remains in Slave state

    12 nodes configured
    40 resource instances configured
    
    Online: [ controller-1 controller-2 ]
    OFFLINE: [ controller-0 ]                   <<------
    GuestOnline: [ galera-bundle-0@controller-2 galera-bundle-1@controller-1 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ]
    
    Full list of resources:
    
    Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp13/openstack-rabbitmq:pcmklatest]
    rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Stopped
    rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started controller-1
    rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started controller-2
    Docker container set: galera-bundle [192.168.24.1:8787/rhosp13/openstack-mariadb:pcmklatest]
    galera-bundle-0      (ocf::heartbeat:galera):        Slave controller-2
    galera-bundle-1      (ocf::heartbeat:galera):        Slave controller-1
    galera-bundle-2      (ocf::heartbeat:galera):        Stopped
    Docker container set: redis-bundle [192.168.24.1:8787/rhosp13/openstack-redis:pcmklatest]
    redis-bundle-0       (ocf::heartbeat:redis): Stopped
    redis-bundle-1       (ocf::heartbeat:redis): Master controller-1
    redis-bundle-2       (ocf::heartbeat:redis): Slave controller-2
    ip-192.x.x.x       (ocf::heartbeat:IPaddr2):       Started controller-2
    ip-172.x.x.x       (ocf::heartbeat:IPaddr2):       Started controller-1
    ip-172.x.x.x       (ocf::heartbeat:IPaddr2):       Started controller-2
    ip-172.x.x.x       (ocf::heartbeat:IPaddr2):       Started controller-1
    ip-172.x.x.x       (ocf::heartbeat:IPaddr2):       Started controller-1
    ip-172.x.x.x       (ocf::heartbeat:IPaddr2):       Started controller-2
    Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest]
    haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Stopped
    haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started controller-1
    haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started controller-2
    Docker container: openstack-cinder-volume [192.168.24.1:8787/rhosp13/openstack-cinder-volume:pcmklatest]
    openstack-cinder-volume-docker-0     (ocf::heartbeat:docker):        Started controller-2
    stonith-fence_rhevm-566f32200003       (stonith:fence_rhevm):  Started controller-1
    stonith-fence_rhevm-566f3220001a       (stonith:fence_rhevm):  Started controller-2
    stonith-fence_rhevm-566f3220001d       (stonith:fence_rhevm):  Started controller-1
    

Environment

  • Red Hat OpenStack Platform 13
  • Red Hat OpenStack Platform 16.1
  • Red Hat OpenStack Platform 16.2
  • Red Hat OpenStack Platform 17.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content