service openstack-cinder-volume-docker-0 is missing after minor upgrade

Solution In Progress - Updated -

Issue

  • Following a minor update, openstack-cinder-volume-docker-0 stopped and never restarted:
[root@overcloud-controller-0 cinder]# pcs status Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum Last updated: Tue May  7 14:59:24 2019 Last change: Tue May  7 14:00:55 2019 by hacluster via crmd on overcloud-controller-1

15 nodes configured
67 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 ovn-dbs-bundle-0@overcloud-controller-0 ovn-dbs-bundle-1@overcloud-controller-1 ovn-dbs-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [10.10.10.10::8787/rhosp13/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started overcloud-controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started overcloud-controller-1
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started overcloud-controller-2
 Docker container set: galera-bundle [10.10.10.10::8787/rhosp13/openstack-mariadb:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Master overcloud-controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Master overcloud-controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Master overcloud-controller-2
 Docker container set: redis-bundle [10.10.10.10::8787/rhosp13/openstack-redis:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Slave overcloud-controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Master overcloud-controller-1
   redis-bundle-2       (ocf::heartbeat:redis): Slave overcloud-controller-2
 ip-10.151.111.160      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.32.133.151       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 ip-10.151.111.69       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 ip-10.151.111.70       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.151.111.204      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.151.109.12       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Docker container set: haproxy-bundle [10.10.10.10::8787/rhosp13/openstack-haproxy:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started overcloud-controller-0
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started overcloud-controller-1
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started overcloud-controller-2
 Docker container set: ovn-dbs-bundle [10.10.10.10::8787/rhosp13/openstack-ovn-northd:pcmklatest]
   ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Master overcloud-controller-0
   ovn-dbs-bundle-1     (ocf::ovn:ovndb-servers):       Slave overcloud-controller-1
   ovn-dbs-bundle-2     (ocf::ovn:ovndb-servers):       Slave overcloud-controller-2
 stonith-fence_compute-fence-nova       (stonith:fence_compute):        Started overcloud-controller-2
 Clone Set: compute-unfence-trigger-clone [compute-unfence-trigger]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]  nova-evacuate  (ocf::openstack:NovaEvacuate):  Started overcloud-controller-1
 stonith-fence_ipmilan-e4434b5adcc0     (stonith:fence_ipmilan):        Started overcloud-controller-1
 stonith-fence_ipmilan-e4434b5aca88     (stonith:fence_ipmilan):        Started overcloud-controller-2
 stonith-fence_ipmilan-e4434b5aca08     (stonith:fence_ipmilan):        Started overcloud-controller-2
 Docker container: openstack-cinder-volume [10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest]
   openstack-cinder-volume-docker-0     (ocf::heartbeat:docker):        Stopped
 Docker container: openstack-cinder-backup [10.10.10.10::8787/rhosp13/openstack-cinder-backup:pcmklatest]
   openstack-cinder-backup-docker-0     (ocf::heartbeat:docker):        Started overcloud-controller-2

Failed Actions:
* openstack-cinder-volume-docker-0_start_0 on overcloud-controller-2 'unknown error' (1): call=142, status=complete, exitreason='failed to pull image 10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest',
    last-rc-change='Tue May  7 14:09:35 2019', queued=0ms, exec=263ms
* openstack-cinder-volume-docker-0_start_0 on overcloud-controller-0 'unknown error' (1): call=109, status=complete, exitreason='failed to pull image 10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest',
    last-rc-change='Tue May  7 14:08:54 2019', queued=0ms, exec=334ms
* openstack-cinder-volume-docker-0_start_0 on overcloud-controller-1 'unknown error' (1): call=144, status=complete, exitreason='failed to pull image 10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest',
    last-rc-change='Tue May  7 14:09:48 2019', queued=0ms, exec=258ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
  • Containers using the tag latest were not updated

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In