service openstack-cinder-volume-docker-0 is missing after minor upgrade

Solution In Progress - Updated -

Issue

  • Following a minor update, openstack-cinder-volume-docker-0 stopped and never restarted:
[root@overcloud-controller-0 cinder]# pcs status Cluster name: tripleo_cluster
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum Last updated: Tue May  7 14:59:24 2019 Last change: Tue May  7 14:00:55 2019 by hacluster via crmd on overcloud-controller-1

15 nodes configured
67 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 ovn-dbs-bundle-0@overcloud-controller-0 ovn-dbs-bundle-1@overcloud-controller-1 ovn-dbs-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [10.10.10.10::8787/rhosp13/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started overcloud-controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started overcloud-controller-1
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started overcloud-controller-2
 Docker container set: galera-bundle [10.10.10.10::8787/rhosp13/openstack-mariadb:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Master overcloud-controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Master overcloud-controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Master overcloud-controller-2
 Docker container set: redis-bundle [10.10.10.10::8787/rhosp13/openstack-redis:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Slave overcloud-controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Master overcloud-controller-1
   redis-bundle-2       (ocf::heartbeat:redis): Slave overcloud-controller-2
 ip-10.151.111.160      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.32.133.151       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 ip-10.151.111.69       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2
 ip-10.151.111.70       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.151.111.204      (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-10.151.109.12       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Docker container set: haproxy-bundle [10.10.10.10::8787/rhosp13/openstack-haproxy:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started overcloud-controller-0
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started overcloud-controller-1
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started overcloud-controller-2
 Docker container set: ovn-dbs-bundle [10.10.10.10::8787/rhosp13/openstack-ovn-northd:pcmklatest]
   ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Master overcloud-controller-0
   ovn-dbs-bundle-1     (ocf::ovn:ovndb-servers):       Slave overcloud-controller-1
   ovn-dbs-bundle-2     (ocf::ovn:ovndb-servers):       Slave overcloud-controller-2
 stonith-fence_compute-fence-nova       (stonith:fence_compute):        Started overcloud-controller-2
 Clone Set: compute-unfence-trigger-clone [compute-unfence-trigger]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]  nova-evacuate  (ocf::openstack:NovaEvacuate):  Started overcloud-controller-1
 stonith-fence_ipmilan-e4434b5adcc0     (stonith:fence_ipmilan):        Started overcloud-controller-1
 stonith-fence_ipmilan-e4434b5aca88     (stonith:fence_ipmilan):        Started overcloud-controller-2
 stonith-fence_ipmilan-e4434b5aca08     (stonith:fence_ipmilan):        Started overcloud-controller-2
 Docker container: openstack-cinder-volume [10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest]
   openstack-cinder-volume-docker-0     (ocf::heartbeat:docker):        Stopped
 Docker container: openstack-cinder-backup [10.10.10.10::8787/rhosp13/openstack-cinder-backup:pcmklatest]
   openstack-cinder-backup-docker-0     (ocf::heartbeat:docker):        Started overcloud-controller-2

Failed Actions:
* openstack-cinder-volume-docker-0_start_0 on overcloud-controller-2 'unknown error' (1): call=142, status=complete, exitreason='failed to pull image 10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest',
    last-rc-change='Tue May  7 14:09:35 2019', queued=0ms, exec=263ms
* openstack-cinder-volume-docker-0_start_0 on overcloud-controller-0 'unknown error' (1): call=109, status=complete, exitreason='failed to pull image 10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest',
    last-rc-change='Tue May  7 14:08:54 2019', queued=0ms, exec=334ms
* openstack-cinder-volume-docker-0_start_0 on overcloud-controller-1 'unknown error' (1): call=144, status=complete, exitreason='failed to pull image 10.10.10.10:8787/custom/openstack-cinder-volume-custom:pcmklatest',
    last-rc-change='Tue May  7 14:09:48 2019', queued=0ms, exec=258ms


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
  • Containers using the tag latest were not updated

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content