OVN northdb and southdb crashed after applying a new tag

Issue

• OVN northdb and southdb crashed after a change to apply pcmklatest tag

• Before the change:

[root@overcloud-controller-0 ~]#  pcs status |grep ovn-n
 Docker container set: ovn-dbs-bundle [satellite.localdomain:5000/osp13-containers-ovn-northd:13.0-hotfix-bz1766410-v2-20200110         .1578623869]
[root@overcloud-controller-0 ~]# docker images |grep ovn-n
satellite.localdomain:5000/production-osp_containers-osp13-containers-ovn-northd                  pcmklatest                                              e283e80ff8aa        3 months ago        869 MB
satellite.localdomain:5000/osp13-containers-ovn-northd                                                 13.0-hotfix-bz1766410-v2         -20200110.1578623869   e283e80ff8aa        3 months ago        869 MB

• Update ovn container tag with:

[root@overcloud-controller-0 ~]# pcs resource bundle update ovn-dbs-bundle container image=satellite.localdomain:5000/production-osp_containers-osp13-containers-ovn-northd:pcmklatest

• After the change, ovn-bundle restarted correctly:

[2020-04-17 09:33:18]   Docker container set: ovn-dbs-bundle [satellite.localdomain:5000/production-osp_containers-osp13-containers-ovn-northd:pcmklatest]
[2020-04-17 09:33:18]     ovn-dbs-bundle-0(ocf::ovn:ovndb-servers):Master overcloud-controller-0
[2020-04-17 09:33:18]     ovn-dbs-bundle-1(ocf::ovn:ovndb-servers):Slave iad1-ctl02
[2020-04-17 09:33:18]     ovn-dbs-bundle-2(ocf::ovn:ovndb-servers):Slave iad1-ctl03

• However, ovn-northd was empty which caused all instances lost connectivity:

[root@overcloud-controller-0 ~]# docker exec -it ovn-dbs-bundle-docker-0 ovn-nbctl show
[root@overcloud-controller-0 ~]# 

• ovn-db was rebuilt with below sync utility.

# neutron-ovn-db-sync-util --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --ovn-neutron_sync_mode=repair

• 2000 instances were down due to this issue