glance is not starting

Issue

• When updating openstack from 16.1.2 to 16.1.4, glance is restarting all the time with the following error in the log:

2021-03-25T13:29:04.045002990+00:00 stderr F + sudo -E kolla_set_configs
2021-03-25T13:29:04.209634288+00:00 stderr F INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
2021-03-25T13:29:04.210207093+00:00 stderr F INFO:__main__:Validating config file
2021-03-25T13:29:04.210417349+00:00 stderr F INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
2021-03-25T13:29:04.210555287+00:00 stderr F INFO:__main__:Copying service configuration files
2021-03-25T13:29:04.212578093+00:00 stderr F INFO:__main__:Deleting /etc/glance/glance-api.conf
2021-03-25T13:29:04.213395023+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-api.conf to /etc/glance/glance-
api.conf
2021-03-25T13:29:04.214698620+00:00 stderr F INFO:__main__:Deleting /etc/glance/glance-cache.conf
2021-03-25T13:29:04.214964852+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-cache.conf to /etc/glance/glanc
e-cache.conf
2021-03-25T13:29:04.215916997+00:00 stderr F INFO:__main__:Deleting /etc/glance/glance-image-import.conf
2021-03-25T13:29:04.216151209+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/glance/glance-image-import.conf to /etc/glanc
e/glance-image-import.conf
2021-03-25T13:29:04.217106184+00:00 stderr F INFO:__main__:Deleting /etc/my.cnf.d/tripleo.cnf
2021-03-25T13:29:04.217288584+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/my.cnf.d/tripleo.cnf to /etc/my.cnf.d/tripleo
.cnf
2021-03-25T13:29:04.218414667+00:00 stderr F INFO:__main__:Deleting /etc/ceph/ceph.conf
2021-03-25T13:29:04.218618622+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.conf to /etc/ceph/ceph.conf
2021-03-25T13:29:04.219291494+00:00 stderr F INFO:__main__:Deleting /etc/ceph/ceph.client.openstack.keyring
2021-03-25T13:29:04.219480338+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.openstack.keyring to /etc/ceph/c
eph.client.openstack.keyring
2021-03-25T13:29:04.220140211+00:00 stderr F INFO:__main__:Deleting /etc/ceph/ceph.client.manila.keyring
2021-03-25T13:29:04.220302758+00:00 stderr F INFO:__main__:Copying /var/lib/kolla/config_files/src-ceph/ceph.client.manila.keyring to /etc/ceph/ceph
.client.manila.keyring
2021-03-25T13:29:04.220914368+00:00 stderr F INFO:__main__:Writing out command to execute
2021-03-25T13:29:04.222678952+00:00 stderr F INFO:__main__:Setting permission for /var/lib/glance
2021-03-25T13:29:04.222865911+00:00 stderr F INFO:__main__:Setting permission for /var/lib/glance/os_glance_staging_store
2021-03-25T13:29:04.223008834+00:00 stderr F INFO:__main__:Setting permission for /var/lib/glance/os_glance_tasks_store
2021-03-25T13:29:04.223459109+00:00 stderr F INFO:__main__:Setting permission for /etc/ceph/ceph.client.openstack.keyring
2021-03-25T13:29:04.242331014+00:00 stderr F ++ cat /run_command
2021-03-25T13:29:04.246642234+00:00 stderr F + CMD='/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glan
ce/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
2021-03-25T13:29:04.246642234+00:00 stderr F + ARGS=
2021-03-25T13:29:04.246688265+00:00 stderr F + sudo kolla_copy_cacerts
2021-03-25T13:29:04.273670573+00:00 stderr F + [[ ! -n '' ]]
2021-03-25T13:29:04.273670573+00:00 stderr F + . kolla_extend_start
2021-03-25T13:29:04.273860933+00:00 stderr F ++ [[ ! -d /var/log/kolla/glance ]]
2021-03-25T13:29:04.274511688+00:00 stderr F +++ stat -c %a /var/log/kolla/glance
2021-03-25T13:29:04.278890420+00:00 stderr F ++ [[ 2755 != \7\5\5 ]]
2021-03-25T13:29:04.278890420+00:00 stderr F ++ chmod 755 /var/log/kolla/glance
2021-03-25T13:29:04.282906434+00:00 stderr F ++ . /usr/local/bin/kolla_glance_extend_start
2021-03-25T13:29:04.283078586+00:00 stderr F +++ [[ -n '' ]]
2021-03-25T13:29:04.283260109+00:00 stderr F +++ [[ -n '' ]]
2021-03-25T13:29:04.283830070+00:00 stderr F ++++ whoami
2021-03-25T13:29:04.289057543+00:00 stderr F +++ [[ glance == \r\o\o\t ]]
2021-03-25T13:29:04.289159038+00:00 stderr F + echo 'Running command: '\''/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'\'''
2021-03-25T13:29:04.289182506+00:00 stdout F Running command: '/usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf'
2021-03-25T13:29:04.289275632+00:00 stderr F + exec /usr/bin/glance-api --config-file /usr/share/glance/glance-api-dist.conf --config-file /etc/glance/glance-api.conf --config-file /etc/glance/glance-image-import.conf
2021-03-25T13:29:07.017939185+00:00 stderr F /builddir/build/BUILD/ceph-14.2.11/src/mon/MonMap.cc: In function 'void mon_info_t::decode(ceph::buffer::v14_2_0::list::const_iterator&)' thread 7fdaf29ac700 time 2021-03-25 13:29:07.017855
2021-03-25T13:29:07.017939185+00:00 stderr F /builddir/build/BUILD/ceph-14.2.11/src/mon/MonMap.cc: 80: FAILED ceph_assert(weight == 10)
2021-03-25T13:29:07.021294722+00:00 stderr F  ceph version 14.2.11-95.el8cp (1d6087ae858e7c8e72fe7390c3522c7e0d951240) nautilus (stable)
2021-03-25T13:29:07.021294722+00:00 stderr F  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7fdafa81c358]
2021-03-25T13:29:07.021294722+00:00 stderr F  2: (()+0x276572) [0x7fdafa81c572]
2021-03-25T13:29:07.021294722+00:00 stderr F  3: (mon_info_t::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)+0x922) [0x7fdafac07f92]
2021-03-25T13:29:07.021294722+00:00 stderr F  4: (MonMap::decode(ceph::buffer::v14_2_0::list::iterator_impl<true>&)+0x98a) [0x7fdafac089aa]
2021-03-25T13:29:07.021294722+00:00 stderr F  5: (MonClient::handle_monmap(MMonMap*)+0x13d) [0x7fdafabf59fd]
2021-03-25T13:29:07.021294722+00:00 stderr F  6: (MonClient::ms_dispatch(Message*)+0x35b) [0x7fdafabfe6db]
2021-03-25T13:29:07.021294722+00:00 stderr F  7: (DispatchQueue::entry()+0x12cc) [0x7fdafaa71fec]
2021-03-25T13:29:07.021294722+00:00 stderr F  8: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdafab273f1]
2021-03-25T13:29:07.021294722+00:00 stderr F  9: (()+0x82de) [0x7fdb124562de]
2021-03-25T13:29:07.021294722+00:00 stderr F  10: (clone()+0x43) [0x7fdb119fde83]

• We found the following RHBZ #1936210 which shows exactly the same error.

• As I understand RedHat has fixed this in ceph, so I was wondering when the fixed ceph version will make it into the images of openstack or did I missunderstand it and the fix is in the server part of ceph?

• We are working with an external ceph cluster.

• We have reverted back to 16.1.2 for the moment, services are restored.