ovirt-ha-broker crashes with error "Host id is not set"

Issue

• Occasionally,ovirt-ha-broker crashes: (broker.log)

StatusStorageThread::ERROR::2021-06-30 07:04:25,746::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to update state.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 82, in run
    if (self._status_broker._inquire_whiteboard_lock() or
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 195, in _inquire_whiteboard_lock
    self.host_id, self._lease_file)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 130, in host_id
    raise ex.HostIdNotLockedError("Host id is not set")
HostIdNotLockedError: Host id is not set
StatusStorageThread::ERROR::2021-06-30 07:04:25,747::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_restart) Trying to restart the broker

• In /var/log/messages we can see:

<hostname> journal:ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.Action.status_monitor ERROR Error in RPC call: submonitor id not found
<hostname> journal:ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Unhandled monitoring loop exception#012Traceback (most recent cal
l last):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 449, in start_monitoring#012    self._monitoring_loop()#012  File "/usr/lib/python2.
7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 468, in _monitoring_loop#012    for old_state, state, delay in self.fsm:#012  File "/usr/lib/python2.7/site-packages/ovir
t_hosted_engine_ha/lib/fsm/machine.py", line 127, in next#012    new_data = self.refresh(self._state.data)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_mach
ine.py", line 81, in refresh#012    stats.update(self.hosted_engine.collect_stats())#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 781, in 
collect_stats#012    ret = monitor['type'](ret)#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 94, in engine_status#012    in json.loads(sta
tus).iteritems()])#012  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads#012    return _default_decoder.decode(s)#012  File "/usr/lib64/python2.7/json/decoder.py", line 366, 
in decode#012    obj, end = self.raw_decode(s, idx=_w(s, 0).end())#012TypeError: expected string or buffer
<hostname> journal:ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.Action.release_host_id ERROR Error in RPC call: Host id is not set
<hostname> journal:ovirt-ha-broker ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Failed to update state.#012Traceback (most recent call l
ast):#012  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 82, in run#012    if (self._status_broker._inquire_whiteboard_lock() or#012  File "/us
r/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 195, in _inquire_whiteboard_lock#012    self.host_id, self._lease_file)#012  File "/usr/lib/python2.7/site
-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 130, in host_id#012    raise ex.HostIdNotLockedError("Host id is not set")#012HostIdNotLockedError: Host id is not set