Hosted Engine host "non operational", cannot connect to Storage Pool

Issue

• A host in a Hosted Engine environment was in a non-operational state.

• Occurred after one of two hosts had crashed.

• No VMs could be started.

• The vdsm logs on the host running the HE VM contained the following errors;

AttributeError: 'HAClient' object has no attribute 'get_local_host_id'

CannotConnectMultiplePools: Cannot connect pool, already connected to another pool: ("['4ece8aa6-5ac6-4071-87b6-da8603f310d0']",)

• The full sequence that transpired was;

  • One of two hosts in a Hosted Engine environment had encountered a hardware failure and had to be shutdown.
  • A specific VM then failed to start due to a scsi inquiry failure against a direct LUN.
  • The HE VM and the host running this VM were rebooted.
  • The host could then not be added to the Storage Pool (Data Center) and went into a "non operational" state.
  • The HE VM was up, the RHEV Manager was accessible, but no other VMs could be started.