Hypervisor is in non-operational mode with repeated "org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing" mesage in rhevm.log

Issue

• RHEV-H changed to Non-Operational mode with below error messages.

In rhevm.log:

rhevm.log:2012-11-07 02:41:51,074 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-61) vds::refreshVdsStats Failed getVdsStats,  vds = 6ef2ba
64-025d-11e2-89ae-3351bd2e9880 : error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing

• Service vdsmd status shows, the service is running.

# cat sos_commands/vdsm/etc.init.d.vdsmd_status 
VDS daemon server is running

• But all vdsclient commands fail on this host:

# cat sos_commands/vdsm/vdsClient_-s_0_list 
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsClient.py", line 1987, in <module>
    code, message = commands[command][0](commandArgs)
  File "/usr/share/vdsm/vdsClient.py", line 189, in do_list
    for conf in response['vmList']:
KeyError: 'vmList'

# cat sos_commands/vdsm/vdsClient_-s_0_getVdsCapabilities 
Recovering from crash or Initializing