After upgrading RHEVM to 3.1, sanlock errors prevent any host from becoming SPM

Issue

• RHEV upgraded from 3.0 to 3.1
• After changing the compatibility mode to 3.1, and the storage is contending.
• Each host contends for SPM status, and fails
• All virtual guests are still running fine
• /var/log/sanlock.log shows the following errors:

74181 [3041]: s315 lockspace eeb51e7a-da6e-4b43-8789-49e1b3c142e1:3:/dev/eeb51e7a-da6e-4b43-8789-49e1b3c142e1/ids:0
74201 [24039]: s315 wdmd_connect failed -111
74201 [24039]: s315 create_watchdog failed -1
74202 [3041]: s315 add_lockspace fail result -203
74323 [3042]: s316 lockspace eeb51e7a-da6e-4b43-8789-49e1b3c142e1:3:/dev/eeb51e7a-da6e-4b43-8789-49e1b3c142e1/ids:0
74343 [24593]: s316 wdmd_connect failed -111
74343 [24593]: s316 create_watchdog failed -1
74344 [3042]: s316 add_lockspace fail result -203

• /var/log/messages log on the hosts show the following:

hpasrd[2981]: Starting with poll 1 and timeout 600.
hpasrd[2981]: Setting the watchdog timer.
hpasrd[2981]: Using watchdog HP iLO2+ HW Watchdog Timer.
rhnsd[3003]: Red Hat Network Services Daemon starting up, check in interval 60 minutes.
wdmd[3023]: wdmd started S0 H1 G179
wdmd[3023]: no /dev/watchdog, load a watchdog driver

• Seeing these errors in /var/log/vdsm/vdsm.log:

3c8f8535-1a32-49f5-9fe1-e5d1e31122cc::ERROR::2013-07-22 12:04:19,191::sp::312::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sp.py", line 270, in startSpm
    self._upgradePool(expectedDomVersion, __securityOverride=True)
  File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 410, in _upgradePool
    self._convertDomain(self.masterDomain, str(targetDomVersion))
  File "/usr/share/vdsm/storage/sp.py", line 1032, in _convertDomain
    domain.getRealDomain(), isMsd, targetFormat)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 360, in convert
    converter(repoPath, hostId, imageRepo, isMsd)
  File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 126, in v3DomainConverter
    newClusterLock.acquireHostId(hostId, async=False)
  File "/usr/share/vdsm/storage/clusterlock.py", line 188, in acquireHostId
    raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id: ('eeb51e7a-da6e-4b43-8789-49e1b3c142e1', SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception'))
3c8f8535-1a32-49f5-9fe1-e5d1e31122cc::ERROR::2013-07-22 12:04:19,192::sp::313::Storage.StoragePool::(startSpm) failed: Cannot acquire host id: ('eeb51e7a-da6e-4b43-8789-49e1b3c142e1', SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception'))