After upgrading RHEVM to 3.1, sanlock errors prevent any host from becoming SPM
Issue
- RHEV upgraded from 3.0 to 3.1
- After changing the compatibility mode to 3.1, and the storage is contending.
- Each host contends for SPM status, and fails
- All virtual guests are still running fine
/var/log/sanlock.logshows the following errors:
74181 [3041]: s315 lockspace eeb51e7a-da6e-4b43-8789-49e1b3c142e1:3:/dev/eeb51e7a-da6e-4b43-8789-49e1b3c142e1/ids:0
74201 [24039]: s315 wdmd_connect failed -111
74201 [24039]: s315 create_watchdog failed -1
74202 [3041]: s315 add_lockspace fail result -203
74323 [3042]: s316 lockspace eeb51e7a-da6e-4b43-8789-49e1b3c142e1:3:/dev/eeb51e7a-da6e-4b43-8789-49e1b3c142e1/ids:0
74343 [24593]: s316 wdmd_connect failed -111
74343 [24593]: s316 create_watchdog failed -1
74344 [3042]: s316 add_lockspace fail result -203
/var/log/messageslog on the hosts show the following:
hpasrd[2981]: Starting with poll 1 and timeout 600.
hpasrd[2981]: Setting the watchdog timer.
hpasrd[2981]: Using watchdog HP iLO2+ HW Watchdog Timer.
rhnsd[3003]: Red Hat Network Services Daemon starting up, check in interval 60 minutes.
wdmd[3023]: wdmd started S0 H1 G179
wdmd[3023]: no /dev/watchdog, load a watchdog driver
- Seeing these errors in
/var/log/vdsm/vdsm.log:
3c8f8535-1a32-49f5-9fe1-e5d1e31122cc::ERROR::2013-07-22 12:04:19,191::sp::312::Storage.StoragePool::(startSpm) Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sp.py", line 270, in startSpm
self._upgradePool(expectedDomVersion, __securityOverride=True)
File "/usr/share/vdsm/storage/securable.py", line 68, in wrapper
return f(self, *args, **kwargs)
File "/usr/share/vdsm/storage/sp.py", line 410, in _upgradePool
self._convertDomain(self.masterDomain, str(targetDomVersion))
File "/usr/share/vdsm/storage/sp.py", line 1032, in _convertDomain
domain.getRealDomain(), isMsd, targetFormat)
File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 360, in convert
converter(repoPath, hostId, imageRepo, isMsd)
File "/usr/share/vdsm/storage/imageRepository/formatConverter.py", line 126, in v3DomainConverter
newClusterLock.acquireHostId(hostId, async=False)
File "/usr/share/vdsm/storage/clusterlock.py", line 188, in acquireHostId
raise se.AcquireHostIdFailure(self._sdUUID, e)
AcquireHostIdFailure: Cannot acquire host id: ('eeb51e7a-da6e-4b43-8789-49e1b3c142e1', SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception'))
3c8f8535-1a32-49f5-9fe1-e5d1e31122cc::ERROR::2013-07-22 12:04:19,192::sp::313::Storage.StoragePool::(startSpm) failed: Cannot acquire host id: ('eeb51e7a-da6e-4b43-8789-49e1b3c142e1', SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception'))
Environment
- Red Hat Enterprise Virtualization 3.1
- kernel-2.6.32-358.11.1.el6.x86_64
- vdsm-4.10.2-22.0.el6ev
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
