RHV - Storage Domains down and no host is the SPM due to improper permissions in export domain
Issue
- None of the hosts in the cluster are set as the SPM.
- [Errno 13] Permission Denied errors are appearing in RHV-M.
- All storage domains are down, and DC is unresponsive.
There may be messages similar to:
VDSM example.local command HSMGetTaskStatusVDS failed: [Errno 13] Permission denied
VDSM example.local command HSMGetAllTasksStatusesVDS failed: Not SPM
VDSM example.local command HSMGetTaskStatusVDS failed: [Errno 13] Permission denied
VDSM example.local command HSMGetAllTasksStatusesVDS failed: Not SPM
VDSM example.local command HSMGetTaskStatusVDS failed: [Errno 13] Permission denied
Error from vdsm.log:
2018-03-02 15:01:21,654-0300 ERROR (monitor/83aeade) [storage.Monitor] Error checking domain xxxxxxxxxxxxxxxxxx (monitor:426)
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 407, in _checkDomainStatus
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 50, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 153, in _findDomain
return findMethod(sdUUID)
File "/usr/share/vdsm/storage/nfsSD.py", line 126, in findDomain
return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/fileSD.py", line 369, in __init__
validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint)
File "/usr/share/vdsm/storage/fileSD.py", line 98, in validateFileSystemFeatures
oop.getProcessPool(sdUUID).directTouch(testFilePath)
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch
ioproc.touch(path, flags, mode)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 577, in touch
self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 13] Permission denied
Permissions are incorrect, the correct permissions should be vdsm:kvm (36:36):
/rhev/data-center/mnt:
total 21
drwxr-xr-x. 3 nobody nobody 4 Dec 6 16:36 10.0.0.1:_mnt_export_xxxxxxxxxxxxx
drwxr-xr-x. 3 nobody nobody 4 Dec 5 12:06 10.0.0.1:_mnt_iso_xxxxxxxxxxxxx
drwxr-xr-x. 10 vdsm kvm 4096 Jan 9 09:11 blockSD
/rhev/data-center/mnt/10.0.0.1:_mnt_export/xxxxxxxxxxxxx:
total 1
drwxr-xr-x. 5 nobody nobody 5 Sep 14 22:45 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-rwxr-xr-x. 1 nobody nobody 0 Feb 8 17:13 __DIRECT_IO_TEST__
Environment
- Red Hat Virtualization 4.0
- Red Hat Virtualization 4.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
