RHV - Storage Domains down and no host is the SPM due to improper permissions in export domain

Issue

• None of the hosts in the cluster are set as the SPM.
• [Errno 13] Permission Denied errors are appearing in RHV-M.
• All storage domains are down, and DC is unresponsive.

There may be messages similar to:

VDSM example.local command HSMGetTaskStatusVDS failed: [Errno 13] Permission denied
VDSM example.local command HSMGetAllTasksStatusesVDS failed: Not SPM
VDSM example.local command HSMGetTaskStatusVDS failed: [Errno 13] Permission denied
VDSM example.local command HSMGetAllTasksStatusesVDS failed: Not SPM
VDSM example.local command HSMGetTaskStatusVDS failed: [Errno 13] Permission denied

Error from vdsm.log:

2018-03-02 15:01:21,654-0300 ERROR (monitor/83aeade) [storage.Monitor] Error checking domain xxxxxxxxxxxxxxxxxx (monitor:426)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/monitor.py", line 407, in _checkDomainStatus
    self.domain.selftest()
  File "/usr/share/vdsm/storage/sdc.py", line 50, in __getattr__
    return getattr(self.getRealDomain(), attrName)
  File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 136, in _realProduce
    domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 153, in _findDomain
    return findMethod(sdUUID)
  File "/usr/share/vdsm/storage/nfsSD.py", line 126, in findDomain
    return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/fileSD.py", line 369, in __init__
    validateFileSystemFeatures(manifest.sdUUID, manifest.mountpoint)
  File "/usr/share/vdsm/storage/fileSD.py", line 98, in validateFileSystemFeatures
    oop.getProcessPool(sdUUID).directTouch(testFilePath)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 320, in directTouch
    ioproc.touch(path, flags, mode)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 577, in touch
    self.timeout)
  File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 461, in _sendCommand
    raise OSError(errcode, errstr)
OSError: [Errno 13] Permission denied

Permissions are incorrect, the correct permissions should be vdsm:kvm (36:36):

/rhev/data-center/mnt:
total 21
drwxr-xr-x.  3 nobody nobody    4 Dec  6 16:36 10.0.0.1:_mnt_export_xxxxxxxxxxxxx
drwxr-xr-x.  3 nobody nobody    4 Dec  5 12:06 10.0.0.1:_mnt_iso_xxxxxxxxxxxxx
drwxr-xr-x. 10 vdsm   kvm    4096 Jan  9 09:11 blockSD

/rhev/data-center/mnt/10.0.0.1:_mnt_export/xxxxxxxxxxxxx:
total 1
drwxr-xr-x. 5 nobody nobody 5 Sep 14 22:45 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-rwxr-xr-x. 1 nobody nobody 0 Feb  8 17:13 __DIRECT_IO_TEST__