2016-12-21 06:41:33,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:41:33,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) HostName = rhev-h1.example.com 2016-12-21 06:41:33,491 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:41:33,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-90) FINISH, ConnectStoragePoolVDSCommand, log id: 49e5507 2016-12-21 06:41:33,491 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:41:33,491 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-90) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:41:33,506 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-46) [406bd786] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:41:33,526 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-1) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:41:43,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:41:43,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) HostName = rhev-h1.example.com 2016-12-21 06:41:43,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:41:43,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-12) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:41:43,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 381cda19 2016-12-21 06:42:00,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-30) [5e838ff7] FINISH, ConnectStorageServerVDSCommand, return: {ba8b7d16-80ea-4804-8a6e-109dff5c8919=0}, log id: 5721bfc4 2016-12-21 06:42:00,093 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [305b188] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:42 AM 2016-12-21 06:42:43,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:42:43,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) HostName = rhev-h1.example.com 2016-12-21 06:42:43,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:42:43,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, ConnectStoragePoolVDSCommand, log id: 381cda19 2016-12-21 06:42:43,704 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:42:43,704 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-12) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:42:43,725 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-28) [59f089a3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:42:43,746 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-31) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:42:53,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:42:53,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) HostName = rhev-h1.example.com 2016-12-21 06:42:53,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-83) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:42:53,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-83) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:42:53,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 73d03d2b 2016-12-21 06:43:53,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:43:53,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) HostName = rhev-h1.example.com 2016-12-21 06:43:53,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:43:53,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-83) FINISH, ConnectStoragePoolVDSCommand, log id: 73d03d2b 2016-12-21 06:43:53,919 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:43:53,919 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-83) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:43:53,939 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-16) [4b838eb3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:43:53,955 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-13) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:44:03,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:44:03,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) HostName = rhev-h1.example.com 2016-12-21 06:44:03,940 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-100) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:44:03,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-100) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:44:03,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 28ec605b 2016-12-21 06:45:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) Autorecovering 3 storage domains 2016-12-21 06:45:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) Autorecovering storage domains id: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 2016-12-21 06:45:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [ae7fa66] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 Type: Storage 2016-12-21 06:45:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [ae7fa66] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:45 AM 2016-12-21 06:45:00,013 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-3) [2962545] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:45:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-3) [2962545] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 06905dd9-33a2-4b92-a73e-8c85a4e0f375, connection: 192.168.50.250:/home/export-nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1c790506 2016-12-21 06:45:04,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:45:04,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) HostName = rhev-h1.example.com 2016-12-21 06:45:04,116 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:45:04,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 28ec605b 2016-12-21 06:45:04,116 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:45:04,116 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-100) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:45:04,148 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-50) [2c429112] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:45:04,196 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-21) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:45:14,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:45:14,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = rhev-h1.example.com 2016-12-21 06:45:14,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-70) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:45:14,150 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-70) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:45:14,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 42fe2c65 2016-12-21 06:46:00,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-3) [2962545] FINISH, ConnectStorageServerVDSCommand, return: {06905dd9-33a2-4b92-a73e-8c85a4e0f375=0}, log id: 1c790506 2016-12-21 06:46:00,034 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [ae7fa66] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:46 AM 2016-12-21 06:46:00,034 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [ae7fa66] Autorecovering storage domains id: f05ef6f9-4307-4dc0-ad59-df471caef3e9 2016-12-21 06:46:00,036 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [3d9a3833] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f05ef6f9-4307-4dc0-ad59-df471caef3e9 Type: Storage 2016-12-21 06:46:00,036 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [3d9a3833] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:46 AM 2016-12-21 06:46:00,040 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-28) [4401d368] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:46:00,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-28) [4401d368] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f9b36ad4-abe6-42eb-a105-d43af2548c68, connection: 192.168.50.200:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 12301552 2016-12-21 06:46:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-28) [4401d368] FINISH, ConnectStorageServerVDSCommand, return: {f9b36ad4-abe6-42eb-a105-d43af2548c68=0}, log id: 12301552 2016-12-21 06:46:00,060 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [3d9a3833] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:46 AM 2016-12-21 06:46:00,060 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [3d9a3833] Autorecovering storage domains id: 477c507a-1b94-4565-89cf-b6d8f0b98693 2016-12-21 06:46:00,061 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [15b27812] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:46:00,062 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [15b27812] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:46 AM 2016-12-21 06:46:00,066 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-29) [e9c8017] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:46:00,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-29) [e9c8017] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ba8b7d16-80ea-4804-8a6e-109dff5c8919, connection: 192.168.50.250:/home/nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 73394d95 2016-12-21 06:46:14,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:46:14,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = rhev-h1.example.com 2016-12-21 06:46:14,293 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:46:14,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, ConnectStoragePoolVDSCommand, log id: 42fe2c65 2016-12-21 06:46:14,293 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:46:14,293 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-70) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:46:14,311 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-38) [5f9d4414] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:46:14,325 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-35) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:46:24,307 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-32) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:46:24,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-32) HostName = rhev-h1.example.com 2016-12-21 06:46:24,308 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-32) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:46:24,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:46:24,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 708c2870 2016-12-21 06:47:00,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-29) [e9c8017] FINISH, ConnectStorageServerVDSCommand, return: {ba8b7d16-80ea-4804-8a6e-109dff5c8919=0}, log id: 73394d95 2016-12-21 06:47:00,084 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-33) [15b27812] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:47 AM 2016-12-21 06:47:24,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:47:24,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) HostName = rhev-h1.example.com 2016-12-21 06:47:24,478 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:47:24,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, ConnectStoragePoolVDSCommand, log id: 708c2870 2016-12-21 06:47:24,478 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:47:24,478 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:47:24,504 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-20) [2144e1af] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:47:24,522 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-25) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:47:34,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:47:34,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) HostName = rhev-h1.example.com 2016-12-21 06:47:34,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:47:34,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-7) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:47:34,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 2550c80b 2016-12-21 06:48:34,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:48:34,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) HostName = rhev-h1.example.com 2016-12-21 06:48:34,685 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:48:34,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, ConnectStoragePoolVDSCommand, log id: 2550c80b 2016-12-21 06:48:34,685 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:48:34,685 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:48:34,701 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-29) [21e60c50] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:48:34,715 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-26) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:48:44,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:48:44,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = rhev-h1.example.com 2016-12-21 06:48:44,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:48:44,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:48:44,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 4ff67ee4 2016-12-21 06:49:44,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:49:44,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = rhev-h1.example.com 2016-12-21 06:49:44,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:49:44,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, ConnectStoragePoolVDSCommand, log id: 4ff67ee4 2016-12-21 06:49:44,866 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:49:44,866 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:49:44,880 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-39) [4bdc539e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:49:44,893 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-42) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:49:54,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:49:54,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = rhev-h1.example.com 2016-12-21 06:49:54,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:49:54,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:49:54,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 7cdaad88 2016-12-21 06:50:00,008 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) Autorecovering 3 storage domains 2016-12-21 06:50:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) Autorecovering storage domains id: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 2016-12-21 06:50:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [1f14e150] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 Type: Storage 2016-12-21 06:50:00,009 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [1f14e150] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:50 AM 2016-12-21 06:50:00,014 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-21) [b26bc03] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-21) [b26bc03] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 06905dd9-33a2-4b92-a73e-8c85a4e0f375, connection: 192.168.50.250:/home/export-nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28958738 2016-12-21 06:50:55,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:50:55,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = rhev-h1.example.com 2016-12-21 06:50:55,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:50:55,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7cdaad88 2016-12-21 06:50:55,053 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:50:55,053 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:50:55,069 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-2) [f276926] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:50:55,083 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-31) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:51:00,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-21) [b26bc03] FINISH, ConnectStorageServerVDSCommand, return: {06905dd9-33a2-4b92-a73e-8c85a4e0f375=0}, log id: 28958738 2016-12-21 06:51:00,046 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [1f14e150] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:51 AM 2016-12-21 06:51:00,048 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) [1f14e150] Autorecovering storage domains id: f05ef6f9-4307-4dc0-ad59-df471caef3e9 2016-12-21 06:51:00,049 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [72be8f3a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f05ef6f9-4307-4dc0-ad59-df471caef3e9 Type: Storage 2016-12-21 06:51:00,049 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [72be8f3a] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:51 AM 2016-12-21 06:51:00,060 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-28) [11b3a569] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:51:00,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-28) [11b3a569] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f9b36ad4-abe6-42eb-a105-d43af2548c68, connection: 192.168.50.200:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7eff5e57 2016-12-21 06:51:00,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-28) [11b3a569] FINISH, ConnectStorageServerVDSCommand, return: {f9b36ad4-abe6-42eb-a105-d43af2548c68=0}, log id: 7eff5e57 2016-12-21 06:51:00,097 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [72be8f3a] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:51 AM 2016-12-21 06:51:00,098 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) [72be8f3a] Autorecovering storage domains id: 477c507a-1b94-4565-89cf-b6d8f0b98693 2016-12-21 06:51:00,099 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [1d6c87a6] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:51:00,099 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [1d6c87a6] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:51 AM 2016-12-21 06:51:00,106 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-33) [24e72ce7] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:51:00,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-33) [24e72ce7] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ba8b7d16-80ea-4804-8a6e-109dff5c8919, connection: 192.168.50.250:/home/nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 65e1390 2016-12-21 06:51:05,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:51:05,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-15) HostName = rhev-h1.example.com 2016-12-21 06:51:05,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-15) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:51:05,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-15) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:51:05,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 4f98a401 2016-12-21 06:52:00,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-33) [24e72ce7] FINISH, ConnectStorageServerVDSCommand, return: {ba8b7d16-80ea-4804-8a6e-109dff5c8919=0}, log id: 65e1390 2016-12-21 06:52:00,133 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-53) [1d6c87a6] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:52 AM 2016-12-21 06:52:05,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:52:05,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) HostName = rhev-h1.example.com 2016-12-21 06:52:05,263 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:52:05,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-15) FINISH, ConnectStoragePoolVDSCommand, log id: 4f98a401 2016-12-21 06:52:05,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:52:05,263 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-15) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:52:05,277 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-16) [2bb3b22b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:52:05,293 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-12) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:52:15,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:52:15,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-39) HostName = rhev-h1.example.com 2016-12-21 06:52:15,283 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:52:15,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-39) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:52:15,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: ceaa48c 2016-12-21 06:53:15,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:53:15,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) HostName = rhev-h1.example.com 2016-12-21 06:53:15,450 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:53:15,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, ConnectStoragePoolVDSCommand, log id: ceaa48c 2016-12-21 06:53:15,450 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-39) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:53:15,450 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-39) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:53:15,467 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-50) [28ce94ae] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:53:15,483 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-6) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:53:25,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:53:25,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) HostName = rhev-h1.example.com 2016-12-21 06:53:25,472 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-61) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:53:25,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-61) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:53:25,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 2416a9ce 2016-12-21 06:54:25,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:54:25,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) HostName = rhev-h1.example.com 2016-12-21 06:54:25,636 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:54:25,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-61) FINISH, ConnectStoragePoolVDSCommand, log id: 2416a9ce 2016-12-21 06:54:25,636 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:54:25,636 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-61) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:54:25,654 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-34) [47372bd5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:54:25,669 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-38) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:54:35,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:54:35,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-39) HostName = rhev-h1.example.com 2016-12-21 06:54:35,663 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-39) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:54:35,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-39) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:54:35,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 59774c65 2016-12-21 06:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) Autorecovering 3 storage domains 2016-12-21 06:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) Autorecovering storage domains id: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 2016-12-21 06:55:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [2078f636] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 Type: Storage 2016-12-21 06:55:00,006 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [2078f636] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:55 AM 2016-12-21 06:55:00,010 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-4) [2c103a04] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:55:00,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-4) [2c103a04] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 06905dd9-33a2-4b92-a73e-8c85a4e0f375, connection: 192.168.50.250:/home/export-nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 25a1736b 2016-12-21 06:55:35,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:55:35,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) HostName = rhev-h1.example.com 2016-12-21 06:55:35,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:55:35,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, ConnectStoragePoolVDSCommand, log id: 59774c65 2016-12-21 06:55:35,873 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-39) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:55:35,873 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-39) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:55:35,887 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-20) [22ace022] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:55:35,901 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-43) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:55:45,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:55:45,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) HostName = rhev-h1.example.com 2016-12-21 06:55:45,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-98) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:55:45,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-98) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:55:45,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 3bd2e9ac 2016-12-21 06:56:00,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-4) [2c103a04] FINISH, ConnectStorageServerVDSCommand, return: {06905dd9-33a2-4b92-a73e-8c85a4e0f375=0}, log id: 25a1736b 2016-12-21 06:56:00,037 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [2078f636] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:56 AM 2016-12-21 06:56:00,037 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) [2078f636] Autorecovering storage domains id: f05ef6f9-4307-4dc0-ad59-df471caef3e9 2016-12-21 06:56:00,042 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [d0b279d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f05ef6f9-4307-4dc0-ad59-df471caef3e9 Type: Storage 2016-12-21 06:56:00,042 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [d0b279d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:56 AM 2016-12-21 06:56:00,050 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-37) [4f283b0b] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:56:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-37) [4f283b0b] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f9b36ad4-abe6-42eb-a105-d43af2548c68, connection: 192.168.50.200:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 57e9f6d6 2016-12-21 06:56:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-37) [4f283b0b] FINISH, ConnectStorageServerVDSCommand, return: {f9b36ad4-abe6-42eb-a105-d43af2548c68=0}, log id: 57e9f6d6 2016-12-21 06:56:00,072 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [d0b279d] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:56 AM 2016-12-21 06:56:00,072 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) [d0b279d] Autorecovering storage domains id: 477c507a-1b94-4565-89cf-b6d8f0b98693 2016-12-21 06:56:00,073 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [1819b62c] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:56:00,073 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [1819b62c] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 6:56 AM 2016-12-21 06:56:00,077 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-38) [76e18918] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 06:56:00,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-38) [76e18918] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ba8b7d16-80ea-4804-8a6e-109dff5c8919, connection: 192.168.50.250:/home/nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7475c3ca 2016-12-21 06:56:46,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:56:46,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) HostName = rhev-h1.example.com 2016-12-21 06:56:46,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:56:46,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, ConnectStoragePoolVDSCommand, log id: 3bd2e9ac 2016-12-21 06:56:46,045 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:56:46,045 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-98) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:56:46,063 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-10) [3e5bc26a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:56:46,087 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-26) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:56:56,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:56:56,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) HostName = rhev-h1.example.com 2016-12-21 06:56:56,064 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:56:56,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:56:56,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 3ff7f8ad 2016-12-21 06:57:00,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-38) [76e18918] FINISH, ConnectStorageServerVDSCommand, return: {ba8b7d16-80ea-4804-8a6e-109dff5c8919=0}, log id: 7475c3ca 2016-12-21 06:57:00,097 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-80) [1819b62c] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 6:57 AM 2016-12-21 06:57:56,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:57:56,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) HostName = rhev-h1.example.com 2016-12-21 06:57:56,253 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:57:56,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, ConnectStoragePoolVDSCommand, log id: 3ff7f8ad 2016-12-21 06:57:56,253 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:57:56,253 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:57:56,270 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-44) [32ae4cfe] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:57:56,289 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-42) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:58:06,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:58:06,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) HostName = rhev-h1.example.com 2016-12-21 06:58:06,275 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-34) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:58:06,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-34) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:58:06,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 3e2c499c 2016-12-21 06:59:06,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 06:59:06,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) HostName = rhev-h1.example.com 2016-12-21 06:59:06,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:59:06,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, ConnectStoragePoolVDSCommand, log id: 3e2c499c 2016-12-21 06:59:06,454 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 06:59:06,454 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-34) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 06:59:06,467 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-2) [2dc54632] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 06:59:06,486 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-31) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 06:59:16,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 06:59:16,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] HostName = rhev-h1.example.com 2016-12-21 06:59:16,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 06:59:16,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) [3160be6d] hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 06:59:16,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 15a58648 2016-12-21 07:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [305b188] Autorecovering 3 storage domains 2016-12-21 07:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [305b188] Autorecovering storage domains id: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 2016-12-21 07:00:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [6cd5ca7] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 Type: Storage 2016-12-21 07:00:00,005 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [6cd5ca7] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:00 AM 2016-12-21 07:00:00,012 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-8) [7d246c3f] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-8) [7d246c3f] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 06905dd9-33a2-4b92-a73e-8c85a4e0f375, connection: 192.168.50.250:/home/export-nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c9af5da 2016-12-21 07:00:16,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:00:16,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] HostName = rhev-h1.example.com 2016-12-21 07:00:16,626 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:00:16,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] FINISH, ConnectStoragePoolVDSCommand, log id: 15a58648 2016-12-21 07:00:16,626 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:00:16,626 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:00:16,650 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-14) [19170a2b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:00:16,676 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-33) [3160be6d] Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:00:26,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:00:26,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-72) HostName = rhev-h1.example.com 2016-12-21 07:00:26,647 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-72) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:00:26,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-72) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:00:26,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 1021a00c 2016-12-21 07:01:00,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-8) [7d246c3f] FINISH, ConnectStorageServerVDSCommand, return: {06905dd9-33a2-4b92-a73e-8c85a4e0f375=0}, log id: 4c9af5da 2016-12-21 07:01:00,047 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [6cd5ca7] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 7:01 AM 2016-12-21 07:01:00,048 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [6cd5ca7] Autorecovering storage domains id: f05ef6f9-4307-4dc0-ad59-df471caef3e9 2016-12-21 07:01:00,048 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [6dfd78bc] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f05ef6f9-4307-4dc0-ad59-df471caef3e9 Type: Storage 2016-12-21 07:01:00,048 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [6dfd78bc] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:01 AM 2016-12-21 07:01:00,055 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-34) [8cd20e6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:01:00,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-34) [8cd20e6] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f9b36ad4-abe6-42eb-a105-d43af2548c68, connection: 192.168.50.200:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 365836b 2016-12-21 07:01:00,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-34) [8cd20e6] FINISH, ConnectStorageServerVDSCommand, return: {f9b36ad4-abe6-42eb-a105-d43af2548c68=0}, log id: 365836b 2016-12-21 07:01:00,071 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [6dfd78bc] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 7:01 AM 2016-12-21 07:01:00,072 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [6dfd78bc] Autorecovering storage domains id: 477c507a-1b94-4565-89cf-b6d8f0b98693 2016-12-21 07:01:00,072 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [744c9d0d] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:01:00,072 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [744c9d0d] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:01 AM 2016-12-21 07:01:00,076 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-37) [4d4dcfaf] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:01:00,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-37) [4d4dcfaf] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ba8b7d16-80ea-4804-8a6e-109dff5c8919, connection: 192.168.50.250:/home/nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 46a7ecd0 2016-12-21 07:01:26,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:01:26,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) HostName = rhev-h1.example.com 2016-12-21 07:01:26,812 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:01:26,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, ConnectStoragePoolVDSCommand, log id: 1021a00c 2016-12-21 07:01:26,812 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-72) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:01:26,812 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-72) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:01:26,825 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-11) [447ae719] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:01:26,839 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-47) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:01:36,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:01:36,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) HostName = rhev-h1.example.com 2016-12-21 07:01:36,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-41) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:01:36,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-41) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:01:36,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 107132ed 2016-12-21 07:02:00,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-37) [4d4dcfaf] FINISH, ConnectStorageServerVDSCommand, return: {ba8b7d16-80ea-4804-8a6e-109dff5c8919=0}, log id: 46a7ecd0 2016-12-21 07:02:00,097 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-14) [744c9d0d] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 7:02 AM 2016-12-21 07:02:36,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:02:36,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) HostName = rhev-h1.example.com 2016-12-21 07:02:36,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:02:36,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-41) FINISH, ConnectStoragePoolVDSCommand, log id: 107132ed 2016-12-21 07:02:36,984 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:02:36,984 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-41) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:02:36,998 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-32) [66bf23] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:02:37,016 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-34) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:02:47,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:02:47,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) HostName = rhev-h1.example.com 2016-12-21 07:02:47,005 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-4) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:02:47,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-4) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:02:47,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 66e0cc7d 2016-12-21 07:03:47,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:03:47,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) HostName = rhev-h1.example.com 2016-12-21 07:03:47,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:03:47,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-4) FINISH, ConnectStoragePoolVDSCommand, log id: 66e0cc7d 2016-12-21 07:03:47,201 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:03:47,201 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-4) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:03:47,222 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-12) [72de891c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:03:47,252 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-23) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:03:57,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:03:57,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) HostName = rhev-h1.example.com 2016-12-21 07:03:57,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:03:57,240 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:03:57,242 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 4bf6eb4d 2016-12-21 07:04:57,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:04:57,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) HostName = rhev-h1.example.com 2016-12-21 07:04:57,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:04:57,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, ConnectStoragePoolVDSCommand, log id: 4bf6eb4d 2016-12-21 07:04:57,394 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:04:57,394 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:04:57,406 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-9) [59fe39b9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:04:57,419 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-10) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:05:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering 3 storage domains 2016-12-21 07:05:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) Autorecovering storage domains id: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 2016-12-21 07:05:00,006 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [29ec098e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 Type: Storage 2016-12-21 07:05:00,006 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [29ec098e] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:05 AM 2016-12-21 07:05:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-26) [6d2d2923] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-26) [6d2d2923] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 06905dd9-33a2-4b92-a73e-8c85a4e0f375, connection: 192.168.50.250:/home/export-nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1f344e70 2016-12-21 07:05:07,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:05:07,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] HostName = rhev-h1.example.com 2016-12-21 07:05:07,413 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:05:07,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-5) [3160be6d] hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:05:07,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 18a9663f 2016-12-21 07:06:00,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-26) [6d2d2923] FINISH, ConnectStorageServerVDSCommand, return: {06905dd9-33a2-4b92-a73e-8c85a4e0f375=0}, log id: 1f344e70 2016-12-21 07:06:00,036 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [29ec098e] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 7:06 AM 2016-12-21 07:06:00,037 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) [29ec098e] Autorecovering storage domains id: f05ef6f9-4307-4dc0-ad59-df471caef3e9 2016-12-21 07:06:00,037 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [5ab00522] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f05ef6f9-4307-4dc0-ad59-df471caef3e9 Type: Storage 2016-12-21 07:06:00,037 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [5ab00522] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:06 AM 2016-12-21 07:06:00,042 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-35) [59433e7a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:06:00,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-35) [59433e7a] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: f9b36ad4-abe6-42eb-a105-d43af2548c68, connection: 192.168.50.200:/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 72b754db 2016-12-21 07:06:00,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-35) [59433e7a] FINISH, ConnectStorageServerVDSCommand, return: {f9b36ad4-abe6-42eb-a105-d43af2548c68=0}, log id: 72b754db 2016-12-21 07:06:00,060 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [5ab00522] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 7:06 AM 2016-12-21 07:06:00,060 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) [5ab00522] Autorecovering storage domains id: 477c507a-1b94-4565-89cf-b6d8f0b98693 2016-12-21 07:06:00,061 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [11c09dd] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:06:00,061 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [11c09dd] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:06 AM 2016-12-21 07:06:00,065 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-32) [2c3bf58f] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:06:00,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-32) [2c3bf58f] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: ba8b7d16-80ea-4804-8a6e-109dff5c8919, connection: 192.168.50.250:/home/nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 66ebf470 2016-12-21 07:06:07,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:06:07,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] HostName = rhev-h1.example.com 2016-12-21 07:06:07,579 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:06:07,579 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] FINISH, ConnectStoragePoolVDSCommand, log id: 18a9663f 2016-12-21 07:06:07,579 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:06:07,579 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-5) [3160be6d] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:06:07,602 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-1) [33030c00] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:06:07,622 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-44) [3160be6d] Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:06:17,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:06:17,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-21) HostName = rhev-h1.example.com 2016-12-21 07:06:17,599 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-21) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:06:17,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-21) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:06:17,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 6c76d2a0 2016-12-21 07:07:00,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-32) [2c3bf58f] FINISH, ConnectStorageServerVDSCommand, return: {ba8b7d16-80ea-4804-8a6e-109dff5c8919=0}, log id: 66ebf470 2016-12-21 07:07:00,082 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-25) [11c09dd] ConnectDomainToStorage. After Connect all hosts to pool. Time:12/21/16 7:07 AM 2016-12-21 07:07:17,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:07:17,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) HostName = rhev-h1.example.com 2016-12-21 07:07:17,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:07:17,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-21) FINISH, ConnectStoragePoolVDSCommand, log id: 6c76d2a0 2016-12-21 07:07:17,794 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:07:17,794 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-21) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:07:17,808 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-24) [4c5dc5c2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:07:17,822 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-31) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:07:27,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:07:27,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = rhev-h1.example.com 2016-12-21 07:07:27,809 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-47) Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:07:27,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:07:27,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 54254651 2016-12-21 07:08:27,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:08:27,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) HostName = rhev-h1.example.com 2016-12-21 07:08:27,973 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:08:27,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, ConnectStoragePoolVDSCommand, log id: 54254651 2016-12-21 07:08:27,973 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:08:27,973 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-47) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:08:27,995 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-38) [7e7fd8c5] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:08:28,011 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-16) Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:08:37,991 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:08:37,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] HostName = rhev-h1.example.com 2016-12-21 07:08:37,992 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:08:38,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-65) [6b7ae976] hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:08:38,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 521dc7d0 2016-12-21 07:09:38,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693']] 2016-12-21 07:09:38,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] HostName = rhev-h1.example.com 2016-12-21 07:09:38,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] Command ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:09:38,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] FINISH, ConnectStoragePoolVDSCommand, log id: 521dc7d0 2016-12-21 07:09:38,154 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] IrsBroker::Failed::GetStoragePoolInfoVDS 2016-12-21 07:09:38,154 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-65) [6b7ae976] Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=6ece980d-7a76-4aa5-a447-22f8fa853d06, msdUUID=477c507a-1b94-4565-89cf-b6d8f0b98693' 2016-12-21 07:09:38,167 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-7-thread-50) [6d6c3536] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 477c507a-1b94-4565-89cf-b6d8f0b98693 Type: Storage 2016-12-21 07:09:38,185 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-7-thread-46) [6b7ae976] Finished reconstruct for pool 6ece980d-7a76-4aa5-a447-22f8fa853d06. Clearing event queue 2016-12-21 07:09:48,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) [52c0d769] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] 2016-12-21 07:09:48,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) [52c0d769] HostName = rhev-h1.example.com 2016-12-21 07:09:48,168 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-18) [52c0d769] Command HSMGetAllTasksStatusesVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2016-12-21 07:09:48,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-18) [52c0d769] hostFromVds::selectedVds - rhev-h1.example.com, spmStatus Unknown_Pool, storage pool dc1 2016-12-21 07:09:48,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler_Worker-18) [52c0d769] START, ConnectStoragePoolVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, vdsId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 6ece980d-7a76-4aa5-a447-22f8fa853d06, masterVersion = 2), log id: 10bc7df 2016-12-21 07:10:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) Autorecovering 3 storage domains 2016-12-21 07:10:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) Autorecovering storage domains id: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 2016-12-21 07:10:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-43) [1732b793] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 0dbb1723-9ca5-4128-b7c5-c7349fcb0dd5 Type: Storage 2016-12-21 07:10:00,008 INFO [org.ovirt.engine.core.bll.storage.ConnectDomainToStorageCommand] (DefaultQuartzScheduler_Worker-43) [1732b793] ConnectDomainToStorage. Before Connect all hosts to pool. Time:12/21/16 7:10 AM 2016-12-21 07:10:00,011 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-7-thread-14) [1ca22127] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2016-12-21 07:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-7-thread-14) [1ca22127] START, ConnectStorageServerVDSCommand(HostName = rhev-h1.example.com, HostId = c3047f52-c8c2-448b-ace4-4da73f69fd14, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 06905dd9-33a2-4b92-a73e-8c85a4e0f375, connection: 192.168.50.250:/home/export-nfs, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 141c43