Storage domain is down in RHEV Manager after removing a lun from external storage.

Solution Verified - Updated -

Issue

  • My production storage domain is down
  • After removing a FibreChannel lun, all storage domains are showing RED and down in RHEVManager
  • In the events of the manager I can see the following:
Failed to activate Storage Domain DB-DATA (Data Center EXAMPLE) 

No valid Data Storage Domains are available in Data Center EXAMPLE (please check your storage infrastructure).

Invalid status on Data Center EXAMPLE. Setting Data Center status to Non-Responsive (On host hypervisor01, Error: Network error during communication with the Host.).
  • The engine log contained;
2014-05-29 14:07:33,234 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [66f352b8] domain cbf5910f-9993-4bd4-97e4-ccc23656404f:DB-DATA in problem. vds: host02
2014-05-29 14:07:33,235 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [66f352b8] domain 03a23919-b1e8-4542-8287-8cb855aba8cd:db-export in problem. vds: host02
2014-05-29 14:07:33,235 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-40) [66f352b8] domain 072b0933-1497-4368-a78c-36c2059a44a4:db-iso in problem. vds: host02

................. etc., etc., .................

2014-05-29 14:04:28,449 ERROR [org.ovirt.engine.core.bll.storage.GetDeviceListQuery] (http-/0.0.0.0:8443-6) Query GetDeviceListQuery failed. Exception message is VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSNetworkException: 
2014-05-29 14:04:40,852 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (http-/0.0.0.0:8443-28) XML RPC error in command GetDeviceListVDS ( HostName = host02 ), the error was: java.util.concurrent.TimeoutException, TimeoutException:  
2014-05-29 14:04:40,852 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (http-/0.0.0.0:8443-28) FINISH, GetDeviceListVDSCommand, log id: 789c88f9
2014-05-29 14:04:40,852 ERROR [org.ovirt.engine.core.bll.storage.GetDeviceListQuery] (http-/0.0.0.0:8443-28) Query GetDeviceListQuery failed. Exception message is VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSNetworkException: 
2014-05-29 14:06:16,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (http-/0.0.0.0:8443-9) XML RPC error in command GetDeviceListVDS ( HostName = host02 ), the error was: java.util.concurrent.TimeoutException, TimeoutException:  
2014-05-29 14:06:16,962 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (http-/0.0.0.0:8443-9) FINISH, GetDeviceListVDSCommand, log id: 22a79b03
2014-05-29 14:06:16,962 ERROR [org.ovirt.engine.core.bll.storage.GetDeviceListQuery] (http-/0.0.0.0:8443-9) Query GetDeviceListQuery failed. Exception message is VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSNetworkException: 
2014-05-29 14:06:19,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (http-/0.0.0.0:8443-5) XML RPC error in command GetDeviceListVDS ( HostName = host02 ), the error was: java.util.concurrent.TimeoutException, TimeoutException:  
2014-05-29 14:06:19,550 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetDeviceListVDSCommand] (http-/0.0.0.0:8443-5) FINISH, GetDeviceListVDSCommand, log id: 419a1466
2014-05-29 14:06:19,550 ERROR [org.ovirt.engine.core.bll.storage.GetDeviceListQuery] (http-/0.0.0.0:8443-5) Query GetDeviceListQuery failed. Exception message is VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSNetworkException:
  • The vdsm logs on the host show that the Storage Domain Monitor threads stopped responding;
$ grep DomainMonitorThread vdsm.log* | tail -n 2
Thread-19::DEBUG::2014-05-29 14:00:23,610::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 03a23919-b1e8-4542-8287-8cb855aba8cd
Thread-19::DEBUG::2014-05-29 14:05:25,198::domainMonitor::192::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 03a23919-b1e8-4542-8287-8cb855aba8cd

Environment

  • Red Hat Enterprise Virtualization 3.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.