Why is RHEV 3 storage domain frequently marked as "in problem" and then "recovered from problem"?

Solution Verified - Updated -

Issue

  • In /var/log/ovirt-engine/engine.log there are frequent logs as:
2014-03-27 19:03:54,991 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-13) domain $Domain_ID in problem. vds: host1.example.com
2014-03-27 19:04:34,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-37) Domain c$Domain_ID recovered from problem. vds: host1.example.com
2014-03-27 19:04:34,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-37) Domain $Domain_ID has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2016-03-26 16:24:23,288 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-15) Domain $Domain_ID check timeot 51.3 is too big
2016-03-26 16:24:23,292 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-15) Storage Domain $Storage_Domain_Name of pool $Storage_Pool_Name is in problem in host $host_name
2016-03-26 16:24:23,318 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host $host_name reports about one of the Active Storage Domains as Problematic.
2016-03-26 16:24:23,343 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-79) [27d897cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: $ID Type: VDS
2016-03-26 16:24:23,362 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [27d897cc] START, SetVdsStatusVDSCommand(HostName = $host_name, HostId = $host_ID, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5d95cfd

Environment

  • Red Hat Enterprise Virtualization 3.x
    • Red Hat Enterprise Virtualization Hypervisor (RHEV-H) 6.x and 7.x
    • Red Hat Enterprise Linux 6 and 7 Hosts
    • Hosts/Hypervisors with vdsm version 4.17 or lower.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In