Why is RHEV 3 storage domain frequently marked as "in problem" and then "recovered from problem"?

Solution Verified - Updated -

Issue

  • In /var/log/ovirt-engine/engine.log there are frequent logs as:
2014-03-27 19:03:54,991 WARN  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-13) domain $Domain_ID in problem. vds: host1.example.com
2014-03-27 19:04:34,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-37) Domain c$Domain_ID recovered from problem. vds: host1.example.com
2014-03-27 19:04:34,220 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-37) Domain $Domain_ID has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer.
2016-03-26 16:24:23,288 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (org.ovirt.thread.pool-7-thread-15) Domain $Domain_ID check timeot 51.3 is too big
2016-03-26 16:24:23,292 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-7-thread-15) Storage Domain $Storage_Domain_Name of pool $Storage_Pool_Name is in problem in host $host_name
2016-03-26 16:24:23,318 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-15) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host $host_name reports about one of the Active Storage Domains as Problematic.
2016-03-26 16:24:23,343 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-79) [27d897cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: $ID Type: VDS
2016-03-26 16:24:23,362 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [27d897cc] START, SetVdsStatusVDSCommand(HostName = $host_name, HostId = $host_ID, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5d95cfd

Environment

  • Red Hat Enterprise Virtualization 3.x
    • Red Hat Enterprise Virtualization Hypervisor (RHEV-H) 6.x and 7.x
    • Red Hat Enterprise Linux 6 and 7 Hosts
    • Hosts/Hypervisors with vdsm version 4.17 or lower.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content