sanlock errors prevent RHEV storage domain from activating

Solution Verified - Updated -

Issue

  • Primary storage domain will not activate
  • Hypervisors flip between Normal and contending, but never achieve SPM
  • On the hypervisor, /var/log/sanlock.log contains errors like:
2014-08-27 14:26:42+0000 2244 [14497]: s30 add_lockspace fail result -223  #<-- corrupted ids lockspace

or:

2014-08-26 20:50:23+0000 93116 [14500]: r12 cmd_acquire 3,11,32306 acquire_token -243 #<-- corrupted leases lockspace

or:

verify_leader 2 wrong space name db3200ed-f6ad-48f3-b987-2efae14c7ea3 db3600ed-f6ad-48f3-b987-2efae14c7ea7 /dev/db3600ed-f6ad-48f3-b987-2efae14c7ea7/ids #<-- incorrect space_name
  • On the hypervisor, /var/log/vdsm/vdsm.log contains errors like:
Thread-14::DEBUG::2014-08-27 13:46:33,040::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False
  • On the RHEV Manager, /var/log/ovirt-engine/engine.log contains errors like:
2014-08-27 09:55:53,144 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) [69569e6a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot acquire host id, code = 661

or:

2014-08-27 10:04:16,744 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [e26044d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock, code = 651

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.x
  • Red Hat Enterprise Virtualization Hypervisor (RHEVH or RHEL+H)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content