sanlock errors prevent RHV storage domain from activating

Issue

• Primary storage domain will not activate
• Hypervisors flip between Normal and contending, but never achieve SPM
• On the hypervisor, /var/log/sanlock.log contains errors like:

2014-08-27 14:26:42+0000 2244 [14497]: s30 add_lockspace fail result -223  #<-- corrupted ids lockspace

or:

2014-08-26 20:50:23+0000 93116 [14500]: r12 cmd_acquire 3,11,32306 acquire_token -243 #<-- corrupted leases lockspace

or:

verify_leader 2 wrong space name db3200ed-f6ad-48f3-b987-2efae14c7ea3 db3600ed-f6ad-48f3-b987-2efae14c7ea7 /dev/db3600ed-f6ad-48f3-b987-2efae14c7ea7/ids #<-- incorrect space_name

• On the hypervisor, /var/log/vdsm/vdsm.log contains errors like:

Thread-14::DEBUG::2014-08-27 13:46:33,040::clusterlock::226::SANLock::(hasHostId) Unable to inquire sanlock lockspace status, returning False

• On the RHEV Manager, /var/log/ovirt-engine/engine.log contains errors like:

2014-08-27 09:55:53,144 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-88) [69569e6a] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot acquire host id, code = 661

or:

2014-08-27 10:04:16,744 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [e26044d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = Cannot obtain lock, code = 651