Failed to AttachStorageDomainVDS, error = Resource timeout

Issue

• RHEV-H went into Non Operational state after it got "Failed to AttachStorageDomainVDS, error = Resource timeout" error.

from vdc-log.txt

    2011-12-01 21:14:23,344::resource::667::irs::Owner.releaseAll requests [] resources []
    Thread-1764400::INFO::2011-12-01 21:14:23,346::dispatcher::95::irs::Run and protect: attachStorageDomain, args: ( sdUUID=476c00d6-ab67-4b5d-8e91-46f93be6dd26 spUUID=d3721b7f-2e98-4b87-97ef-ae54fcd07a51)
    2011-12-01 21:14:23,347::task::574::irs::Task b7060b4b-81b5-4fd3-aa86-a0ee137e158f: moving from state init -> state preparing
    2011-12-01 21:14:23,348::resource::157::irs::resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 after incref ref 4
    2011-12-01 21:14:23,349::resource::295::irs::Resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51: b7060b4b-81b5-4fd3-aa86-a0ee137e158f acquire exclusive (120000)
    2011-12-01 21:14:23,350::resource::301::irs::Resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 - lockstate shared
    2011-12-01 21:14:23,350::resource::241::irs::Resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 - b7060b4b-81b5-4fd3-aa86-a0ee137e158f wait for exclusive access 120000 ms
    2011-12-01 21:14:23,351::resource::587::irs::Owner b7060b4b-81b5-4fd3-aa86-a0ee137e158f: req _registered b7060b4b-81b5-4fd3-aa86-a0ee137e158f: request for Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 - exclusive: waiting
    2011-12-01 21:14:23,352::task::574::irs::Task b7060b4b-81b5-4fd3-aa86-a0ee137e158f: _resourcesAcquired: Storage.d3721b7f-2e98-4b87-97ef-ae54fcd07a51 (exclusive)
    2011-12-01 21:14:23,353::task::574::irs::Task b7060b4b-81b5-4fd3-aa86-a0ee137e158f: moving from state preparing -> state blocked
    2011-12-01 21:14:23,354::task::574::irs::Task b7060b4b-81b5-4fd3-aa86-a0ee137e158f: ref 1 aborting False
    Dummy-1619739::DEBUG::2011-12-01 21:14:23,684::misc::119::irs::SUCCESS: <err> = '2000+0 records in\n2000+0 records out\n1024000 bytes (1.0 MB) copied, 0.497166 seconds, 2.1 MB/s\n'; <rc> = 0
    2011-12-01 21:16:23,356::resource::569::irs::Owner b7060b4b-81b5-4fd3-aa86-a0ee137e158f: req _canceled b7060b4b-81b5-4fd3-aa86-a0ee137e158f: request for Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 - exclusive: waiting
    2011-12-01 21:16:23,357::resource::260::irs::Resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 - b7060b4b-81b5-4fd3-aa86-a0ee137e158f acquired False
    Thread-1764400::ERROR::2011-12-01 21:16:23,358::misc::62::irs::Resource timeout: ('b7060b4b-81b5-4fd3-aa86-a0ee137e158f: resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 after 120000 ms',)
    Thread-1764400::ERROR::2011-12-01 21:16:23,399::misc::63::irs::Traceback (most recent call last):
      File "/usr/share/vdsm/storage/task.py", line 974, in _run
        return fn(*args, **kargs)
      File "/usr/share/vdsm/storage/spm.py", line 107, in run
        return self.func(*args, **kwargs)
      File "/usr/share/vdsm/storage/spm.py", line 939, in public_attachStorageDomain
        vars.task.getExclusiveLock(STORAGE, spUUID)
      File "/usr/share/vdsm/storage/task.py", line 1413, in getExclusiveLock
        self.resOwner.acquire(namespace, resName, resource.LockType.exclusive, timeout)
      File "/usr/share/vdsm/storage/resource.py", line 620, in acquire
        raise se.ResourceTimeout("%s: resource %s after %s ms" % (str(self), resource, timeout_ms))
    ResourceTimeout: Resource timeout: ('b7060b4b-81b5-4fd3-aa86-a0ee137e158f: resource Storage/d3721b7f-2e98-4b87-97ef-ae54fcd07a51 after 120000 ms',)