fence_vmware_rest fails with "Exception: Operation timed out after 5001 milliseconds with 0 out of -1 bytes received"
Issue
- When manually fencing a cluster node with
pcs stonith fence <node name>
the fencing device is able to power off the node but it's not able to power it back on, and consequently the fencing operation is reported as failed:
# pcs stonith fence node1
Error: unable to fence 'node1'
- Alternatively, when manually fencing a node, the fencing device is able to reboot it but the command still gives the same error message; even though the node was able to reboot and rejoin the cluster.
- In the logs the
fence_vmware_rest
agent ends up hitting the following Python exception:
python: detected unhandled Python exception in '/usr/sbin/fence_vmware_rest'
fence_vmware_rest stderr: [ Traceback (most recent call last): ]
fence_vmware_rest stderr: [ File "/usr/sbin/fence_vmware_rest", line 183, in <module> ]
fence_vmware_rest stderr: [ main() ]
fence_vmware_rest stderr: [ File "/usr/sbin/fence_vmware_rest", line 178, in main ]
fence_vmware_rest stderr: [ result = fence_action(conn, options, set_power_status, get_power_status, get_list) ]
fence_vmware_rest stderr: [ File "/usr/share/fence/fencing.py", line 917, in fence_action ]
fence_vmware_rest stderr: [ if not set_multi_power_fn(connection, options, set_power_fn, get_power_fn, sync_set_power_fn): ]
fence_vmware_rest stderr: [ File "/usr/share/fence/fencing.py", line 808, in set_multi_power_fn ]
fence_vmware_rest stderr: [ return async_set_multi_power_fn(connection, options, set_power_fn, get_power_fn, retry_attempts) ]
fence_vmware_rest stderr: [ File "/usr/share/fence/fencing.py", line 771, in async_set_multi_power_fn ]
fence_vmware_rest stderr: [ set_power_fn(connection, options) ]
fence_vmware_rest stderr: [ File "/usr/sbin/fence_vmware_rest", line 31, in set_power_status ]
fence_vmware_rest stderr: [ send_command(conn, "vcenter/vm/{}/power/{}".format(options["id"], action), "POST") ]
fence_vmware_rest stderr: [ File "/usr/sbin/fence_vmware_rest", line 112, in send_command ]
fence_vmware_rest stderr: [ raise Exception(e[1]) ]
fence_vmware_rest stderr: [ Exception: Operation timed out after 5001 milliseconds with 0 out of -1 bytes received ]
- These messages can also be seen during a fencing operation triggered by the cluster itself.
Environment
- Red Hat Enterprise Linux 7 (with High Availability or Resilient Storage Add-on)
- VMware hypervisor
fence_vmware_rest
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.