fence_vmware_rest fails with "Exception: Operation timed out after 5001 milliseconds with 0 out of -1 bytes received"

Solution Verified - Updated -

Issue

  • When manually fencing a cluster node with pcs stonith fence <node name> the fencing device is able to power off the node but it's not able to power it back on, and consequently the fencing operation is reported as failed:
# pcs stonith fence node1
Error: unable to fence 'node1'
  • Alternatively, when manually fencing a node, the fencing device is able to reboot it but the command still gives the same error message; even though the node was able to reboot and rejoin the cluster.
  • In the logs the fence_vmware_rest agent ends up hitting the following Python exception:
python: detected unhandled Python exception in '/usr/sbin/fence_vmware_rest'
fence_vmware_rest stderr: [ Traceback (most recent call last): ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 183, in <module> ]
fence_vmware_rest stderr: [     main() ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 178, in main ]
fence_vmware_rest stderr: [     result = fence_action(conn, options, set_power_status, get_power_status, get_list) ]
fence_vmware_rest stderr: [   File "/usr/share/fence/fencing.py", line 917, in fence_action ]
fence_vmware_rest stderr: [     if not set_multi_power_fn(connection, options, set_power_fn, get_power_fn, sync_set_power_fn): ]
fence_vmware_rest stderr: [   File "/usr/share/fence/fencing.py", line 808, in set_multi_power_fn ]
fence_vmware_rest stderr: [     return async_set_multi_power_fn(connection, options, set_power_fn, get_power_fn, retry_attempts) ]
fence_vmware_rest stderr: [   File "/usr/share/fence/fencing.py", line 771, in async_set_multi_power_fn ]
fence_vmware_rest stderr: [     set_power_fn(connection, options) ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 31, in set_power_status ]
fence_vmware_rest stderr: [     send_command(conn, "vcenter/vm/{}/power/{}".format(options["id"], action), "POST") ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 112, in send_command ]
fence_vmware_rest stderr: [     raise Exception(e[1]) ]
fence_vmware_rest stderr: [ Exception: Operation timed out after 5001 milliseconds with 0 out of -1 bytes received ]
  • These messages can also be seen during a fencing operation triggered by the cluster itself.

Environment

  • Red Hat Enterprise Linux 7 (with High Availability or Resilient Storage Add-on)
  • VMware hypervisor
  • fence_vmware_rest

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In