fence_vmware_rest fails with "Exception: Operation timed out after 5001 milliseconds with 0 out of -1 bytes received"

Solution Verified - Updated -

Issue

  • When manually fencing a cluster node with pcs stonith fence <node name> the fencing device is able to power off the node but it's not able to power it back on, and consequently the fencing operation is reported as failed:
# pcs stonith fence node1
Error: unable to fence 'node1'
  • Alternatively, when manually fencing a node, the fencing device is able to reboot it but the command still gives the same error message; even though the node was able to reboot and rejoin the cluster.
  • In the logs the fence_vmware_rest agent ends up hitting the following Python exception:
python: detected unhandled Python exception in '/usr/sbin/fence_vmware_rest'
fence_vmware_rest stderr: [ Traceback (most recent call last): ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 183, in <module> ]
fence_vmware_rest stderr: [     main() ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 178, in main ]
fence_vmware_rest stderr: [     result = fence_action(conn, options, set_power_status, get_power_status, get_list) ]
fence_vmware_rest stderr: [   File "/usr/share/fence/fencing.py", line 917, in fence_action ]
fence_vmware_rest stderr: [     if not set_multi_power_fn(connection, options, set_power_fn, get_power_fn, sync_set_power_fn): ]
fence_vmware_rest stderr: [   File "/usr/share/fence/fencing.py", line 808, in set_multi_power_fn ]
fence_vmware_rest stderr: [     return async_set_multi_power_fn(connection, options, set_power_fn, get_power_fn, retry_attempts) ]
fence_vmware_rest stderr: [   File "/usr/share/fence/fencing.py", line 771, in async_set_multi_power_fn ]
fence_vmware_rest stderr: [     set_power_fn(connection, options) ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 31, in set_power_status ]
fence_vmware_rest stderr: [     send_command(conn, "vcenter/vm/{}/power/{}".format(options["id"], action), "POST") ]
fence_vmware_rest stderr: [   File "/usr/sbin/fence_vmware_rest", line 112, in send_command ]
fence_vmware_rest stderr: [     raise Exception(e[1]) ]
fence_vmware_rest stderr: [ Exception: Operation timed out after 5001 milliseconds with 0 out of -1 bytes received ]
  • These messages can also be seen during a fencing operation triggered by the cluster itself.

Environment

  • Red Hat Enterprise Linux 7 (with High Availability or Resilient Storage Add-on)
  • VMware hypervisor
  • fence_vmware_rest

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content