fence_azure_arm throws TypeError exception "Failed: unsupported operand type(s) for -=: 'Retry' and 'int'"

Solution In Progress - Updated -

Issue

  • fence_azure_arm logs intermittent errors during monitor operations: fence_azure_arm: Failed: unsupported operand type(s) for -=: 'Retry' and 'int'
Aug  4 13:02:09 node42 fence_azure_arm: Failed: unsupported operand type(s) for -=: 'Retry' and 'int'
Aug  4 13:02:09 node42 fence_azure_arm: Please use '-h' for usage
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [ 2019-08-04 13:02:09,863 ERROR: Failed: unsupported operand type(s) for -=: 'Retry' and 'int' ]
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [  ]
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [ 2019-08-04 13:02:09,863 ERROR: Please use '-h' for usage ]
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [  ]
  • The fencing of a cluster node failed as shown below (with Python traceback enabled for debugging):
Aug  4 13:03:52 node42 python: detected unhandled Python exception in '/usr/sbin/fence_azure_arm'
Aug  4 13:03:52 node42 python: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [ Traceback (most recent call last): ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/sbin/fence_azure_arm", line 247, in <module> ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     main() ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/sbin/fence_azure_arm", line 243, in main ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     result = fence_action([compute_client, network_client], options, set_power_status, get_power_status, get_nodes_list) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/share/fence/fencing.py", line 917, in fence_action ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     if not set_multi_power_fn(connection, options, set_power_fn, get_power_fn, sync_set_power_fn): ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/share/fence/fencing.py", line 808, in set_multi_power_fn ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     return async_set_multi_power_fn(connection, options, set_power_fn, get_power_fn, retry_attempts) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/share/fence/fencing.py", line 771, in async_set_multi_power_fn ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     set_power_fn(connection, options) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/sbin/fence_azure_arm", line 118, in set_power_status ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     compute_client.virtual_machines.deallocate(rgName, vmName) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/azure/mgmt/compute/compute/v2016_04_30_preview/operations/virtual_machines_operations.py", line 554, in deallocate ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     get_long_running_status, long_running_operation_timeout) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/msrestazure/azure_operation.py", line 345, in __init__ ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     self._response = send_cmd() ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/azure/mgmt/compute/compute/v2016_04_30_preview/operations/virtual_machines_operations.py", line 517, in long_running_send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     return self._client.send(request, header_parameters, **operation_config) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/msrest/service_client.py", line 194, in send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     **kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests_oauthlib/oauth2_session.py", line 360, in request ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     headers=headers, data=data, **kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     resp = self.send(prep, **send_kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     r = adapter.send(request, **kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     timeout=timeout ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     _stacktrace=sys.exc_info()[2]) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 226, in increment ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     total -= 1 ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [ TypeError: unsupported operand type(s) for -=: 'Retry' and 'int' ]
Aug  4 13:03:52 node42 stonith-ng[1517]:   error: Operation 'reboot' [126301] (call 3 from crmd.1521) for host 'node43' with device 'rsc_st_azure' returned: -201 (Generic Pacemaker error)
Aug  4 13:03:52 node42 stonith-ng[1517]:  notice: Couldn't find anyone to fence (reboot) node43 with any device
Aug  4 13:03:52 node42 stonith-ng[1517]:   error: Operation reboot of node43 by node42 for crmd.1521@node42.cc3ad369: Timer expired

Environment

  • Red Hat Enterprise Linux 7 (with the High Availability Add-on)
  • fence_azure_arm
  • Microsoft Azure virtual machine

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content