fence_azure_arm throws TypeError exception "Failed: unsupported operand type(s) for -=: 'Retry' and 'int'"

Solution In Progress - Updated -

Issue

  • fence_azure_arm logs intermittent errors during monitor operations: fence_azure_arm: Failed: unsupported operand type(s) for -=: 'Retry' and 'int'
Aug  4 13:02:09 node42 fence_azure_arm: Failed: unsupported operand type(s) for -=: 'Retry' and 'int'
Aug  4 13:02:09 node42 fence_azure_arm: Please use '-h' for usage
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [ 2019-08-04 13:02:09,863 ERROR: Failed: unsupported operand type(s) for -=: 'Retry' and 'int' ]
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [  ]
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [ 2019-08-04 13:02:09,863 ERROR: Please use '-h' for usage ]
Aug  4 13:02:09 node42 stonith-ng[1517]: warning: fence_azure_arm[120876] stderr: [  ]
  • The fencing of a cluster node failed as shown below (with Python traceback enabled for debugging):
Aug  4 13:03:52 node42 python: detected unhandled Python exception in '/usr/sbin/fence_azure_arm'
Aug  4 13:03:52 node42 python: can't communicate with ABRT daemon, is it running? [Errno 2] No such file or directory
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [ Traceback (most recent call last): ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/sbin/fence_azure_arm", line 247, in <module> ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     main() ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/sbin/fence_azure_arm", line 243, in main ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     result = fence_action([compute_client, network_client], options, set_power_status, get_power_status, get_nodes_list) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/share/fence/fencing.py", line 917, in fence_action ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     if not set_multi_power_fn(connection, options, set_power_fn, get_power_fn, sync_set_power_fn): ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/share/fence/fencing.py", line 808, in set_multi_power_fn ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     return async_set_multi_power_fn(connection, options, set_power_fn, get_power_fn, retry_attempts) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/share/fence/fencing.py", line 771, in async_set_multi_power_fn ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     set_power_fn(connection, options) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/sbin/fence_azure_arm", line 118, in set_power_status ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     compute_client.virtual_machines.deallocate(rgName, vmName) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/azure/mgmt/compute/compute/v2016_04_30_preview/operations/virtual_machines_operations.py", line 554, in deallocate ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     get_long_running_status, long_running_operation_timeout) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/msrestazure/azure_operation.py", line 345, in __init__ ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     self._response = send_cmd() ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/azure/mgmt/compute/compute/v2016_04_30_preview/operations/virtual_machines_operations.py", line 517, in long_running_send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     return self._client.send(request, header_parameters, **operation_config) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/msrest/service_client.py", line 194, in send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     **kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests_oauthlib/oauth2_session.py", line 360, in request ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     headers=headers, data=data, **kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     resp = self.send(prep, **send_kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     r = adapter.send(request, **kwargs) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     timeout=timeout ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     _stacktrace=sys.exc_info()[2]) ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 226, in increment ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [     total -= 1 ]
Aug  4 13:03:52 node42 stonith-ng[1517]: warning: fence_azure_arm[126301] stderr: [ TypeError: unsupported operand type(s) for -=: 'Retry' and 'int' ]
Aug  4 13:03:52 node42 stonith-ng[1517]:   error: Operation 'reboot' [126301] (call 3 from crmd.1521) for host 'node43' with device 'rsc_st_azure' returned: -201 (Generic Pacemaker error)
Aug  4 13:03:52 node42 stonith-ng[1517]:  notice: Couldn't find anyone to fence (reboot) node43 with any device
Aug  4 13:03:52 node42 stonith-ng[1517]:   error: Operation reboot of node43 by node42 for crmd.1521@node42.cc3ad369: Timer expired

Environment

  • Red Hat Enterprise Linux 7 (with the High Availability Add-on)
  • fence_azure_arm
  • Microsoft Azure virtual machine

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In