The error "ImportError: /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so: invalid ELF header" occurs when use ansible to remotely control one of the nodes in OCP

Solution Verified - Updated -

Environment

  • OpenShift Container Platform
    • 4

Issue

The following error occurs when use ansible to remotely control one of the nodes in OCP

ansible [core 2.14.2]
  config file = /home/aixadm/ansible/ansible.cfg
  configured module search path = ['/home/aixadm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /home/aixadm/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.2 (main, Feb 17 2023, 09:28:16) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
Using /home/aixadm/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so: invalid ELF header
twi900corewrkt7 | FAILED! => {
    "changed": false,
    "module_stderr": "Shared connection to twi900corewrkt7 closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/var/home/core/.ansible/tmp/ansible-tmp-1704444303.7320876-1457593-258479084943589/AnsiballZ_command.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/var/home/core/.ansible/tmp/ansible-tmp-1704444303.7320876-1457593-258479084943589/AnsiballZ_command.py\", line 24, in _ansiballz_main\r\n    import base64\r\n  File \"/usr/lib64/python3.6/base64.py\", line 10, in <module>\r\n    import struct\r\n  File \"/usr/lib64/python3.6/struct.py\", line 13, in <module>\r\n    from _struct import *\r\nImportError: /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so: invalid ELF header\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

Resolution

Delete the unexpected nodes out of the cluster and reinstall the OS, then join to the cluster.

Root Cause

The error means the /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so file is not a valid shared library and this library is damaged.

Diagnostic Steps

$ ansible <hostname> -m shell -a ‘uname -r’ -vv

ansible [core 2.14.2]
  config file = /home/aixadm/ansible/ansible.cfg
  configured module search path = ['/home/aixadm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /home/aixadm/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.2 (main, Feb 17 2023, 09:28:16) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
Using /home/aixadm/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so: invalid ELF header
twi900corewrkt7 | FAILED! => {
    "changed": false,
    "module_stderr": "Shared connection to twi900corewrkt7 closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/var/home/core/.ansible/tmp/ansible-tmp-1704444303.7320876-1457593-258479084943589/AnsiballZ_command.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/var/home/core/.ansible/tmp/ansible-tmp-1704444303.7320876-1457593-258479084943589/AnsiballZ_command.py\", line 24, in _ansiballz_main\r\n    import base64\r\n  File \"/usr/lib64/python3.6/base64.py\", line 10, in <module>\r\n    import struct\r\n  File \"/usr/lib64/python3.6/struct.py\", line 13, in <module>\r\n    from _struct import *\r\nImportError: /usr/lib64/python3.6/lib-dynload/_struct.cpython-36m-x86_64-linux-gnu.so: invalid ELF header\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments