Accessing Ansible Automation Platform UI fails with "Internal Server Error".

Solution Verified - Updated -

Environment

  • Ansible Automation Platform 2.4

Issue

  • On the clustered environment, one of the controller UI shows:

    Internal Server Error

Resolution

  • Verify the tower_version set on all the controllers along with the permissions :

    # su - awx
$ cat /var/lib/awx/.tower_version
$ ls -lhtZ /var/lib/awx/.tower_version

  • Once the version mismatch is confirmed, execute the setup installer:

    # ./setup.sh

Root Cause

  • tower_version mismatch on clustered nodes causing tower-processes:awx-daphne service.

    rsyslog-4xx-recovery                     RUNNING   pid 430261, uptime 16:17:00
tower-processes:awx-callback-receiver    RUNNING   pid 430263, uptime 16:17:00
tower-processes:awx-daphne               FATAL     Exited too quickly (process log may have details)
tower-processes:awx-dispatcher           RUNNING   pid 430262, uptime 16:17:00
tower-processes:awx-rsyslog-configurer   RUNNING   pid 430268, uptime 16:17:00
tower-processes:awx-rsyslogd             RUNNING   pid 430313, uptime 16:16:57
tower-processes:awx-uwsgi                RUNNING   pid 430264, uptime 16:17:00
tower-processes:awx-ws-heartbeat         RUNNING   pid 430269, uptime 16:17:00
tower-processes:awx-wsrelay              RUNNING   pid 430266, uptime 16:17:00

Diagnostic Steps

  • Following logs can be seen in sosreport:

    Traceback (most recent call last):
File "/var/lib/awx/venv/awx/bin/daphne", line 8, in <module>
sys.exit(CommandLineInterface.entrypoint())
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/daphne/cli.py", line 170, in entrypoint
cls().run(sys.argv[1:])
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/daphne/cli.py", line 232, in run
application = import_by_path(args.application)
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/daphne/utils.py", line 12, in import_by_path
target = importlib.import_module(module_path)
File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/asgi.py", line 34, in <module>
raise Exception("Missing or incorrect metadata for controller version.  Ensure controller was installed using the setup playbook.") from e
Exception: Missing or incorrect metadata for controller version.  Ensure controller was installed using the setup playbook.

  • nginx logs:

    2024/11/19 09:14:36 [crit] 120170#0: *62 connect() to unix:/var/run/tower/daphne.sock failed (2: No such file or directory) while connecting to upstream, client: XXX.XXX.XXX.XXX, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://unix:/var/run/tower/daphne.sock:/websocket/", host: "<Controller01>.example.com"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments