Execution Environment page in Private Automation Hub fails to load with HTTP 500

Solution Verified - Updated -

Environment

  • Ansible Automation Platform 2.2
  • Private Automation Hub - 4.5

Issue

  • The Execution Environment (web UI) pages of Private Automation Hub won't load and trigger HTTP 500 status codes after upgrading Ansible Automation Platform to 2.2

Resolution

  • Run the following command on your AH node:

       # pulpcore-manager datarepair-2327 --dry-run

  • It should return the output like below:

       Remotes with un-encrypted fields: 157
   Remotes encrypted multiple times: 200

  • If either of those numbers are larger than 0 (and if user have encountered this error, they should be), run the following command:

      # pulpcore-manager datarepair-2327

  • It will result in the output like this: Finished. (xxx remotes fixed)

Root Cause

  • The issue is with the db migration between pulpcore versions and encryption of some fields in the database. The migration/encryption sometime fails and the original content remains in the database which makes decryption failed.

    pulpcore-manager datarepair-2327 --dry-run : Just validates the content and check if there is any un-encrypted fields

    pulpcore-manager datarepair-2327 : To Fix a migration mistake resulting in improperly encrypted remote fields

Diagnostic Steps

  • The following output can be found in /var/log/messages in Private Automation Hub:

    Jul 01 16:11:41 automationhub gunicorn[5681]: pulp [None]: django.request:ERROR: Internal Server Error: /api/galaxy/_ui/v1/execution-environments/repositories/
Jul 01 16:11:41 automationhub gunicorn[5681]: Traceback (most recent call last):
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
Jul 01 16:11:41 automationhub gunicorn[5681]:     rel_obj = self.field.get_cached_value(instance)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
Jul 01 16:11:41 automationhub gunicorn[5681]:     return instance._state.fields_cache[cache_name]
Jul 01 16:11:41 automationhub gunicorn[5681]: KeyError: 'remote'
Jul 01 16:11:41 automationhub gunicorn[5681]: During handling of the above exception, another exception occurred:
Jul 01 16:11:41 automationhub gunicorn[5681]: Traceback (most recent call last):
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib64/python3.9/site-packages/cryptography/fernet.py", line 119, in _verify_signature
Jul 01 16:11:41 automationhub gunicorn[5681]:     h.verify(data[-32:])
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/primitives/hmac.py", line 74, in verify
Jul 01 16:11:41 automationhub gunicorn[5681]:     ctx.verify(signature)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 75, in verify
Jul 01 16:11:41 automationhub gunicorn[5681]:     raise InvalidSignature("Signature did not match digest.")
Jul 01 16:11:41 automationhub gunicorn[5681]: cryptography.exceptions.InvalidSignature: Signature did not match digest.
Jul 01 16:11:41 automationhub gunicorn[5681]: During handling of the above exception, another exception occurred:
Jul 01 16:11:41 automationhub gunicorn[5681]: Traceback (most recent call last):
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
Jul 01 16:11:41 automationhub gunicorn[5681]:     response = get_response(request)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
Jul 01 16:11:41 automationhub gunicorn[5681]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Jul 01 16:11:41 automationhub gunicorn[5681]:     return view_func(*args, **kwargs)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
Jul 01 16:11:41 automationhub gunicorn[5681]:     return self.dispatch(request, *args, **kwargs)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
Jul 01 16:11:41 automationhub gunicorn[5681]:     return qs.get(self.field.get_reverse_related_filter(instance))
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 431, in get
Jul 01 16:11:41 automationhub gunicorn[5681]:     num = len(clone)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
Jul 01 16:11:41 automationhub gunicorn[5681]:     self._fetch_all()
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Jul 01 16:11:41 automationhub gunicorn[5681]:     self._result_cache = list(self._iterable_class(self))
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 68, in __iter__
Jul 01 16:11:41 automationhub gunicorn[5681]:     for row in compiler.results_iter(results):
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Jul 01 16:11:41 automationhub gunicorn[5681]:     value = converter(value, expression, connection)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib/python3.9/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Jul 01 16:11:41 automationhub gunicorn[5681]:     return force_str(self._fernet.decrypt(force_bytes(value)))
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib64/python3.9/site-packages/cryptography/fernet.py", line 80, in decrypt
Jul 01 16:11:41 automationhub gunicorn[5681]:     return self._decrypt_data(data, timestamp, time_info)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib64/python3.9/site-packages/cryptography/fernet.py", line 137, in _decrypt_data
Jul 01 16:11:41 automationhub gunicorn[5681]:     self._verify_signature(data)
Jul 01 16:11:41 automationhub gunicorn[5681]:   File "/usr/lib64/python3.9/site-packages/cryptography/fernet.py", line 121, in _verify_signature
Jul 01 16:11:41 automationhub gunicorn[5681]:     raise InvalidToken
Jul 01 16:11:41 automationhub gunicorn[5681]: cryptography.fernet.InvalidToken

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments