Ansible Automation Platform upgrade from 2.4 to 2.5 failing in the "Migrate the gateway database schema" task for the Automation Gateway node

Issue

• In an environment with Automation Controller and Automation Private Hub being upgraded from the 2.4 version to the 2.5 version through the setup bundle with RPM, Automation Gateway being installed from zero along with Redis in standalone mode, and Postgres as a managed database along with Automation Controller, the upgrade process fails in the following task:

  TASK [ansible.automation_platform_installer.automationgateway : Migrate the gateway database schema] ***
  [WARNING]: Module remote_tmp /var/lib/ansible-automation-platform/gateway/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually
  fatal: [automationgateway.example.local]: FAILED! => {"changed": false, "cmd": "aap-gateway-manage migrate", "delta": "0:00:01.696548", "end": "2025-01-09 15:05:44.293504", "msg": "non-zero return code", "rc": 1, "start": "2025-01-09 15:05:42.596956", "stderr": "2025-01-09 18:05:43,982 INFO      ansible_base.lib.redis.client Removing setting cluster_error_retry_attempts from connection settings because its invalid for standalone mode\nTraceback (most recent call last):\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 130, in _verify_signature\n    h.verify(data[-32:])\ncryptography.exceptions.InvalidSignature: Signature did not match digest.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/bin/aap-gateway-manage\", line 8, in <module>\n    sys.exit(manage())\n             ^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/__init__.py\", line 22, in manage\n    execute_from_command_line(sys.argv)\n  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line\n    utility.execute()\n  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 436, in execute\n    self.fetch_command(subcommand).run_from_argv(self.argv)\n  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 412, in run_from_argv\n    self.execute(*args, **cmd_options)\n  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py\", line 383, in handle\n    emit_post_migrate_signal(\n  File \"/usr/lib/python3.11/site-packages/django/core/management/sql.py\", line 52, in emit_post_migrate_signal\n    models.signals.post_migrate.send(\n  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 176, in send\n    return [\n           ^\n  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 177, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/apps.py\", line 15, in _initialize_preferences\n    initialize_preferences()\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/utils/preferences.py\", line 138, in initialize_preferences\n    for preference_name in gateway_preference_manager.keys():\n  File \"<frozen _collections_abc>\", line 836, in __iter__\n  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 46, in __iter__\n    return self.all().__iter__()\n           ^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 220, in all\n    a.update(self.load_from_db(cache=True))\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 226, in load_from_db\n    db_prefs = {p.preference.identifier(): p for p in self.queryset}\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 398, in __iter__\n    self._fetch_all()\n  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 1881, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 122, in __iter__\n    obj = model_cls.from_db(\n          ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/models/preference.py\", line 28, in from_db\n    instance.value = ansible_encryption.decrypt_string(instance.value)\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/ansible_base/lib/utils/encryption.py\", line 123, in decrypt_string\n    decrypted_value = smart_str(self.decrypt(encrypted))\n                                ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 89, in decrypt\n    return self._decrypt_data(data, timestamp, time_info)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 148, in _decrypt_data\n    self._verify_signature(data)\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 132, in _verify_signature\n    raise InvalidToken\ncryptography.fernet.InvalidToken", "stderr_lines": ["2025-01-09 18:05:43,982 INFO      ansible_base.lib.redis.client Removing setting cluster_error_retry_attempts from connection settings because its invalid for standalone mode", "Traceback (most recent call last):", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 130, in _verify_signature", "    h.verify(data[-32:])", "cryptography.exceptions.InvalidSignature: Signature did not match digest.", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/bin/aap-gateway-manage\", line 8, in <module>", "    sys.exit(manage())", "             ^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/__init__.py\", line 22, in manage", "    execute_from_command_line(sys.argv)", "  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line", "    utility.execute()", "  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 436, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 412, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py\", line 383, in handle", "    emit_post_migrate_signal(", "  File \"/usr/lib/python3.11/site-packages/django/core/management/sql.py\", line 52, in emit_post_migrate_signal", "    models.signals.post_migrate.send(", "  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 176, in send", "    return [", "           ^", "  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 177, in <listcomp>", "    (receiver, receiver(signal=self, sender=sender, **named))", "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/apps.py\", line 15, in _initialize_preferences", "    initialize_preferences()", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/utils/preferences.py\", line 138, in initialize_preferences", "    for preference_name in gateway_preference_manager.keys():", "  File \"<frozen _collections_abc>\", line 836, in __iter__", "  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 46, in __iter__", "    return self.all().__iter__()", "           ^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 220, in all", "    a.update(self.load_from_db(cache=True))", "             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 226, in load_from_db", "    db_prefs = {p.preference.identifier(): p for p in self.queryset}", "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 398, in __iter__", "    self._fetch_all()", "  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 1881, in _fetch_all", "    self._result_cache = list(self._iterable_class(self))", "                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 122, in __iter__", "    obj = model_cls.from_db(", "          ^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/models/preference.py\", line 28, in from_db", "    instance.value = ansible_encryption.decrypt_string(instance.value)", "                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/ansible_base/lib/utils/encryption.py\", line 123, in decrypt_string", "    decrypted_value = smart_str(self.decrypt(encrypted))", "                                ^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 89, in decrypt", "    return self._decrypt_data(data, timestamp, time_info)", "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 148, in _decrypt_data", "    self._verify_signature(data)", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 132, in _verify_signature", "    raise InvalidToken", "cryptography.fernet.InvalidToken"], "stdout": "Operations to perform:\n  Apply all migrations: aap_gateway_api, admin, auth, contenttypes, dab_activitystream, dab_authentication, dab_oauth2_provider, dab_rbac, dab_resource_registry, dynamic_preferences, flags, oauth2_provider, sessions, social_django\nRunning migrations:\n  No migrations to apply.", "stdout_lines": ["Operations to perform:", "  Apply all migrations: aap_gateway_api, admin, auth, contenttypes, dab_activitystream, dab_authentication, dab_oauth2_provider, dab_rbac, dab_resource_registry, dynamic_preferences, flags, oauth2_provider, sessions, social_django", "Running migrations:", "  No migrations to apply."]}
  
  [...]
  
  PLAY RECAP *********************************************************************
  automationgateway.example.local : ok=189  changed=70   unreachable=0    failed=1    skipped=186  rescued=0    ignored=1   
  automationhub.example.local   : ok=74   changed=7    unreachable=0    failed=0    skipped=97   rescued=0    ignored=0   
  ansiblecontroller.example.local : ok=122  changed=13   unreachable=0    failed=0    skipped=110  rescued=0    ignored=1   
  localhost                  : ok=0    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
  

• Running the command aap-gateway-manage migrate directly into the Automation Gateway node gives the following output:

  # aap-gateway-manage migrate
  Operations to perform:
  Apply all migrations: aap_gateway_api, admin, auth, contenttypes, dab_activitystream, dab_authentication, dab_oauth2_provider, dab_rbac,   dab_resource_registry, dynamic_preferences, flags, oauth2_provider, sessions, social_django
  Running migrations:
  No migrations to apply.
  2025-01-09 18:19:22,818 INFO      ansible_base.lib.redis.client Removing setting cluster_error_retry_attempts from connection settings because its invalid for standalone mode
  Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 130, in _verify_signature
    h.verify(data[-32:])
  cryptography.exceptions.InvalidSignature: Signature did not match digest.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
  File "/usr/bin/aap-gateway-manage", line 8, in <module>
    sys.exit(manage())
             ^^^^^^^^
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/__init__.py", line 22, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 383, in handle
    emit_post_migrate_signal(
  File "/usr/lib/python3.11/site-packages/django/core/management/sql.py", line 52, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
           ^
  File "/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/apps.py", line 15, in _initialize_preferences
    initialize_preferences()
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/utils/preferences.py", line 138, in initialize_preferences
    for preference_name in gateway_preference_manager.keys():
  File "<frozen _collections_abc>", line 836, in __iter__
  File "/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py", line 46, in __iter__
    return self.all().__iter__()
           ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py", line 220, in all
    a.update(self.load_from_db(cache=True))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py", line 226, in load_from_db
    db_prefs = {p.preference.identifier(): p for p in self.queryset}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 398, in __iter__
    self._fetch_all()
  File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 122, in __iter__
    obj = model_cls.from_db(
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/models/preference.py", line 28, in from_db
    instance.value = ansible_encryption.decrypt_string(instance.value)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ansible_base/lib/utils/encryption.py", line 123, in decrypt_string
    decrypted_value = smart_str(self.decrypt(encrypted))
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 89, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 148, in _decrypt_data
    self._verify_signature(data)
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 132, in _verify_signature
    raise InvalidToken
  cryptography.fernet.InvalidToken
  

• The error occurs even in a fresh install of the Automation Gateway node.