Ansible Automation Platform 2.4 から 2.5 へのアップグレードが Automation Gateway ノードの "Migrate the gateway database schema" タスクで失敗する

Issue

• Automation Controller と Automation Private Hub が RPM のセットアップバンドルを通じてバージョン 2.4 からバージョン 2.5 にアップグレードされ、Automation Gateway がスタンドアロンモードの Redis とともにゼロからインストールされ、Postgres が Automation Controller とともにマネージドデータベースとしてインストールされている環境では、次のタスクでアップグレードプロセスが失敗します。

  TASK [ansible.automation_platform_installer.automationgateway : Migrate the gateway database schema] ***
  [WARNING]: Module remote_tmp /var/lib/ansible-automation-platform/gateway/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually
  fatal: [automationgateway.example.local]: FAILED! => {"changed": false, "cmd": "aap-gateway-manage migrate", "delta": "0:00:01.696548", "end": "2025-01-09 15:05:44.293504", "msg": "non-zero return code", "rc": 1, "start": "2025-01-09 15:05:42.596956", "stderr": "2025-01-09 18:05:43,982 INFO      ansible_base.lib.redis.client Removing setting cluster_error_retry_attempts from connection settings because its invalid for standalone mode\nTraceback (most recent call last):\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 130, in _verify_signature\n    h.verify(data[-32:])\ncryptography.exceptions.InvalidSignature: Signature did not match digest.\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/bin/aap-gateway-manage\", line 8, in <module>\n    sys.exit(manage())\n             ^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/__init__.py\", line 22, in manage\n    execute_from_command_line(sys.argv)\n  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line\n    utility.execute()\n  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 436, in execute\n    self.fetch_command(subcommand).run_from_argv(self.argv)\n  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 412, in run_from_argv\n    self.execute(*args, **cmd_options)\n  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 458, in execute\n    output = self.handle(*args, **options)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 106, in wrapper\n    res = handle_func(*args, **kwargs)\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py\", line 383, in handle\n    emit_post_migrate_signal(\n  File \"/usr/lib/python3.11/site-packages/django/core/management/sql.py\", line 52, in emit_post_migrate_signal\n    models.signals.post_migrate.send(\n  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 176, in send\n    return [\n           ^\n  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 177, in <listcomp>\n    (receiver, receiver(signal=self, sender=sender, **named))\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/apps.py\", line 15, in _initialize_preferences\n    initialize_preferences()\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/utils/preferences.py\", line 138, in initialize_preferences\n    for preference_name in gateway_preference_manager.keys():\n  File \"<frozen _collections_abc>\", line 836, in __iter__\n  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 46, in __iter__\n    return self.all().__iter__()\n           ^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 220, in all\n    a.update(self.load_from_db(cache=True))\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 226, in load_from_db\n    db_prefs = {p.preference.identifier(): p for p in self.queryset}\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 398, in __iter__\n    self._fetch_all()\n  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 1881, in _fetch_all\n    self._result_cache = list(self._iterable_class(self))\n                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 122, in __iter__\n    obj = model_cls.from_db(\n          ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/models/preference.py\", line 28, in from_db\n    instance.value = ansible_encryption.decrypt_string(instance.value)\n                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib/python3.11/site-packages/ansible_base/lib/utils/encryption.py\", line 123, in decrypt_string\n    decrypted_value = smart_str(self.decrypt(encrypted))\n                                ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 89, in decrypt\n    return self._decrypt_data(data, timestamp, time_info)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 148, in _decrypt_data\n    self._verify_signature(data)\n  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 132, in _verify_signature\n    raise InvalidToken\ncryptography.fernet.InvalidToken", "stderr_lines": ["2025-01-09 18:05:43,982 INFO      ansible_base.lib.redis.client Removing setting cluster_error_retry_attempts from connection settings because its invalid for standalone mode", "Traceback (most recent call last):", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 130, in _verify_signature", "    h.verify(data[-32:])", "cryptography.exceptions.InvalidSignature: Signature did not match digest.", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", "  File \"/bin/aap-gateway-manage\", line 8, in <module>", "    sys.exit(manage())", "             ^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/__init__.py\", line 22, in manage", "    execute_from_command_line(sys.argv)", "  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 442, in execute_from_command_line", "    utility.execute()", "  File \"/usr/lib/python3.11/site-packages/django/core/management/__init__.py\", line 436, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 412, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 458, in execute", "    output = self.handle(*args, **options)", "             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/core/management/base.py\", line 106, in wrapper", "    res = handle_func(*args, **kwargs)", "          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py\", line 383, in handle", "    emit_post_migrate_signal(", "  File \"/usr/lib/python3.11/site-packages/django/core/management/sql.py\", line 52, in emit_post_migrate_signal", "    models.signals.post_migrate.send(", "  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 176, in send", "    return [", "           ^", "  File \"/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py\", line 177, in <listcomp>", "    (receiver, receiver(signal=self, sender=sender, **named))", "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/apps.py\", line 15, in _initialize_preferences", "    initialize_preferences()", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/utils/preferences.py\", line 138, in initialize_preferences", "    for preference_name in gateway_preference_manager.keys():", "  File \"<frozen _collections_abc>\", line 836, in __iter__", "  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 46, in __iter__", "    return self.all().__iter__()", "           ^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 220, in all", "    a.update(self.load_from_db(cache=True))", "             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py\", line 226, in load_from_db", "    db_prefs = {p.preference.identifier(): p for p in self.queryset}", "               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 398, in __iter__", "    self._fetch_all()", "  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 1881, in _fetch_all", "    self._result_cache = list(self._iterable_class(self))", "                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/django/db/models/query.py\", line 122, in __iter__", "    obj = model_cls.from_db(", "          ^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/aap_gateway_api/models/preference.py\", line 28, in from_db", "    instance.value = ansible_encryption.decrypt_string(instance.value)", "                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib/python3.11/site-packages/ansible_base/lib/utils/encryption.py\", line 123, in decrypt_string", "    decrypted_value = smart_str(self.decrypt(encrypted))", "                                ^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 89, in decrypt", "    return self._decrypt_data(data, timestamp, time_info)", "           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 148, in _decrypt_data", "    self._verify_signature(data)", "  File \"/usr/lib64/python3.11/site-packages/cryptography/fernet.py\", line 132, in _verify_signature", "    raise InvalidToken", "cryptography.fernet.InvalidToken"], "stdout": "Operations to perform:\n  Apply all migrations: aap_gateway_api, admin, auth, contenttypes, dab_activitystream, dab_authentication, dab_oauth2_provider, dab_rbac, dab_resource_registry, dynamic_preferences, flags, oauth2_provider, sessions, social_django\nRunning migrations:\n  No migrations to apply.", "stdout_lines": ["Operations to perform:", "  Apply all migrations: aap_gateway_api, admin, auth, contenttypes, dab_activitystream, dab_authentication, dab_oauth2_provider, dab_rbac, dab_resource_registry, dynamic_preferences, flags, oauth2_provider, sessions, social_django", "Running migrations:", "  No migrations to apply."]}
  
  [...]
  
  PLAY RECAP *********************************************************************
  automationgateway.example.local : ok=189  changed=70   unreachable=0    failed=1    skipped=186  rescued=0    ignored=1   
  automationhub.example.local   : ok=74   changed=7    unreachable=0    failed=0    skipped=97   rescued=0    ignored=0   
  ansiblecontroller.example.local : ok=122  changed=13   unreachable=0    failed=0    skipped=110  rescued=0    ignored=1   
  localhost                  : ok=0    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
  

• Automation Gateway ノードに aap-gateway-manage migrate コマンドを直接実行すると、次のように出力されます。

  # aap-gateway-manage migrate
  Operations to perform:
  Apply all migrations: aap_gateway_api, admin, auth, contenttypes, dab_activitystream, dab_authentication, dab_oauth2_provider, dab_rbac,   dab_resource_registry, dynamic_preferences, flags, oauth2_provider, sessions, social_django
  Running migrations:
  No migrations to apply.
  2025-01-09 18:19:22,818 INFO      ansible_base.lib.redis.client Removing setting cluster_error_retry_attempts from connection settings because its invalid for standalone mode
  Traceback (most recent call last):
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 130, in _verify_signature
    h.verify(data[-32:])
  cryptography.exceptions.InvalidSignature: Signature did not match digest.
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
  File "/usr/bin/aap-gateway-manage", line 8, in <module>
    sys.exit(manage())
             ^^^^^^^^
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/__init__.py", line 22, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 383, in handle
    emit_post_migrate_signal(
  File "/usr/lib/python3.11/site-packages/django/core/management/sql.py", line 52, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
           ^
  File "/usr/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/apps.py", line 15, in _initialize_preferences
    initialize_preferences()
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/utils/preferences.py", line 138, in initialize_preferences
    for preference_name in gateway_preference_manager.keys():
  File "<frozen _collections_abc>", line 836, in __iter__
  File "/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py", line 46, in __iter__
    return self.all().__iter__()
           ^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py", line 220, in all
    a.update(self.load_from_db(cache=True))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/dynamic_preferences/managers.py", line 226, in load_from_db
    db_prefs = {p.preference.identifier(): p for p in self.queryset}
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 398, in __iter__
    self._fetch_all()
  File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/django/db/models/query.py", line 122, in __iter__
    obj = model_cls.from_db(
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/aap_gateway_api/models/preference.py", line 28, in from_db
    instance.value = ansible_encryption.decrypt_string(instance.value)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ansible_base/lib/utils/encryption.py", line 123, in decrypt_string
    decrypted_value = smart_str(self.decrypt(encrypted))
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 89, in decrypt
    return self._decrypt_data(data, timestamp, time_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 148, in _decrypt_data
    self._verify_signature(data)
  File "/usr/lib64/python3.11/site-packages/cryptography/fernet.py", line 132, in _verify_signature
    raise InvalidToken
  cryptography.fernet.InvalidToken
  

• Automation Gateway ノードを新規にインストールする場合でも、エラーが発生します。