Red Hat Satellite upgrade to 6.16 failed and lost the session of the server due to discrepancy in postgresql version.

Issue

• satellite-maintain upgrade run gets stuck in the following step:

  2025-08-06 05:01:09 [DEBUG ] [root] Warning: /etc/foreman-installer/custom-hiera.yaml: file does not contain a valid yaml hash
  
  2025-08-06 05:01:09 [INFO  ] [root] ... finished loading default values from puppet modules.
  2025-08-06 05:01:09 [INFO  ] [pre_values] Executing hooks in group pre_values
  2025-08-06 05:01:09 [INFO  ] [pre_values] All hooks in group pre_values finished
  2025-08-06 05:01:09 [DEBUG ] [root] Running installer with args [[]]
  2025-08-06 05:01:09 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
  2025-08-06 05:01:09 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/dns.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/encoding.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/hostname.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/ipv6
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/noexec
  2025-08-06 05:01:10 [INFO  ] [pre_validations] Executing hooks in group pre_validations
  2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/01-reset_data.rb returned nil
  2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/12-check_certs_tar.rb returned nil
  

• Post an unsuccessful upgrade, the satellite-installer is stuck at below step :

   # satellite-installer -S satellite
       2025-02-05 15:14:59 [NOTICE] [root] Loading installer configuration. This will take some time.
       2025-02-05 15:15:03 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
       2025-02-05 15:15:03 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. 
       See --full-help for definitions.
  

  And logs similar to: /var/log/foreman-installer/satellite.log

  
  2025-08-06 05:01:09 [INFO  ] [root] ... finished loading default values from puppet modules.
  2025-08-06 05:01:09 [INFO  ] [pre_values] Executing hooks in group pre_values
  2025-08-06 05:01:09 [INFO  ] [pre_values] All hooks in group pre_values finished
  2025-08-06 05:01:09 [DEBUG ] [root] Running installer with args [[]]
  2025-08-06 05:01:09 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
  2025-08-06 05:01:09 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/dns.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/encoding.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/hostname.rb
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/ipv6
  2025-08-06 05:01:10 [DEBUG ] [checks] Executing checker: /usr/share/foreman-installer/checks/noexec
  2025-08-06 05:01:10 [INFO  ] [pre_validations] Executing hooks in group pre_validations
  2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/01-reset_data.rb returned nil
  2025-08-06 05:01:10 [DEBUG ] [pre_validations] Hook /usr/share/foreman-installer/hooks/pre_validations/12-check_certs_tar.rb returned nil
  

• Multiple services in failed state :

   # satellite-maintain service status -b
       Running Status Services
       ===================================================
       Get status of applicable services:
  
       Displaying the following service(s):
        redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@7.service, pulpcore-worker@8.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
         / displaying redis                                 [OK]
        / displaying postgresql                            [FAIL]
       - displaying pulpcore-api                          [FAIL]
       - displaying pulpcore-content                      [OK]
       - displaying pulpcore-worker@1.service             [FAIL]
       - displaying pulpcore-worker@2.service             [OK]
       - displaying pulpcore-worker@3.service             [FAIL]
       - displaying pulpcore-worker@4.service             [FAIL]
       - displaying pulpcore-worker@5.service             [FAIL]
       - displaying pulpcore-worker@6.service             [FAIL]
       - displaying pulpcore-worker@7.service             [OK]
       - displaying pulpcore-worker@8.service             [FAIL]
       - displaying tomcat                                [OK]
       - displaying dynflow-sidekiq@orchestrator          [FAIL]
       - displaying foreman                               [FAIL]
       - displaying httpd                                 [OK]
       - displaying dynflow-sidekiq@worker-1              [FAIL]
       - displaying dynflow-sidekiq@worker-hosts-queue-1  [FAIL]
       - displaying foreman-proxy                         [OK]
       - All services displayed                                              [FAIL]
       Some services are not running (postgresql, pulpcore-api, pulpcore-worker@1.service, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@5.service, pulpcore-worker@6.service, pulpcore-worker@8.service, dynflow-sidekiq@orchestrator, foreman, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1)
       --------------------------------------------------------------------------------
        Scenario [Status Services] failed.