IPA replica server upgrade failed ,failing to start with "Failed to authenticate to CA REST API"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • IdM/IPA
  • IPA-4.6.8-5.el7_9.10

Issue

  • While upgrading ipa-4.6.8-5.el7_9.10 to ipa-4.6.8-5.el7_9.16, upgrade failed.
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration]
Certmonger certificate renewal configuration already up-to-date
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Authorizing RA Agent to manage lightweight CAs]
[Ensuring Lightweight CAs container exists in Dogtag database]
[Adding default OCSP URI configuration]
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

Resolution

1. yum update ipa-*

2. Check the service status after start and stop of ipa service.

ipactl stop

ipactl start --force

ipactl status

3. If still issue is not resolved then check the ipara and ra agent certs and it's serial number

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
# ldapsearch -D "cn=directory manager" -W -b "uid=pkidbuser,ou=People,o=ipaca"

# cat /var/lib/ipa/ra-agent.pem
# ldapsearch -D "cn=Directory Manager" -W -b uid=ipara,ou=people,o=ipaca

4. Match the data from the given cert on filesystem and it's serial number.

ex. $ openssl x509 -text -in a.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 268304393 (0xffe0009)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = redhat-example.PRIVATE, CN = Certificate Authority
        Validity
            Not Before: Jul 13 12:45:06 2023 GMT
            Not After : Jul  2 12:45:06 2025 GMT
        Subject: O = BPS-PG.PRIVATE, CN = IPA RA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

if serial number does not match then need to correct it.

# ldapmodify -h localhost -D "cn=Directory Manager" -W
dn: uid=ipara,ou=people,o=ipaca
changetype: modify
replace: description
description: 2;268304393;CN=Certificate Authority,O=redhat-example.private;CN=IPA RA,O=redhat-example.private

5. Then after a few minutes restart the IDM replica

ipactl restart

6. Run the upgrade again on replica system.

Root Cause

  • certificate serial number mismatch.

Diagnostic Steps

  • Cannot authenticate agent with certificate Serial 0xffe0009 Subject DN CN=redhat-example.privae. Error: User not found.
  • ca does not start.
2024-02-13T11:58:30Z INFO PKIX already enabled
2024-02-13T11:58:30Z INFO [Authorizing RA Agent to modify profiles]
2024-02-13T11:58:30Z INFO [Authorizing RA Agent to manage lightweight CAs]
2024-02-13T11:58:30Z INFO [Ensuring Lightweight CAs container exists in Dogtag database]
2024-02-13T11:58:30Z DEBUG Created connection context.ldap2_1
2024-02-13T11:58:30Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-redhat-example.socket from SchemaCache
2024-02-13T11:58:30Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-redhat-example.socket conn=
2024-02-13T11:58:30Z DEBUG Destroyed connection context.ldap2_140
2024-02-13T11:58:30Z INFO [Adding default OCSP URI configuration]
2024-02-13T11:58:30Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2024-02-13T11:58:30Z INFO [Migrating certificate profiles to LDAP]
2024-02-13T11:58:30Z DEBUG Created connection context.ldap2_1401
2024-02-13T11:58:30Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-redhat-example.socket from SchemaCache
2024-02-13T11:58:30Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-redhat-example.socket conn=
2024-02-13T11:58:30Z DEBUG Destroyed connection context.ldap2_140153823265616
2024-02-13T11:58:30Z DEBUG request GET https://redhat-example.private:8443/ca/rest/account/login
2024-02-13T11:58:30Z DEBUG request body ''
2024-02-13T11:58:30Z DEBUG response status 401
2024-02-13T11:58:30Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 UTC
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Tue, 13 Feb 2024 11:58:30 GMT

2024-02-13T11:58:30Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:xxx,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:xxx,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2024-02-13T11:58:30Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2024-02-13T11:58:30Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run
    server.upgrade()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2190, in upgrade
    upgrade_configuration()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2062, in upgrade_configuration
    ca_enable_ldap_profile_subsystem(ca)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 439, in ca_enable_ldap_profile_subsystem
    cainstance.migrate_profiles_to_ldap()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2029, in migrate_profiles_to_ldap
    _create_dogtag_profile(profile_id, profile_data, overwrite=False)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2035, in _create_dogtag_profile
    with api.Backend.ra_certprofile as profile_api:
  File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1315, in __enter__
    raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))

2024-02-13T11:58:30Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2024-02-13T11:58:30Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
2024-02-13T11:58:30Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
  • Component
  • ipa

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments