IPA server upgrade failed with error "RemoteRetrieveError: Failed to authenticate to CA REST API"
Environment
- Red Hat Enterprise Linux (ALL versions)
- IdM/IPA Master/Replica
Issue
- While upgrading IPA, upgrade failed with below error.
[Upgrading CA schema]
CA schema update complete (no changes)
[Verifying that CA audit signing cert has 2 year validity]
[Update certmonger certificate renewal configuration]
Certmonger certificate renewal configuration already up-to-date
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Authorizing RA Agent to manage lightweight CAs]
[Ensuring Lightweight CAs container exists in Dogtag database]
[Adding default OCSP URI configuration]
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
Resolution
- Check the
SubsystemCertandRA-agentcertificate and their serial number.
For SubsystemCert cert-pki-ca / pkidbuser
-
Check for both certificate and Serial number in LDAP and actual cert.
# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n "subsystemCert cert-pki-ca" -a <-- For certificate -----BEGIN CERTIFICATE----- MIID3jCCAkagAwIBAgIBBDANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQKDAhJUEEu ....snip.... m9lBlj3KzPd3tbNkp3mjm95VksArZw/HgIvi+P0zkTC1BQ== -----END CERTIFICATE----- # certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n "subsystemCert cert-pki-ca" | head -n 10 <------- For serial number Certificate: Data: Version: 3 (0x2) Serial Number: 68304393 (0xffe0009) <---------------- ^^^^^^^^ Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption Issuer: "CN=Certificate Authority,O=IPA.TEST" Validity: Not Before: Fri Apr 17 06:21:35 2026 Not After : Thu Apr 06 06:21:35 2028 Subject: "CN=CA Subsystem,O=IPA.TEST" # ldapsearch -D "cn=directory manager" -W -b "uid=pkidbuser,ou=People,o=ipaca" <---- In Database ...snip... description: 2;93875;CN=Certificate Authority,O=IPA.TEST;CN=CA Subsystem,O=IPA.TEST ^^^^^ userCertificate:: MIID3jCCAkagAwIBAgIBBDANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQKDAhJ UEEuVEVTVDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTI2MDQxNzA2MjEzNVoXD ...snip.... bjTL4PiJ2GC21+jP56KeCvV0nL7XTSoGhVO0w/0Bvmfhh9fUcRIKbNwdjd4bvcg0Am9lBlj3KzPd3 tbNkp3mjm95VksArZw/HgIvi+P0zkTC1BQ== ...snip.... -
Correct the values if either Serial number or Certificate doesn't match. If it matches, Skip to IPARA/RA-Agent
-
For Serial Number:
- Obtain serial number and update in LDAP Database
# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n "subsystemCert cert-pki-ca" | grep -i serial Serial Number: 68304393 (0xffe0009) <---------------- ^^^^^^^^ # ldapmodify -h localhost -D "cn=Directory Manager" -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: description description:2;SERIAL_NUMBER;CN=Certificate Authority,O=IPA.TEST;CN=CA Subsystem,O=IPA.TEST <-- Replace domain accordingly Press Ctrl+D-
For Certificate:
- Get the certificate in a single line and update.
# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n "subsystemCert cert-pki-ca" -a | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}' # ldapmodify -h localhost -D "cn=Directory Manager" -W dn: uid=pkidbuser,ou=people,o=ipaca changetype: modify replace: userCertificate userCertificate:CERTIFICATE_BLOB_FROM_ABOVE_COMMAND_WITHOUT_ANY_SPACE Press Ctrl+D -
For RA-agent/ IPARA:
-
Check for both certificate and Serial number in LDAP and actual cert.
# cat /var/lib/ipa/ra-agent.pem <--- For certificate -----BEGIN CERTIFICATE----- MIID2DCCAkCgAwIBAgIBBzANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQKDAhJUEEu ...snip... b8qh4t7lz9pltk8aOWMQb1RZbdUAqlrD0HhrIg== -----END CERTIFICATE----- # openssl x509 -in /var/lib/ipa/ra-agent.pem --noout -text | head -n 10 <--- For Serial Number Certificate: Data: Version: 3 (0x2) Serial Number: 745 (0x7) <------------ ^^^ Signature Algorithm: sha256WithRSAEncryption Issuer: O=IPA.TEST, CN=Certificate Authority Validity Not Before: Apr 17 06:23:14 2026 GMT Not After : Apr 6 06:23:14 2028 GMT Subject: O=IPA.TEST, CN=IPA RA # ldapsearch -D "cn=Directory Manager" -W -b uid=ipara,ou=people,o=ipaca <--- In Database ...snip........ userCertificate:: MIID2DCCAkCgAwIBAgIBBzANBgkqhkiG9w0BAQsFADAzMREwDwYDVQQKDAhJ UEEuVEVTVDEeMBwGA1UEAwwVQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTI2MDQxNzA2MjMxNFoXD TI4MDQwNjA2MjMxNFowJDERMA8GA1UECgwISVBBLlRFU1QxDzANBgNVBAMTBklQQSBSQTCCASIwDQ ...snip... jMmUXPBwY4GDmm9lOsJfXJb6lbR8bWV9VQL26KLeF8V3Y6Nq2 iVmfUAXU05tF80rIL9yEf1TFs7kvIOfs6X+jnx8C2HgQ8P7VoAUQS0dAwjU59TU+Db8qh4t7lz9pl tk8aOWMQb1RZbdUAqlrD0HhrIg== description: 2;786;CN=Certificate Authority,O=IPA.TEST;CN=IPA RA,O=IPA.TEST ^^^ ....snip.... -
Correct the values if either Serial number or Certificate doesn't match. If it matches, Skip the steps..
-
For Serial Number:
- Obtain serial number and update in LDAP Database
# certutil -L -d /var/lib/pki/pki-tomcat/alias/ -n "subsystemCert cert-pki-ca" | grep -i serial Serial Number: 68304393 (0xffe0009) <---------------- ^^^^^^^^ # ldapmodify -h localhost -D "cn=Directory Manager" -W dn: uid=ipara,ou=people,o=ipaca changetype: modify replace: description description:2;SERIAL_NUMBER;CN=Certificate Authority,O=IPA.TEST;CN=IPA RA,O=IPA.TESTT <-- Replace this accordingly. Press Ctrl+D-
For Certificate:
- Get the certificate in a single line and update.
# cat /var/lib/ipa/ra-agent.pem | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE---- -$/b2;H;b1};:2;${x;s/\s//g;p}' # ldapmodify -h localhost -D "cn=Directory Manager" -W dn: uid=ipara,ou=people,o=ipaca changetype: modify replace: userCertificate userCertificate:CERTIFICATE_BLOB_FROM_ABOVE_COMMAND_WITHOUT_ANY_SPACE Press Ctrl+D -
-
Restart IPA services
# ipactl restart -f -
Run the IPA upgrade again
# ipa-server-upgrade
Root Cause
- At time of certificate renewal, Sometimes, Certificate or it's serial number doesn't get updated properly which causes the mismatch in either cert or serial number.
Diagnostic Steps
- Cannot authenticate agent with certificate Serial 0xffe0009 Subject DN CN=ipa.test. Error: User not found.
- ca does not start.
2024-02-13T11:58:30Z INFO PKIX already enabled
2024-02-13T11:58:30Z INFO [Authorizing RA Agent to modify profiles]
2024-02-13T11:58:30Z INFO [Authorizing RA Agent to manage lightweight CAs]
2024-02-13T11:58:30Z INFO [Ensuring Lightweight CAs container exists in Dogtag database]
2024-02-13T11:58:30Z DEBUG Created connection context.ldap2_1
2024-02-13T11:58:30Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-ipa-test.socket from SchemaCache
2024-02-13T11:58:30Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-ipa-test.socket conn=
2024-02-13T11:58:30Z DEBUG Destroyed connection context.ldap2_140
2024-02-13T11:58:30Z INFO [Adding default OCSP URI configuration]
2024-02-13T11:58:30Z INFO [Ensuring CA is using LDAPProfileSubsystem]
2024-02-13T11:58:30Z INFO [Migrating certificate profiles to LDAP]
2024-02-13T11:58:30Z DEBUG Created connection context.ldap2_1401
2024-02-13T11:58:30Z DEBUG flushing ldapi://%2fvar%2frun%2fslapd-ipa-test.socket from SchemaCache
2024-02-13T11:58:30Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-ipa-test.socket conn=
2024-02-13T11:58:30Z DEBUG Destroyed connection context.ldap2_140153823265616
2024-02-13T11:58:30Z DEBUG request GET https://redhat-example.private:8443/ca/rest/account/login
2024-02-13T11:58:30Z DEBUG request body ''
2024-02-13T11:58:30Z DEBUG response status 401
2024-02-13T11:58:30Z DEBUG response headers Server: Apache-Coyote/1.1
Cache-Control: private
Expires: Thu, 01 Jan 1970 00:00:00 UTC
WWW-Authenticate: Basic realm="Certificate Authority"
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 951
Date: Tue, 13 Feb 2024 11:58:30 GMT
2024-02-13T11:58:30Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:xxx,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:xxx,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 401 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>This request requires HTTP authentication.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
2024-02-13T11:58:30Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2024-02-13T11:58:30Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 54, in run
server.upgrade()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2190, in upgrade
upgrade_configuration()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 2062, in upgrade_configuration
ca_enable_ldap_profile_subsystem(ca)
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 439, in ca_enable_ldap_profile_subsystem
cainstance.migrate_profiles_to_ldap()
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2029, in migrate_profiles_to_ldap
_create_dogtag_profile(profile_id, profile_data, overwrite=False)
File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 2035, in _create_dogtag_profile
with api.Backend.ra_certprofile as profile_api:
File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line 1315, in __enter__
raise errors.RemoteRetrieveError(reason=_('Failed to authenticate to CA REST API'))
2024-02-13T11:58:30Z DEBUG The ipa-server-upgrade command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API
2024-02-13T11:58:30Z ERROR Unexpected error - see /var/log/ipaupgrade.log for details:
RemoteRetrieveError: Failed to authenticate to CA REST API
2024-02-13T11:58:30Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments