How do I manually renew Identity Management (IPA) certificates on RHEL7/RHEL 8 after they have expired? (Master IPA Server)

Solution Verified - Updated -

Red Hat Insights can detect this issue

Proactively detect and remediate issues impacting your systems.
View matching systems and remediation

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Identity Management (IPA) v4

Issue

In normal operation, it’s expected that renewal of IPA subsystem certificates is working smoothly. Unfortunately in reality there are sometimes issues to renew those certificates and a manual recovery is necessary in case certificates are already expired.

Resolution

IMPORTANT - Red Hat Enterprise Linux 7.7 and later supports a simplified process of renewing system certificates when IPA is offline. Instructions for renewing expired certificates with the new ipa-cert-fix utility are available here and should be used in lieu of the procedure below if Red Hat Enterprise Linux 7.7 or later is installed..

DISCLAIMERS AND WARNINGS
This procedure was tested and has been verified to work. However, it is a complicated and potentially error-prone procedure, so please do not hesitate to contact Red Hat Technical Support for assistance if you have any questions or concerns.

This procedure needs to be run on an IPA Master with an embedded CA. Separate instructions for renewing IPA certificates on IPA Replica servers can be found here.

CAUTION: BE SURE TO CREATE BACKUPS OF THE FOLLOWING DIRECTORIES AND FILES BEFORE BEGINNING.

  • /etc/dirsrv/slapd-REALM/*.db
  • /etc/httpd/alias
  • /var/lib/ipa/ra-agent.{key,pem} (>= RHEL-7.4)
  • /etc/pki/pki-tomcat/ca/CS.cfg
  • /var/lib/certmonger
  • /etc/pki/pki-tomcat/alias/

PROCESS

Step 1: List certificates to confirm tracking

  • Run the following command to fix the number of certificates being tracked if there is less than expected:

    nct=`getcert list | grep -i 'Number of certificates and requests being tracked:' |grep -o '[0-9]*'`
    rhelver=`grep -o '[0-9.]*' /etc/redhat-release`
    
    # RHEL 7.2 or older and less than 8 certificates
    if [[ ${nct:-0} < 8 ]] && [[ ${rhelver//*.} -le 2 ]]
    then
    getcert start-tracking -d /etc/httpd/alias -n ipaCert -c dogtag-ipa-ca-renew-agent -p /etc/httpd/alias/pwdfile.txt -B /usr/lib64/ipa/certmonger/renew_ca_cert_pre -C /usr/lib64/ipa/certmonger/renew_ra_cert -T caSubsystemCert
    
    # RHEL 7.3 and less than 8 certificates
    elsif [[ ${nct:-0} < 8 ]] && [[ ${rhelver//*.} -eq 3 ]]
    pin=`grep -i internal= /etc/pki/pki`
          getcert start-tracking -d /etc/pki/pki-tomcat/alias -n "auditSigningCert cert-pki-ca" -c 'dogtag-ipa-ca-renew-agent' -P "$pin" -B /usr/libexec/ipa/certmonger/stop_pkicad -C '/usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"' -T caSignedLogCert
          getcert start-tracking -d /etc/pki/pki-tomcat/alias -n "ocspSigningCert cert-pki-ca" -c 'dogtag-ipa-ca-renew-agent' -P "$pin" -B /usr/libexec/ipa/certmonger/stop_pkicad -C '/usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"' -T caOCSPCert;
          getcert start-tracking -d /etc/pki/pki-tomcat/alias -n "subsystemCert cert-pki-ca" -c 'dogtag-ipa-ca-renew-agent' -P "$pin" -B /usr/libexec/ipa/certmonger/stop_pkicad -C '/usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"' -T caSubsystemCert;
          getcert start-tracking -d /etc/pki/pki-tomcat/alias -n "Server-Cert cert-pki-ca" -c 'dogtag-ipa-ca-renew-agent' -P "$pin"-B /usr/libexec/ipa/certmonger/stop_pkicad -C '/usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"' -T caServerCert
    pin=''
    
    # RHEL 7.4 or newer (i.e: from 7.4 - 8.x) and less than 9 certificates
    elsif [[ ${nct:-0} < 9 ]] && [[ ${rhelver//*.} > 3 ]]
    getcert start-tracking -f /var/lib/ipa/ra-agent.pem -k /var/lib/ipa/ra-agent.key -c dogtag-ipa-ca-renew-agent -B /usr/libexec/ipa/certmonger/renew_ra_cert_pre -C /usr/libexec/ipa/certmonger/renew_ra_cert -T caSubsystemCert 
    fi
    
  • Run the getcert list command again to confirm that 9 certificates (8 certificates for RHEL <= 7.3) are now being tracked.

    More detailed instructions how to renew all the IdM internal certificates can be found on the following page: https://access.redhat.com/articles/4062581

Step 2: Determine when the certificates were last valid and set the date

  • In order for this to work, you will need to set the system clock back to a date and time when the certificates were all still valid. First, we need to stop the NTP service:

    # systemctl stop ntpd
    
  • Then run this script that will set the system clock back to when the certificates were valid (one day before expiration):

    for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
    do
      certdate=$(date -d "`certutil -L -d /etc/pki/pki-tomcat/alias -n "${nickname}" | grep -i after | cut -d: -f2-`" +%s )
      echo "$nickname - $certdate"
      [[ ${newdate:-99999999999} -gt "${certdate}" ]] && newdate=$certdate
    done
    date --set="`date --date=@$[newdate - 86400]`"
    

Step 3: Renew the certificates

  • With the system clock now set back in time to allow the expired certificates to be valid, we need to force renewal on all of the certificates. Restart the certmonger service to resubmit renewal requests for the expired certificates:

    # systemctl restart certmonger
    
  • This will renew the CA subsystem certificates, but the SSL server certificates for the Apache and 389 Directory Server services will have failed to renew. We need to do a little more work to get these certificates renewed so that the IPA CA will be fully operational. Running the getcert list command again should confirm that these certificates still need to be updated.

Step 4: Verify and fix CA subsystem certificate trust permissions

  • The trust attributes of the internal CA subsystem certificates should be set as in the below example (note that the ordering of the certificates may differ - what is important is that the trust attributes are correct). Pass the list -L option to the certutil command to display the attributes that are currently set:

    #  certutil -d /var/lib/pki/pki-tomcat/alias/ -L
    Certificate Nickname                                 Trust Attributes         SSL,S/MIME,JAR/XPI
    
    ocspSigningCert cert-pki-ca                  u,u,u
    subsystemCert cert-pki-ca                     u,u,u
    caSigningCert cert-pki-ca                       CTu,Cu,Cu
    Server-Cert cert-pki-ca                            u,u,u
    auditSigningCert cert-pki-ca                u,u,Pu
    

    Please note that the CA subsystem will NOT run correctly if the trust attributes of its internal certificates are incorrect.

  • If the trust attributes for each certificate are correct, proceed to the next step. If not, set the correct trust attributes with the following command:

    # certutil -M -n "<Certificate Nickname>" -d  /var/lib/pki/pki-tomcat/alias/ -t <Trust attributes>
    

    For example, to correct the trust attributes for the auditSigningCert cert-pki-ca certificate, use the following command:

    # certutil -M -n "auditSigningCert cert-pki-ca" -d  /var/lib/pki/pki-tomcat/alias/ -t u,u,Pu 
    
  • Repeat the above certutil command to correct any other certificates that have incorrect attributes set.

Step 5: Update subsystem certificate blobs in IPA CA configuration file

  • The IPA CA configuration file contains Base64-encoded copies of the CA subsystem certificates. You'll need to confirm that the configuration file contains the current certificates and update as needed.

  • Extract the individual CA subsystem certificates to single line text files in the /tmp directory:

    # for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do certutil -L -d  /var/lib/pki/pki-tomcat/alias/ -n "${nickname}" -a | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}'  > /tmp/"${nickname}.pem"; done
    
  • Review the file /etc/pki/pki-tomcat/ca/CS.cfg and find the blobs for the CA subsystem certificates by their nicknames. Here are the nicknames and values:

nickname value
auditSigningCert cert-pki-ca ca.audit_signing.cert
ocspSigningCert cert-pki-ca ca.ocsp_signing.cert
subsystemCert cert-pki-ca ca.subsystem.cert
Server-Cert cert-pki-ca ca.sslserver.cert
  • Example:

    # grep -E 'ca.audit_signing.cert=|ca.ocsp_signing.cert=|ca.subsystem.cert=|ca.sslserver.cert=' /etc/pki/pki-tomcat/ca/CS.cfg
    
  • Compare the certificate blobs in CS.cfg with the respective copies that you extracted to /tmp earlier; they should match and if so, you can delete the single line text files that you created in /tmp and proceed to the next step. If they do NOT match:

  • Stop the CA service:

    # systemctl stop pki-tomcatd@pki-tomcat.service
    
  • Edit /etc/pki/pki-tomcat/ca/CS.cfg, find the certificate entry that needs to be updated and replace the blobs with the copies that you extracted to single-line files in /tmp. Be sure to back up /etc/pki/pki-tomcat/ca/CS.cfg before editing it! A corrupted copy of this file will cause the CA to fail to start.

  • Restart the CA and verify that it is running:

    # systemctl start pki-tomcatd@pki-tomcat.service
    # systemctl status pki-tomcatd@pki-tomcat.service
    

Step 6: Test CA operation

  • To test the operation of the CA, use the following command:

    # curl --cacert /etc/ipa/ca.crt -v https://`hostname`:8443/ca/ee/ca/getCertChain 
    
  • You should receive a response from the CA similar to the following example:

    * About to connect() to idm.example.com port 8443 (#0)
    *   Trying 10.10.183.106...
    * Connected to idm.example.com (10.10.1.1) port 8443 (#0)
    * Initializing NSS with certpath: sql:/etc/pki/nssdb
    *   CAfile: /etc/ipa/ca.crt
    CApath: none
    * NSS: client certificate not found (nickname not specified)
    * SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
    * Server certificate:
    *     subject: CN=idm.example.com,O=EXAMPLE.COM
    *     start date: Feb 14 18:00:27 2018 GMT
    *     expire date: Feb 04 18:00:27 2020 GMT
    *     common name: idm.example.com
    *     issuer: CN=Certificate Authority,O=EXAMPLE.COM
    > GET /ca/ee/ca/getCertChain HTTP/1.1
    > User-Agent: curl/7.29.0
    > Host: idm.example.com:8443
    > Accept: */*
    >
    < HTTP/1.1 200 OK
    < Server: Apache-Coyote/1.1
    < Content-Type: application/xml
    < Content-Length: 1454
    < Date: Thu, 15 Feb 2018 11:15:46 GMT
    <
    [...]
    
  • Note: try the FQDN of the host if you get this response:

    * NSS error -12276 (SSL_ERROR_BAD_CERT_DOMAIN)
    * Unable to communicate securely with peer: requested domain name does not match the server's certificate.
    

Step 7: Verify and update IPA agent certificate

  • The IPA agent certificate (ipaCert) is used to authenticate with the CA for certificate operations and is stored in LDAP. This entry must be stored correctly in order for communication with the CA to occur.

  • Start by determining the new serial number for the agent certificate:

    RHEL <= 7.3
    # certutil -L -d /etc/httpd/alias -n ipaCert | grep -i serial
    
    RHEL >= 7.4
    # openssl x509 -in /var/lib/ipa/ra-agent.pem -noout -text | grep -i serial
    
  • Verify what data is stored in LDAP for this entry:

    # ldapsearch -x -h localhost -p 389 -D 'cn=directory manager' -W -b uid=ipara,ou=People,o=ipaca
    
  • Compare the serial number for the agent certificate in the description attribute with the current one. The serial number is the 2nd number. The format of the description line is:

    2;[serial number];[issuer subject];[subject]
    
  • If the serial numbers are identical, the entry has already been updated. Proceed to the next step. If the serial numbers do NOT match, run the following commands.

  • Extract the Base64-encoded value of the agent certificate to a single line file in /tmp:

    RHEL <= 7.3
    # certutil -L -d /etc/httpd/alias -n ipaCert -a  | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}'  > /tmp/ipaCert.pem
    
    RHEL >= 7.4
    # cat /var/lib/ipa/ra-agent.pem   | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}'  > /tmp/ipaCert.pem
    
  • Use the ldapmodify command, replace the previous serial number in the description attribute with the new one and add the updated agent certificate blob with the contents of /tmp/ipaCert.pem.

  • The change will look something like the below example (do NOT include brackets around the new values):

    # ldapmodify -x -h localhost -p 389 -D 'cn=directory manager' -w password
    dn: uid=ipara,ou=people,o=ipaca
    changetype: modify
    add: usercertificate
    usercertificate:: <<PASTE NEW AGENT CERTIFICATE BLOB HERE>>
    -
    replace: description
    description: 2;<<INSERT NEW SERIAL NUMBER HERE>>;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
    
    ^D to save and exit
    
    # systemctl restart httpd
    

Step 8: Renew Directory Server and Apache SSL server certificates

  • Next we need to renew the Directory Server and Apache SSL server certificates:

    # ipa-getcert list -n Server-Cert |grep 'Request ID'
    
  • For each of the two Server-Cert Request IDs listed, run the following command:

    # ipa-getcert resubmit -i [Request ID]
    

Step 9: Return the system clock to the correct time and date and confirm it is now current. To return to the present time

  • Restart the ntpd service

    # systemctl start ntpd
    # date                           
    

Step 10: Return services to production and restart the world

  • Restart services

    # ipactl restart
    
  • Restart the certmonger service

    # systemctl restart certmonger
    

Step 11: Test the IPA Admin user

  • Log into IPA as an administrative user:

    # kinit admin
    

Step 12: Test IPA administrative functions**

  • Finally, to make sure that IPA administrative functions work correctly, run a few commands to ensure expected data is returned and that no errors are reported:

    # ipa cert-show 1
    # ipa user-show <userid>
    # ipa service-find
    # ipa sudorule-find
    # ipa hostgroup-find
    

Root Cause

Several certificates tracked by certmonger are shown with a status of CA_UNREACHABLE:

Request ID '20170213125735':
        status: CA_UNREACHABLE
        ca-error: Internal error
        stuck: no
        key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
        certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
        CA: dogtag-ipa-ca-renew-agent
        issuer: CN=Certificate Authority,O=EXAMPLE.COM
        subject: CN=CA Subsystem,O=EXAMPLE.COM
        expires: 2018-02-11 13:57:14 UTC
        key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
        post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
        track: yes
        auto-renew: yes
  • Component
  • ipa

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Was this helpful?

We appreciate your feedback. Leave a comment if you would like to provide more detail.
It looks like we have some work to do. Leave a comment to let us know how we could improve.

Get notified when this content is updated

You'll get an email whenever this content is updated or others comment. You can manage all of your notifications in your profile

Comments