PKINIT request on IPA client failing with an error "Failed to verify CMS message: invalid digest", causing smart card authentication to fail

Solution Verified - Updated -

Issue

  • PKINIT request on IPA/IdM client failing with an error "Failed to verify CMS message: invalid digest" while issuing kerberos ticket for trusted AD account, causing smart card authentication via GDM to fail:
# KRB5_TRACE=/dev/stdout kinit -X X509_user_identity=PKCS11:module_name=/usr/lib64/opensc-pkcs11.so <aduser>@<addomain>
...
[1234] 1234567890.12345: Processing preauth types: PA-PK-AS-REP (17)
[1234] 1234567890.12345: PKINIT OpenSSL error: Failed to verify CMS message
[1234] 1234567890.12345: PKINIT OpenSSL error: error:03000098:digital envelope routines::invalid digest
[1234] 1234567890.12345: PKINIT client could not verify DH reply
[1234] 1234567890.12345: Preauth module pkinit (17) (real) returned: -1765328304/Failed to verify CMS message: invalid digest <--
  • After configuring Smartcard authentication for AD account son IPA client, sssctl given an error during authentication:
# sssctl user-checks <aduser>@<addomain> -a auth -s gdm-smartcard
...
testing pam_authenticate

PIN for <aduser>:
pam_authenticate for user [<aduser>@<addomain>]: Authentication failure  <--

PAM Environment:
 - PKCS11_LOGIN_TOKEN_NAME=<aduser>

Environment

  • Red Hat Enterprise Linux 9.6
    • (IPA/IdM client)
    • Smartcard Authentication (Yubikey)
    • sssd
    • krb5
      • krb5-pkinit-1.21.1-3.el9
  • Microsoft Active Directory Server 2016
  • IPA-AD Trust Setup (One-Way)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content