Kernel panic on the system with AMD EPYC 7F72/7262 processor and FIPS mode enabled

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 9 (Plow)
  • kernel-core
  • FIPS
  • AMD EPYC-Rome Processor
  • AMD EPYC 7F72 24-Core Processor
  • AMD EPYC 7262 8-Core Processor

Issue

  • The kernel crashed with the following call traces on the system with AMD EPYC 7F72/7262 processor and FIPS mode enabled.
[    1.552978] alg: ecdh-nist-p256: test failed on vector 2, err=-14
[    1.554824] Kernel panic - not syncing: alg: self-tests for ecdh-nist-p256-generic (ecdh-nist-p256) failed in fips mode!
[    1.557950] CPU: 0 PID: 144 Comm: cryptomgr_test Not tainted 5.14.0-162.6.1.el9_1.x86_64 #1
[    1.560379] Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.16.0-3.module+el8.8.0+16781+9f4724c2 04/01/2014
[    1.563083] Call Trace:
[    1.563931]  dump_stack_lvl+0x34/0x48
[    1.567346]  panic+0x102/0x2d4
[    1.568372]  alg_test+0x500/0x510
[    1.569440]  ? pick_next_task+0x55/0x9f0
[    1.570670]  ? dequeue_task_fair+0xb6/0x3c0
[    1.571957]  ? _raw_spin_unlock+0xa/0x30
[    1.573178]  ? finish_task_switch.isra.0+0x8c/0x2a0
[    1.574648]  ? __schedule+0x20e/0x580
[    1.575810]  ? crypto_acomp_scomp_free_ctx+0x30/0x30
[    1.577309]  cryptomgr_test+0x27/0x50
[    1.579885]  kthread+0x149/0x170
[    1.580948]  ? set_kthread_struct+0x50/0x50
[    1.582239]  ret_from_fork+0x22/0x30
[    1.583682] Kernel Offset: 0x10000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.586758] ---[ end Kernel panic - not syncing: alg: self-tests for ecdh-nist-p256-generic (ecdh-nist-p256) failed in fips mode! ]---

Resolution

Red Hat Enterprise Linux 9.0.z (EUS):

  • The issue has been resolved with kernel-core-5.14.0-70.36.1.el9_0 via Errata: RHSA-2022:8973
$ rpm -qp --changelog kernel-core-5.14.0-70.36.1.el9_0.x86_64.rpm | grep 2136523
- crypto: ecdh - disallow plain "ecdh" usage in FIPS mode (Vladis Dronov) [2136523 2126683]
- crypto: dh - disallow plain "dh" usage in FIPS mode (Vladis Dronov) [2136523 2126683]

Red Hat Enterprise Linux 9.2:

The issue has been resolved with kernel-core-5.14.0-284.11.1.el9_2 via Errata: RHSA-2023:2458

$ rpm -qp kernel-core-5.14.0-284.11.1.el9_2.x86_64.rpm --changelog | grep 2126683
- crypto: ecdh - disallow plain "ecdh" usage in FIPS mode (Vladis Dronov) [2126683]
- crypto: dh - disallow plain "dh" usage in FIPS mode (Vladis Dronov) [2126683]

Root Cause

  • The self-tests for ecdh-nist-p256 algorithm failed in FIPS mode.

  • The ecdh-nist-p256 algorithm usage is not allowed in FIPS mode.

  • Bug-fix patch:
commit 9240d3a6b9f6fadfa93b38a3df345277bcc850a3
Author: Vladis Dronov <vdronov@redhat.com>
Date:   Tue Oct 11 17:46:53 2022 +0200

    crypto: ecdh - disallow plain "ecdh" usage in FIPS mode

    Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2126683
    Upstream Status: RHEL-only

    We do not certify ECDH against FIPS 140-3 for the C9S/RHEL9 kernel.
    As a consequence, this algorithm must not be marked with .fips_allowed = 1.

    Signed-off-by: Vladis Dronov <vdronov@redhat.com>

diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index b7118f1d3bf7..939ba7520964 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -4909,14 +4909,12 @@ static const struct alg_test_desc alg_test_descs[] = {
        }, {
                .alg = "ecdh-nist-p256",
                .test = alg_test_kpp,
-               .fips_allowed = 1,
                .suite = {
                        .kpp = __VECS(ecdh_p256_tv_template)
                }
        }, {
                .alg = "ecdh-nist-p384",
                .test = alg_test_kpp,
-               .fips_allowed = 1,
                .suite = {
                        .kpp = __VECS(ecdh_p384_tv_template)
                }

Diagnostic Steps

  • Determine the CPU model.
# grep "model name" proc/cpuinfo | uniq -c
     48 model name  : AMD EPYC 7F72 24-Core Processor
  • Determine if the kernel booted with FIPS enabled.
# cat /proc/cmdline

The fips=1 parameter indicates that the kernel is booted in FIPS enabled mode.
  • Capture the serial console log.
[    1.552978] alg: ecdh-nist-p256: test failed on vector 2, err=-14
[    1.554824] Kernel panic - not syncing: alg: self-tests for ecdh-nist-p256-generic (ecdh-nist-p256) failed in fips mode!
[    1.557950] CPU: 0 PID: 144 Comm: cryptomgr_test Not tainted 5.14.0-162.6.1.el9_1.x86_64 #1
[    1.560379] Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.16.0-3.module+el8.8.0+16781+9f4724c2 04/01/2014
[    1.563083] Call Trace:
[    1.563931]  dump_stack_lvl+0x34/0x48
[    1.567346]  panic+0x102/0x2d4
[    1.568372]  alg_test+0x500/0x510
[    1.569440]  ? pick_next_task+0x55/0x9f0
[    1.570670]  ? dequeue_task_fair+0xb6/0x3c0
[    1.571957]  ? _raw_spin_unlock+0xa/0x30
[    1.573178]  ? finish_task_switch.isra.0+0x8c/0x2a0
[    1.574648]  ? __schedule+0x20e/0x580
[    1.575810]  ? crypto_acomp_scomp_free_ctx+0x30/0x30
[    1.577309]  cryptomgr_test+0x27/0x50
[    1.579885]  kthread+0x149/0x170
[    1.580948]  ? set_kthread_struct+0x50/0x50
[    1.582239]  ret_from_fork+0x22/0x30
[    1.583682] Kernel Offset: 0x10000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    1.586758] ---[ end Kernel panic - not syncing: alg: self-tests for ecdh-nist-p256-generic (ecdh-nist-p256) failed in fips mode! ]---
  • Determine the algorithm name failed in FIPS mode.
[    1.552978] alg: ecdh-nist-p256: test failed on vector 2, err=-14
[    1.554824] Kernel panic - not syncing: alg: self-tests for ecdh-nist-p256-generic (ecdh-nist-p256) failed in fips mode!

Kernel Source: crypto/algboss.c
*******************************
174 static int cryptomgr_test(void *data)
175 {
176         struct crypto_test_param *param = data;
177         u32 type = param->type;
178         int err = 0;
179 
180 #ifdef CONFIG_CRYPTO_MANAGER_DISABLE_TESTS
181         goto skiptest;
182 #endif
183 
184         if (type & CRYPTO_ALG_TESTED)
185                 goto skiptest;
186 
187         err = alg_test(param->driver, param->alg, type, CRYPTO_ALG_TESTED);

Kernel Source: crypto/testmgr.c
*******************************
5602 int alg_test(const char *driver, const char *alg, u32 type, u32 mask)
5603 {
5604         int i;
5605         int j;
5606         int rc;
5607 
5608         if (!fips_enabled && notests) {
5609                 printk_once(KERN_INFO "alg: self-tests disabled\n");
5610                 return 0;
5611         }
5612 
5613         DO_ONCE(testmgr_onetime_init);
5614 
5615         if ((type & CRYPTO_ALG_TYPE_MASK) == CRYPTO_ALG_TYPE_CIPHER) {
5616                 char nalg[CRYPTO_MAX_ALG_NAME];
5617 
5618                 if (snprintf(nalg, sizeof(nalg), "ecb(%s)", alg) >=
5619                     sizeof(nalg))
5620                         return -ENAMETOOLONG;
5621 
5622                 i = alg_find_test(nalg);
5623                 if (i < 0)
5624                         goto notest;
5625 
5626                 if (fips_enabled && !alg_test_descs[i].fips_allowed)
5627                         goto non_fips_alg;
5628 
5629                 rc = alg_test_cipher(alg_test_descs + i, driver, type, mask);
5630                 goto test_done;
5631         }
....
5654 test_done:
5655         if (rc) {
5656                 if (fips_enabled || panic_on_fail) {
5657                         fips_fail_notify();
5658                         panic("alg: self-tests for %s (%s) failed in %s mode!\n",
5659                               driver, alg,
5660                               fips_enabled ? "fips" : "panic_on_fail");
....

Kernel Source: crypto/testmgr.c
*******************************
4150 static const struct alg_test_desc alg_test_descs[] = {
4151         {
....
4910         }, {
4911                 .alg = "ecdh-nist-p256",
4912                 .test = alg_test_kpp,
4913                 .fips_allowed = 1,          <<< - - -
4914                 .suite = {
4915                         .kpp = __VECS(ecdh_p256_tv_template)
4916                 }

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments