Getting warning "WARNING: CPU: X PID: XX at arch/x86/mm/ioremap.c:XXX __ioremap_caller"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 9
  • Intel Xeon 6766E
  • Intel® Xeon® 6

Issue

  • The kernel panics with the following traces in RHEL 9.3 with an Intel Xeon CPU.
[  251.402617] ioremap on RAM at 0x0000000199bf1300 - 0x0000000199bf52ff
[  251.402630] WARNING: CPU: 0 PID: 23 at arch/x86/mm/ioremap.c:215 __ioremap_caller+0x34b/0x360
[  251.402657] Modules linked in: intel_uncore(+) binfmt_misc snd_seq_dummy snd_hrtimer nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_de
frag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr vfat fat snd_hda_codec_hdmi x86_pkg_temp_thermal snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec coretemp snd_hda_core snd_hwdep snd_seq lpfc nouveau snd_seq_
device pmt_crashlog ses pmt_telemetry nvmet_fc enclosure intel_sdsi pmt_class cxl_pmem kvm_intel nvmet libnvdimm cxl_port snd_pcm ipmi_ssif irdma mxm_wmi video kvm nvme_fc i40e irqbypass bnxt_re nvme_fabrics drm_display_helper nvme snd_ti
mer cec ib_uverbs snd drm_ttm_helper ttm soundcore pcspkr ib_core nvme_core nvme_common isst_if_mmio scsi_transport_fc isst_if_common idxd intel_vsec idxd_bus acpi_ipmi ipmi_si ipmi_devintf cxl_acpi cxl_core ipmi_msghandler joydev xfs lib
crc32c sd_mod t10_pi sg ast drm_kms_helper
[  251.402743]  syscopyarea sysfillrect sysimgblt ahci i2c_algo_bit crct10dif_pclmul ice libahci drm_shmem_helper crc32_pclmul crc32c_intel qat_4xxx ghash_clmulni_intel bnxt_en drm smartpqi gnss libata intel_qat scsi_transport_sas wmi crc
8 dm_mirror dm_region_hash dm_log dm_mod fuse
[  251.402770] CPU: 0 PID: 23 Comm: cpuhp/0 Kdump: loaded Not tainted 5.14.0-362.8.1.el9_3.x86_64 #1
[  251.402775] Hardware name: To be filled by O.E.M. /Not Filled, BIOS 00.13.02 06/10/2024
[  251.402777] RIP: 0010:__ioremap_caller+0x34b/0x360

Resolution

  • The support for Sierra Forest CPU is introduced in RHEL 9.4.z and/or later.

Root Cause

  • Sierra Forest CPU is not supported by RHEL 9.3.

Diagnostic Steps

The intel_uncore module is failing during MMIO mapping as below:

  • Backtrace
crash> bt
PID: 23       TASK: ffff974c487f1d00  CPU: 0    COMMAND: "cpuhp/0"
 #0 [ffffab4f0db17a88] machine_kexec at ffffffff8b476ec7
 #1 [ffffab4f0db17ae0] __crash_kexec at ffffffff8b5ea4fa
 #2 [ffffab4f0db17ba0] panic at ffffffff8bffc53e
 #3 [ffffab4f0db17c20] __warn.cold at ffffffff8bffc77a
 #4 [ffffab4f0db17c58] report_bug at ffffffff8ba41f2a
 #5 [ffffab4f0db17c98] handle_bug at ffffffff8c05056c
 #6 [ffffab4f0db17ca8] exc_invalid_op at ffffffff8c050774
 #7 [ffffab4f0db17cc0] asm_exc_invalid_op at ffffffff8c200b16
    [exception RIP: __ioremap_caller+843]
    RIP: ffffffff8b48a19b  RSP: ffffab4f0db17d78  RFLAGS: 00010286
    RAX: 0000000000000000  RBX: ffff974ca10c3e00  RCX: 0000000000000027
    RDX: 0000000000000027  RSI: ffffffff8ea7a0a0  RDI: ffff978abce1f888
    RBP: 0000000199bf1300   R8: 80000000fffe1241   R9: 0000000000ffff10
    R10: 0000000000000008  R11: 000000000000000f  R12: 0000000000000002
    R13: 0000000000004000  R14: ffffffffc32b62e5  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffffab4f0db17de0] intel_generic_uncore_mmio_init_box at ffffffffc32b62e5 [intel_uncore]
 #9 [ffffab4f0db17e00] uncore_box_ref at ffffffffc32ab579 [intel_uncore]
#10 [ffffab4f0db17e38] uncore_event_cpu_online at ffffffffc32abcfd [intel_uncore]
#11 [ffffab4f0db17e70] cpuhp_invoke_callback at ffffffff8b50917c
#12 [ffffab4f0db17ec0] cpuhp_thread_fun at ffffffff8b5093bd
#13 [ffffab4f0db17ef8] smpboot_thread_fn at ffffffff8b53de83
#14 [ffffab4f0db17f18] kthread at ffffffff8b53366d
#15 [ffffab4f0db17f50] ret_from_fork at ffffffff8b4031a9
  • Kernel Ring Buffer
[  251.402630] WARNING: CPU: 0 PID: 23 at arch/x86/mm/ioremap.c:215 __ioremap_caller+0x34b/0x360
[  251.402657] Modules linked in: intel_uncore(+) binfmt_misc snd_seq_dummy snd_hrtimer nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_de
frag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink qrtr vfat fat snd_hda_codec_hdmi x86_pkg_temp_thermal snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec coretemp snd_hda_core snd_hwdep snd_seq lpfc nouveau snd_seq_
device pmt_crashlog ses pmt_telemetry nvmet_fc enclosure intel_sdsi pmt_class cxl_pmem kvm_intel nvmet libnvdimm cxl_port snd_pcm ipmi_ssif irdma mxm_wmi video kvm nvme_fc i40e irqbypass bnxt_re nvme_fabrics drm_display_helper nvme snd_ti
mer cec ib_uverbs snd drm_ttm_helper ttm soundcore pcspkr ib_core nvme_core nvme_common isst_if_mmio scsi_transport_fc isst_if_common idxd intel_vsec idxd_bus acpi_ipmi ipmi_si ipmi_devintf cxl_acpi cxl_core ipmi_msghandler joydev xfs lib
crc32c sd_mod t10_pi sg ast drm_kms_helper
[  251.402743]  syscopyarea sysfillrect sysimgblt ahci i2c_algo_bit crct10dif_pclmul ice libahci drm_shmem_helper crc32_pclmul crc32c_intel qat_4xxx ghash_clmulni_intel bnxt_en drm smartpqi gnss libata intel_qat scsi_transport_sas wmi crc
8 dm_mirror dm_region_hash dm_log dm_mod fuse
[  251.402770] CPU: 0 PID: 23 Comm: cpuhp/0 Kdump: loaded Not tainted 5.14.0-362.8.1.el9_3.x86_64 #1
[  251.402775] Hardware name: To be filled by O.E.M. /Not Filled, BIOS 00.13.02 06/10/2024
[  251.402777] RIP: 0010:__ioremap_caller+0x34b/0x360
[  251.402782] Code: 85 c0 0f 89 65 ff ff ff e9 7d fd ff ff 48 8d 54 24 28 48 8d 74 24 18 48 c7 c7 b5 5d 8f 8c c6 05 29 f7 09 02 01 e8 45 d1 07 00 <0f> 0b 45 31 e4 e9 ed fe ff ff e8 76 a5 bc 00 66 0f 1f 44 00 00 90
[  251.402785] RSP: 0018:ffffab4f0db17d78 EFLAGS: 00010286
[  251.402787] RAX: 0000000000000000 RBX: ffff974ca10c3e00 RCX: 0000000000000027
[  251.402790] RDX: 0000000000000027 RSI: ffffffff8ea7a0a0 RDI: ffff978abce1f888
[  251.402791] RBP: 0000000199bf1300 R08: 80000000fffe1241 R09: 0000000000ffff10
[  251.402792] R10: 0000000000000008 R11: 000000000000000f R12: 0000000000000002
[  251.402794] R13: 0000000000004000 R14: ffffffffc32b62e5 R15: 0000000000000000
[  251.402796] FS:  0000000000000000(0000) GS:ffff978abce00000(0000) knlGS:0000000000000000
[  251.402797] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  251.402799] CR2: 00005645aa10a000 CR3: 000000019354c005 CR4: 0000000000770ef0
[  251.402800] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  251.402801] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400
[  251.402802] PKRU: 55555554
[  251.402804] Call Trace:
[  251.402807]  <TASK>
[  251.402809]  ? show_trace_log_lvl+0x1c4/0x2df
[  251.402823]  ? show_trace_log_lvl+0x1c4/0x2df
[  251.402826]  ? intel_generic_uncore_mmio_init_box+0x65/0x90 [intel_uncore]
[  251.402849]  ? __ioremap_caller+0x34b/0x360
[  251.402851]  ? __warn+0x81/0x110
[  251.402858]  ? __ioremap_caller+0x34b/0x360
[  251.402860]  ? report_bug+0x10a/0x140
[  251.402874]  ? handle_bug+0x3c/0x70
[  251.402888]  ? exc_invalid_op+0x14/0x70
[  251.402890]  ? asm_exc_invalid_op+0x16/0x20
[  251.402902]  ? intel_generic_uncore_mmio_init_box+0x65/0x90 [intel_uncore]
[  251.402915]  ? __ioremap_caller+0x34b/0x360
[  251.402917]  ? __ioremap_caller+0x34b/0x360
[  251.402920]  intel_generic_uncore_mmio_init_box+0x65/0x90 [intel_uncore]
[  251.402931]  uncore_box_ref.part.0+0x99/0xc0 [intel_uncore]
[  251.402941]  uncore_event_cpu_online+0x7d/0x160 [intel_uncore]
[  251.402915]  ? __ioremap_caller+0x34b/0x360
[  251.402917]  ? __ioremap_caller+0x34b/0x360
[  251.402920]  intel_generic_uncore_mmio_init_box+0x65/0x90 [intel_uncore]
[  251.402931]  uncore_box_ref.part.0+0x99/0xc0 [intel_uncore]
[  251.402941]  uncore_event_cpu_online+0x7d/0x160 [intel_uncore]
[  251.402951]  ? __pfx_uncore_event_cpu_online+0x10/0x10 [intel_uncore]
[  251.402959]  cpuhp_invoke_callback+0x2bc/0x460
[  251.402965]  ? __pfx_smpboot_thread_fn+0x10/0x10
[  251.402974]  cpuhp_thread_fun+0x8d/0x140
[  251.402977]  smpboot_thread_fn+0xd3/0x1a0
[  251.402980]  kthread+0xdd/0x100
[  251.402985]  ? __pfx_kthread+0x10/0x10
[  251.402987]  ret_from_fork+0x29/0x50
[  251.402997]  </TASK>
[  251.402999] Kernel panic - not syncing: panic_on_warn set ...
[  251.404534] CPU: 0 PID: 23 Comm: cpuhp/0 Kdump: loaded Not tainted 5.14.0-362.8.1.el9_3.x86_64 #1
[  251.406510] Hardware name: To be filled by O.E.M. /Not Filled, BIOS 00.13.02 06/10/2024
[  251.408338] Call Trace:
[  251.409246]  <TASK>
[  251.410098]  dump_stack_lvl+0x34/0x48
[  251.411226]  panic+0xea/0x2e4
[  251.412249]  ? __ioremap_caller+0x34b/0x360
[  251.413461]  __warn.cold+0x26/0x30
[  251.414555]  ? __ioremap_caller+0x34b/0x360
[  251.415779]  report_bug+0x10a/0x140
[  251.416902]  handle_bug+0x3c/0x70
[  251.418000]  exc_invalid_op+0x14/0x70
[  251.419182]  asm_exc_invalid_op+0x16/0x20
[  251.420403] RIP: 0010:__ioremap_caller+0x34b/0x360
[  251.421762] Code: 85 c0 0f 89 65 ff ff ff e9 7d fd ff ff 48 8d 54 24 28 48 8d 74 24 18 48 c7 c7 b5 5d 8f 8c c6 05 29 f7 09 02 01 e8 45 d1 07 00 <0f> 0b 45 31 e4 e9 ed fe ff ff e8 76 a5 bc 00 66 0f 1f 44 00 00 90
[  251.425501] RSP: 0018:ffffab4f0db17d78 EFLAGS: 00010286
[  251.426994] RAX: 0000000000000000 RBX: ffff974ca10c3e00 RCX: 0000000000000027
[  251.428826] RDX: 0000000000000027 RSI: ffffffff8ea7a0a0 RDI: ffff978abce1f888
[  251.430663] RBP: 0000000199bf1300 R08: 80000000fffe1241 R09: 0000000000ffff10
[  251.432512] R10: 0000000000000008 R11: 000000000000000f R12: 0000000000000002
[  251.434371] R13: 0000000000004000 R14: ffffffffc32b62e5 R15: 0000000000000000
[  251.436230]  ? intel_generic_uncore_mmio_init_box+0x65/0x90 [intel_uncore]
[  251.438063]  ? __ioremap_caller+0x34b/0x360
[  251.439440]  intel_generic_uncore_mmio_init_box+0x65/0x90 [intel_uncore]
[  251.441261]  uncore_box_ref.part.0+0x99/0xc0 [intel_uncore]
[  251.442927]  uncore_event_cpu_online+0x7d/0x160 [intel_uncore]
[  251.444634]  ? __pfx_uncore_event_cpu_online+0x10/0x10 [intel_uncore]
[  251.446441]  cpuhp_invoke_callback+0x2bc/0x460
[  251.447926]  ? __pfx_smpboot_thread_fn+0x10/0x10
[  251.449445]  cpuhp_thread_fun+0x8d/0x140
[  251.450844]  smpboot_thread_fn+0xd3/0x1a0
[  251.452263]  kthread+0xdd/0x100
[  251.453550]  ? __pfx_kthread+0x10/0x10
[  251.454927]  ret_from_fork+0x29/0x50
[  251.456286]  </TASK>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments