PowerPC server can't boot after upgrade to 3.10.0-327.4.5.el7
Issue
After upgrade to kernel 3.10.0-327.4.5.el7 ppc64 or ppc64le servers can't boot due to panic early in boot process
[ 61.271589] Unable to handle kernel paging request for data at address 0x00000018
[ 61.271607] Faulting instruction address: 0xc0000000004a7380
[ 61.271612] Oops: Kernel access of bad area, sig: 11 [#1]
[ 61.271615] SMP NR_CPUS=2048 NUMA pSeries
[ 61.271619] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack nf_conntrack iptable_filter pseries_rng ses enclosure sg nfsd auth_rpcgss nfs_acl lockd sunrpc grace binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi scsi_transport_srp ibmveth scsi_tgt ipr libata dm_mirror dm_region_hash dm_log dm_mod
[ 61.271666] CPU: 0 PID: 657 Comm: kworker/0:10 Tainted: G W ------------ 3.10.0-327.4.5.el7.ppc64 #1
[ 61.271671] Workqueue: events .topology_work_fn
[ 61.271675] task: c0000007e6721030 ti: c0000007e7d70000 task.ti: c0000007e7d70000
[ 61.271680] NIP: c0000000004a7380 LR: c00000000012c5cc CTR: 0000000000000000
[ 61.271684] REGS: c0000007e7d73520 TRAP: 0300 Tainted: G W ------------ (3.10.0-327.4.5.el7.ppc64)
[ 61.271688] MSR: 8000000100009032 <SF,EE,ME,IR,DR,RI> CR: 48848024 XER: 00000000
[ 61.271699] CFAR: c000000000009368 DAR: 0000000000000018 DSISR: 40000000 SOFTE: 1
GPR00: c00000000012c5cc c0000007e7d737a0 c0000000013822d0 0000000000000018
GPR04: 0000000000000800 0000000000000000 0000000000000000 0000000000000000
GPR08: 0000000000000000 00000000010b0000 c000000000d36840 0000000000000000
GPR12: 0000000000000000 c000000007b30000 c0000007e6b61400 00000000000000b8
GPR16: 0000000000000800 c0000007e538fe00 0000000000000000 c0000007e538fe00
GPR20: 00000000000000f8 c0000000013d09d8 0000000000000001 0000000000000001
GPR24: c00000000129c147 0000000000000020 0000000000000018 0000000000000800
GPR28: 0000000000000020 0000000000000110 0000000000000000 0000000000000010
[ 61.271760] NIP [c0000000004a7380] .__bitmap_weight+0x70/0x100
[ 61.271765] LR [c00000000012c5cc] .build_sched_domains+0xd3c/0x10b0
[ 61.271768] Call Trace:
[ 61.271772] [c0000007e7d737a0] [c0000000002be454] .kmem_cache_alloc_node_trace+0xb4/0x3a0 (unreliable)
[ 61.271780] [c0000007e7d73850] [c00000000012c5cc] .build_sched_domains+0xd3c/0x10b0
[ 61.271788] [c0000007e7d739a0] [c00000000012d154] .partition_sched_domains+0x484/0x510
[ 61.271794] [c0000007e7d73aa0] [c000000000196518] .rebuild_sched_domains+0x68/0xa0
[ 61.271799] [c0000007e7d73b30] [c000000000063b50] .topology_work_fn+0x10/0x30
[ 61.271804] [c0000007e7d73ba0] [c0000000000ff7d0] .process_one_work+0x1d0/0x690
[ 61.271809] [c0000007e7d73c50] [c0000000000ffe10] .worker_thread+0x180/0x500
[ 61.271814] [c0000007e7d73d30] [c00000000010be28] .kthread+0xe8/0xf0
[ 61.271820] [c0000007e7d73e30] [c00000000000a470] .ret_from_kernel_thread+0x58/0x68
[ 61.271824] Instruction dump:
[ 61.271826] 409d00b4 3bbcffff 3be3fff8 7bbd1f48 3bc00000 7fa3ea14 48000018 60000000
[ 61.271835] 60000000 60000000 60000000 60420000 <e87f0009> 4bbc72b9 60000000 7fbfe840
[ 61.271847] ---[ end trace fd234da35d8d8a95 ]---
If the vmcore is not collected following warning can be observed in logs shortly before crash
[ 61.271435] ------------[ cut here ]------------
[ 61.271439] WARNING: at kernel/sched/core.c:6277
[ 61.271442] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_REJECT nf_
conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack nf_conntrack iptable_filter
pseries_rng ses enclosure sg nfsd auth_rpcgss nfs_acl lockd sunrpc grace binfmt_misc ip_tables
ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi sc
si_transport_srp ibmveth scsi_tgt ipr libata dm_mirror dm_region_hash dm_log dm_mod
[ 61.271485] CPU: 0 PID: 657 Comm: kworker/0:10 Not tainted 3.10.0-327.4.5.el7.ppc64 #1
[ 61.271490] Workqueue: events .topology_work_fn
[ 61.271493] task: c0000007e6721030 ti: c0000007e7d70000 task.ti: c0000007e7d70000
[ 61.271495] NIP: c00000000012c900 LR: c00000000012c5f8 CTR: 0000000000000000
[ 61.271497] REGS: c0000007e7d735d0 TRAP: 0700 Not tainted (3.10.0-327.4.5.el7.ppc64)
[ 61.271498] MSR: 8000000100029032 <SF,EE,ME,IR,DR,RI> CR: 48848022 XER: 00000000
[ 61.271503] CFAR: c00000000012c5bc SOFTE: 1
GPR00: c00000000012c5f8 c0000007e7d73850 c0000000013822d0 0000000000000010
GPR04: 0000000000000800 0000000000000000 0000000000000000 0000000000000000
GPR08: 0000000000000000 00000000010b0000 c000000000d36840 0000000000000000
GPR12: 0000000000000000 c000000007b30000 c0000007e6b61400 00000000000000b8
GPR16: 0000000000000800 c0000007e538fe00 0000000000000000 c0000007e538fe00
GPR20: 00000000000000f8 c0000000013d09d8 0000000000000001 0000000000000001
GPR24: c00000000129c147 0000000000000017 0000000000000000 c0000007e31f7e00
GPR28: c0000000013d0a90 c0000007e6b61400 0000000000000000 c0000000013d5ebc
[ 61.271532] NIP [c00000000012c900] .build_sched_domains+0x1070/0x10b0
[ 61.271535] LR [c00000000012c5f8] .build_sched_domains+0xd68/0x10b0
[ 61.271537] Call Trace:
[ 61.271540] [c0000007e7d73850] [c00000000012c5f8] .build_sched_domains+0xd68/0x10b0 (unrelia
ble)
[ 61.271543] [c0000007e7d739a0] [c00000000012d154] .partition_sched_domains+0x484/0x510
[ 61.271547] [c0000007e7d73aa0] [c000000000196518] .rebuild_sched_domains+0x68/0xa0
[ 61.271550] [c0000007e7d73b30] [c000000000063b50] .topology_work_fn+0x10/0x30
[ 61.271553] [c0000007e7d73ba0] [c0000000000ff7d0] .process_one_work+0x1d0/0x690
[ 61.271555] [c0000007e7d73c50] [c0000000000ffe10] .worker_thread+0x180/0x500
[ 61.271558] [c0000007e7d73d30] [c00000000010be28] .kthread+0xe8/0xf0
[ 61.271562] [c0000007e7d73e30] [c00000000000a470] .ret_from_kernel_thread+0x58/0x68
[ 61.271564] Instruction dump:
[ 61.271566] 4bfffc1c 3ae00000 4bfffc04 7e10a838 4bfffe40 7fe3fb78 48192a29 60000000
[ 61.271572] 39200000 3ae0fff4 f9210078 4bfffbf0 <0fe00000> 4bfffcbc 89580006 694a0001
Environment
RedHat Enterprise Linux 7.2 on ppc64/ppc64le
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.