PowerPC server can't boot after upgrade to 3.10.0-327.4.5.el7

Solution In Progress - Updated -

Issue

After upgrade to kernel 3.10.0-327.4.5.el7 ppc64 or ppc64le servers can't boot due to panic early in boot process

[   61.271589] Unable to handle kernel paging request for data at address 0x00000018
[   61.271607] Faulting instruction address: 0xc0000000004a7380
[   61.271612] Oops: Kernel access of bad area, sig: 11 [#1]
[   61.271615] SMP NR_CPUS=2048 NUMA pSeries
[   61.271619] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack nf_conntrack iptable_filter pseries_rng ses enclosure sg nfsd auth_rpcgss nfs_acl lockd sunrpc grace binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi scsi_transport_srp ibmveth scsi_tgt ipr libata dm_mirror dm_region_hash dm_log dm_mod
[   61.271666] CPU: 0 PID: 657 Comm: kworker/0:10 Tainted: G        W      ------------   3.10.0-327.4.5.el7.ppc64 #1
[   61.271671] Workqueue: events .topology_work_fn
[   61.271675] task: c0000007e6721030 ti: c0000007e7d70000 task.ti: c0000007e7d70000
[   61.271680] NIP: c0000000004a7380 LR: c00000000012c5cc CTR: 0000000000000000
[   61.271684] REGS: c0000007e7d73520 TRAP: 0300   Tainted: G        W      ------------    (3.10.0-327.4.5.el7.ppc64)
[   61.271688] MSR: 8000000100009032 <SF,EE,ME,IR,DR,RI>  CR: 48848024  XER: 00000000
[   61.271699] CFAR: c000000000009368 DAR: 0000000000000018 DSISR: 40000000 SOFTE: 1 
               GPR00: c00000000012c5cc c0000007e7d737a0 c0000000013822d0 0000000000000018 
               GPR04: 0000000000000800 0000000000000000 0000000000000000 0000000000000000 
               GPR08: 0000000000000000 00000000010b0000 c000000000d36840 0000000000000000 
               GPR12: 0000000000000000 c000000007b30000 c0000007e6b61400 00000000000000b8 
               GPR16: 0000000000000800 c0000007e538fe00 0000000000000000 c0000007e538fe00 
               GPR20: 00000000000000f8 c0000000013d09d8 0000000000000001 0000000000000001 
               GPR24: c00000000129c147 0000000000000020 0000000000000018 0000000000000800 
               GPR28: 0000000000000020 0000000000000110 0000000000000000 0000000000000010 
[   61.271760] NIP [c0000000004a7380] .__bitmap_weight+0x70/0x100
[   61.271765] LR [c00000000012c5cc] .build_sched_domains+0xd3c/0x10b0
[   61.271768] Call Trace:
[   61.271772] [c0000007e7d737a0] [c0000000002be454] .kmem_cache_alloc_node_trace+0xb4/0x3a0 (unreliable)
[   61.271780] [c0000007e7d73850] [c00000000012c5cc] .build_sched_domains+0xd3c/0x10b0
[   61.271788] [c0000007e7d739a0] [c00000000012d154] .partition_sched_domains+0x484/0x510
[   61.271794] [c0000007e7d73aa0] [c000000000196518] .rebuild_sched_domains+0x68/0xa0
[   61.271799] [c0000007e7d73b30] [c000000000063b50] .topology_work_fn+0x10/0x30
[   61.271804] [c0000007e7d73ba0] [c0000000000ff7d0] .process_one_work+0x1d0/0x690
[   61.271809] [c0000007e7d73c50] [c0000000000ffe10] .worker_thread+0x180/0x500
[   61.271814] [c0000007e7d73d30] [c00000000010be28] .kthread+0xe8/0xf0
[   61.271820] [c0000007e7d73e30] [c00000000000a470] .ret_from_kernel_thread+0x58/0x68
[   61.271824] Instruction dump:
[   61.271826] 409d00b4 3bbcffff 3be3fff8 7bbd1f48 3bc00000 7fa3ea14 48000018 60000000 
[   61.271835] 60000000 60000000 60000000 60420000 <e87f0009> 4bbc72b9 60000000 7fbfe840 
[   61.271847] ---[ end trace fd234da35d8d8a95 ]---

If the vmcore is not collected following warning can be observed in logs shortly before crash

[   61.271435] ------------[ cut here ]------------
[   61.271439] WARNING: at kernel/sched/core.c:6277
[   61.271442] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_REJECT nf_
conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack nf_conntrack iptable_filter 
pseries_rng ses enclosure sg nfsd auth_rpcgss nfs_acl lockd sunrpc grace binfmt_misc ip_tables 
ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi sc
si_transport_srp ibmveth scsi_tgt ipr libata dm_mirror dm_region_hash dm_log dm_mod
[   61.271485] CPU: 0 PID: 657 Comm: kworker/0:10 Not tainted 3.10.0-327.4.5.el7.ppc64 #1
[   61.271490] Workqueue: events .topology_work_fn
[   61.271493] task: c0000007e6721030 ti: c0000007e7d70000 task.ti: c0000007e7d70000
[   61.271495] NIP: c00000000012c900 LR: c00000000012c5f8 CTR: 0000000000000000
[   61.271497] REGS: c0000007e7d735d0 TRAP: 0700   Not tainted  (3.10.0-327.4.5.el7.ppc64)
[   61.271498] MSR: 8000000100029032 <SF,EE,ME,IR,DR,RI>  CR: 48848022  XER: 00000000
[   61.271503] CFAR: c00000000012c5bc SOFTE: 1 
               GPR00: c00000000012c5f8 c0000007e7d73850 c0000000013822d0 0000000000000010 
               GPR04: 0000000000000800 0000000000000000 0000000000000000 0000000000000000 
               GPR08: 0000000000000000 00000000010b0000 c000000000d36840 0000000000000000 
               GPR12: 0000000000000000 c000000007b30000 c0000007e6b61400 00000000000000b8 
               GPR16: 0000000000000800 c0000007e538fe00 0000000000000000 c0000007e538fe00 
               GPR20: 00000000000000f8 c0000000013d09d8 0000000000000001 0000000000000001 
               GPR24: c00000000129c147 0000000000000017 0000000000000000 c0000007e31f7e00 
               GPR28: c0000000013d0a90 c0000007e6b61400 0000000000000000 c0000000013d5ebc 
[   61.271532] NIP [c00000000012c900] .build_sched_domains+0x1070/0x10b0
[   61.271535] LR [c00000000012c5f8] .build_sched_domains+0xd68/0x10b0
[   61.271537] Call Trace:
[   61.271540] [c0000007e7d73850] [c00000000012c5f8] .build_sched_domains+0xd68/0x10b0 (unrelia
ble)
[   61.271543] [c0000007e7d739a0] [c00000000012d154] .partition_sched_domains+0x484/0x510
[   61.271547] [c0000007e7d73aa0] [c000000000196518] .rebuild_sched_domains+0x68/0xa0
[   61.271550] [c0000007e7d73b30] [c000000000063b50] .topology_work_fn+0x10/0x30
[   61.271553] [c0000007e7d73ba0] [c0000000000ff7d0] .process_one_work+0x1d0/0x690
[   61.271555] [c0000007e7d73c50] [c0000000000ffe10] .worker_thread+0x180/0x500
[   61.271558] [c0000007e7d73d30] [c00000000010be28] .kthread+0xe8/0xf0
[   61.271562] [c0000007e7d73e30] [c00000000000a470] .ret_from_kernel_thread+0x58/0x68
[   61.271564] Instruction dump:
[   61.271566] 4bfffc1c 3ae00000 4bfffc04 7e10a838 4bfffe40 7fe3fb78 48192a29 60000000 
[   61.271572] 39200000 3ae0fff4 f9210078 4bfffbf0 <0fe00000> 4bfffcbc 89580006 694a0001 

Environment

RedHat Enterprise Linux 7.2 on ppc64/ppc64le

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content