The kernel crashes in __queue_work() due to the corrupted workqueue_struct being messed up, overwritten illegally
Issue
- The kernel crashes in __queue_work() with a kernel stack backtrace like this:
[78636409.874892] Unable to handle kernel paging request for data at address 0x1800c02d80000
[78636409.874923] Faulting instruction address: 0xc0000000000f1134
[78636409.874933] Oops: Kernel access of bad area, sig: 11 [#1]
[78636409.874938] SMP NR_CPUS=2048 NUMA pSeries
[78636409.874948] Modules linked in: sg iptable_filter ip_tables cfg80211 rfkill 8021q garp stp mrp llc bonding nx_crypto pseries_rng ses enclosure binfmt_misc uinput xfs dm_service_time sr_mod cdrom sd_mod lpfc bnx2x ipr libata crc_t10dif crct10dif_common scsi_transport_fc mdio ptp pps_core scsi_tgt libcrc32c dm_mirror dm_region_hash dm_log dm_multipath dm_mod
[78636409.875029] CPU: 9 PID: 0 Comm: swapper/9 Not tainted 3.10.0-229.el7.ppc64 #1
[78636409.875036] task: c000003c98659830 ti: c000007b9ff70000 task.ti: c000003c98744000
[78636409.875041] NIP: c0000000000f1134 LR: c0000000000d6bec CTR: c0000000000f13e0
[78636409.875048] REGS: c000007b9ff73990 TRAP: 0300 Not tainted (3.10.0-229.el7.ppc64)
[78636409.875055] MSR: 8000000000009032 <SF,EE,ME,IR,DR,RI> CR: 28000082 XER: 20000000
[78636409.875081] CFAR: c000000000009358 DAR: 0001800c02d80000 DSISR: 40000000 SOFTE: 0
GPR00: c0000000000d6bec c000007b9ff73c10 c00000000130ae00 0000000000000800
GPR04: 000000007fffffff c000007b40e7ae28 be12fcdc8847ae28 4cd2fcdc8847ae28
GPR08: c0000000013509d8 000000007fffffff 0000000000000000 012b9cccc3392294
GPR12: 0000000088000082 c000000007b15100 c000003c98747f90 0000000010200040
GPR16: c000003c98b55428 c000003c98b55828 c000003c98b55c28 0000000000000000
GPR20: 0000000000000000 c0000000015bca70 0000000000000000 000000007fffffff
GPR24: 0000000000000188 c0000000015bca68 c000003c8faa0000 0000000000000009
GPR28: 0000000000000800 0001800c02d80000 c000007b40e7ae48 c000007b40e7ae28
[78636409.875185] NIP [c0000000000f1134] .__queue_work+0x3e4/0x5d0
[78636409.875192] LR [c0000000000d6bec] .call_timer_fn+0x5c/0x180
[78636409.875197] Call Trace:
[78636409.875203] [c000007b9ff73c10] [c000007b9ff73ce0] 0xc000007b9ff73ce0 (unreliable)
[78636409.875211] [c000007b9ff73ce0] [c0000000000d6bec] .call_timer_fn+0x5c/0x180
[78636409.875217] [c000007b9ff73d90] [c0000000000d8a80] .run_timer_softirq+0x190/0x380
[78636409.875227] [c000007b9ff73e90] [c0000000000cd518] .__do_softirq+0x148/0x380
[78636409.875235] [c000007b9ff73f90] [c000000000023f3c] .call_do_softirq+0x14/0x24
[78636409.875242] [c000003c98747710] [c000000000011160] .do_softirq+0x120/0x170
[78636409.875249] [c000003c987477a0] [c0000000000cdaa4] .irq_exit+0x1e4/0x1f0
[78636409.875257] [c000003c98747830] [c00000000001e5bc] .timer_interrupt+0x9c/0xd0
[78636409.875265] [c000003c987478b0] [c000000000002794] decrementer_common+0x114/0x180
[78636409.875276] --- Exception: 901 at .snooze_loop+0x88/0x170
LR = .snooze_loop+0x40/0x170
[78636409.875285] [c000003c98747ba0] [c000003c98747c40] 0xc000003c98747c40 (unreliable)
[78636409.875294] [c000003c98747c40] [c00000000070e254] .cpuidle_idle_call+0x114/0x3c0
[78636409.875304] [c000003c98747d10] [c0000000000872f0] .pseries_lpar_idle+0x10/0x50
[78636409.875310] [c000003c98747d80] [c000000000017b44] .arch_cpu_idle+0x64/0x150
[78636409.875319] [c000003c98747e00] [c0000000001497b0] .cpu_startup_entry+0x270/0x2e0
[78636409.875328] [c000003c98747ed0] [c000000000047b70] .start_secondary+0x310/0x340
[78636409.875339] [c000003c98747f90] [c000000000009a6c] start_secondary_prolog+0x10/0x14
[78636409.875348] Instruction dump:
[78636409.875355] 4bfffe38 60000000 60420000 e93e0031 7fa9f040 41defe80 2fa90000 41defe78
[78636409.875371] e8690040 48025959 60000000 4bfffe68 <e87d0000> 60000000 60420000 48800ce1
[78636409.875388] ---[ end trace 435a40a7f14b7ad8 ]---
[78636409.890826]
[78636409.890840] Sending IPI to other CPUs
[78636409.891995] IPI complete
Environment
- Red Hat Enterprise Linux 7.1 for Power, big endian (kernel-3.10.0-229.el7.ppc64)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.