The kernel crashes in __queue_work() due to the corrupted workqueue_struct being messed up, overwritten illegally

Solution Unverified - Updated -

Issue

  • The kernel crashes in __queue_work() with a kernel stack backtrace like this:
[78636409.874892] Unable to handle kernel paging request for data at address 0x1800c02d80000
[78636409.874923] Faulting instruction address: 0xc0000000000f1134
[78636409.874933] Oops: Kernel access of bad area, sig: 11 [#1]
[78636409.874938] SMP NR_CPUS=2048 NUMA pSeries
[78636409.874948] Modules linked in: sg iptable_filter ip_tables cfg80211 rfkill 8021q garp stp mrp llc bonding nx_crypto pseries_rng ses enclosure binfmt_misc uinput xfs dm_service_time sr_mod cdrom sd_mod lpfc bnx2x ipr libata crc_t10dif crct10dif_common scsi_transport_fc mdio ptp pps_core scsi_tgt libcrc32c dm_mirror dm_region_hash dm_log dm_multipath dm_mod
[78636409.875029] CPU: 9 PID: 0 Comm: swapper/9 Not tainted 3.10.0-229.el7.ppc64 #1
[78636409.875036] task: c000003c98659830 ti: c000007b9ff70000 task.ti: c000003c98744000
[78636409.875041] NIP: c0000000000f1134 LR: c0000000000d6bec CTR: c0000000000f13e0
[78636409.875048] REGS: c000007b9ff73990 TRAP: 0300   Not tainted  (3.10.0-229.el7.ppc64)
[78636409.875055] MSR: 8000000000009032 <SF,EE,ME,IR,DR,RI>  CR: 28000082  XER: 20000000
[78636409.875081] CFAR: c000000000009358 DAR: 0001800c02d80000 DSISR: 40000000 SOFTE: 0 
                  GPR00: c0000000000d6bec c000007b9ff73c10 c00000000130ae00 0000000000000800 
                  GPR04: 000000007fffffff c000007b40e7ae28 be12fcdc8847ae28 4cd2fcdc8847ae28 
                  GPR08: c0000000013509d8 000000007fffffff 0000000000000000 012b9cccc3392294 
                  GPR12: 0000000088000082 c000000007b15100 c000003c98747f90 0000000010200040 
                  GPR16: c000003c98b55428 c000003c98b55828 c000003c98b55c28 0000000000000000 
                  GPR20: 0000000000000000 c0000000015bca70 0000000000000000 000000007fffffff 
                  GPR24: 0000000000000188 c0000000015bca68 c000003c8faa0000 0000000000000009 
                  GPR28: 0000000000000800 0001800c02d80000 c000007b40e7ae48 c000007b40e7ae28 
[78636409.875185] NIP [c0000000000f1134] .__queue_work+0x3e4/0x5d0
[78636409.875192] LR [c0000000000d6bec] .call_timer_fn+0x5c/0x180
[78636409.875197] Call Trace:
[78636409.875203] [c000007b9ff73c10] [c000007b9ff73ce0] 0xc000007b9ff73ce0 (unreliable)
[78636409.875211] [c000007b9ff73ce0] [c0000000000d6bec] .call_timer_fn+0x5c/0x180
[78636409.875217] [c000007b9ff73d90] [c0000000000d8a80] .run_timer_softirq+0x190/0x380
[78636409.875227] [c000007b9ff73e90] [c0000000000cd518] .__do_softirq+0x148/0x380
[78636409.875235] [c000007b9ff73f90] [c000000000023f3c] .call_do_softirq+0x14/0x24
[78636409.875242] [c000003c98747710] [c000000000011160] .do_softirq+0x120/0x170
[78636409.875249] [c000003c987477a0] [c0000000000cdaa4] .irq_exit+0x1e4/0x1f0
[78636409.875257] [c000003c98747830] [c00000000001e5bc] .timer_interrupt+0x9c/0xd0
[78636409.875265] [c000003c987478b0] [c000000000002794] decrementer_common+0x114/0x180
[78636409.875276] --- Exception: 901 at .snooze_loop+0x88/0x170
                      LR = .snooze_loop+0x40/0x170
[78636409.875285] [c000003c98747ba0] [c000003c98747c40] 0xc000003c98747c40 (unreliable)
[78636409.875294] [c000003c98747c40] [c00000000070e254] .cpuidle_idle_call+0x114/0x3c0
[78636409.875304] [c000003c98747d10] [c0000000000872f0] .pseries_lpar_idle+0x10/0x50
[78636409.875310] [c000003c98747d80] [c000000000017b44] .arch_cpu_idle+0x64/0x150
[78636409.875319] [c000003c98747e00] [c0000000001497b0] .cpu_startup_entry+0x270/0x2e0
[78636409.875328] [c000003c98747ed0] [c000000000047b70] .start_secondary+0x310/0x340
[78636409.875339] [c000003c98747f90] [c000000000009a6c] start_secondary_prolog+0x10/0x14
[78636409.875348] Instruction dump:
[78636409.875355] 4bfffe38 60000000 60420000 e93e0031 7fa9f040 41defe80 2fa90000 41defe78 
[78636409.875371] e8690040 48025959 60000000 4bfffe68 <e87d0000> 60000000 60420000 48800ce1 
[78636409.875388] ---[ end trace 435a40a7f14b7ad8 ]---
[78636409.890826] 
[78636409.890840] Sending IPI to other CPUs
[78636409.891995] IPI complete

Environment

  • Red Hat Enterprise Linux 7.1 for Power, big endian (kernel-3.10.0-229.el7.ppc64)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content