"WARNING: CPU: 198 PID: 313357 at kernel/rcutree.c:2221" is logged and then Red Hat Enterprise Linux 7.8 kernel-3.10.0-1127.18.2.el7.ppc64le crashes in wq_worker_waking_up().

Issue

• "WARNING: CPU: 198 PID: 313357 at kernel/rcutree.c:2221" is logged in the kernel ring buffer.

• The kernel crashes in wq_worker_waking_up() just afer that.

[8003198.645146] ------------[ cut here ]------------
[8003198.645370] WARNING: CPU: 198 PID: 313357 at kernel/rcutree.c:2221 rcu_process_callbacks+0x88c/0x8f0
[8003198.645675] Modules linked in: binfmt_misc sg rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding i2c_dev sunrpc at24 ofpart ipmi_powernv ipmi_devintf powernv_flash raid10 ipmi_msghandler mtd ibmpowernv opal_prd powernv_rng i2c_opal ip_tables xfs libcrc32c raid1 ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm tg3 mlxfw devlink drm_panel_orientation_quirks i2c_core ptp nvme pps_core nvme_core dm_mirror dm_region_hash dm_log dm_mod
[8003198.647509] CPU: 198 PID: 313357 Comm: gzip Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.18.2.el7.ppc64le #1
[8003198.647884] task: c0000000069457c0 ti: c00000ffff9cc000 task.ti: c0000110db9dc000
[8003198.648153] NIP: c00000000021752c LR: c000000000216ff0 CTR: c000000000592ca0
[8003198.648371] REGS: c00000ffff9cfb80 TRAP: 0700   Tainted: G               ------------ T  (3.10.0-1127.18.2.el7.ppc64le)
[8003198.648814] MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28022424  XER: 00000000
[8003198.649175] CFAR: c000000000217104 SOFTE: 0 
                 GPR00: c000000000216ff0 c00000ffff9cfe00 c00000000147ba00 0000000000000001 
                 GPR04: f00000003b69e480 c000010f9af03b60 000000000000000a 0000000000002710 
                 GPR08: c000000000175708 0000000000000001 0000000000000001 0000000000000000 
                 GPR12: 0000000000000000 c000000007b9f600 0000000000000000 0000000000400000 
                 GPR16: 0000000000000001 0000000000000000 0000000000000000 7fffffffffffffff 
                 GPR20: c000000000b33e58 c000000001432800 c000010f9af03b60 c00000ffff9cc000 
                 GPR24: c00000012618d428 000000000000000a ffffffffffffffff c000000001429900 
                 GPR28: 0000000000000000 c00000012618d400 0000000000000000 c000010f9af03b60 
[8003198.652428] NIP [c00000000021752c] rcu_process_callbacks+0x88c/0x8f0
[8003198.652638] LR [c000000000216ff0] rcu_process_callbacks+0x350/0x8f0
[8003198.652820] Call Trace:
[8003198.652897] [c00000ffff9cfe00] [c000000000216ff0] rcu_process_callbacks+0x350/0x8f0 (unreliable)
[8003198.653225] [c00000ffff9cfea0] [c000000000101554] __do_softirq+0x154/0x380
[8003198.653450] [c00000ffff9cff90] [c00000000002d87c] call_do_softirq+0x14/0x24
[8003198.653685] [c0000110db9dfd80] [c0000000000186d0] do_softirq+0x130/0x180
[8003198.653907] [c0000110db9dfdc0] [c000000000101ae4] irq_exit+0x1f4/0x200
[8003198.654122] [c0000110db9dfe00] [c000000000027568] timer_interrupt+0x98/0xf0
[8003198.654357] [c0000110db9dfe30] [c000000000002c14] decrementer_common+0x114/0x118
[8003198.654577] Instruction dump:
[8003198.654705] 3d02fff7 8908987c 69080001 0b080000 2fa80000 41fef888 3ce2fff7 9a07987c 
[8003198.654997] 4bfff87c 3d22fff7 8929987e 69290001 <0b090000> 2fa90000 41fefbd4 3cc2fff7 
[8003198.655318] ---[ end trace 120546e37ec2f895 ]---
[8003269.390913] Unable to handle kernel paging request for data at address 0x02000058
[8003269.391179] Faulting instruction address: 0xc000000000131c9c
[8003269.391403] Oops: Kernel access of bad area, sig: 11 [#1]
[8003269.391540] SMP NR_CPUS=2048 NUMA PowerNV
[8003269.391752] Modules linked in: binfmt_misc sg rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding i2c_dev sunrpc at24 ofpart ipmi_powernv ipmi_devintf powernv_flash raid10 ipmi_msghandler mtd ibmpowernv opal_prd powernv_rng i2c_opal ip_tables xfs libcrc32c raid1 ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm tg3 mlxfw devlink drm_panel_orientation_quirks i2c_core ptp nvme pps_core nvme_core dm_mirror dm_region_hash dm_log dm_mod
[8003269.393632] CPU: 198 PID: 0 Comm: swapper/198 Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1127.18.2.el7.ppc64le #1
[8003269.394122] task: c000017f4d1483a0 ti: c00000ffff9c8000 task.ti: c000017f4d198000
[8003269.394387] NIP: c000000000131c9c LR: c000000000131c98 CTR: c000000000171880
[8003269.394627] REGS: c00000ffff9cb690 TRAP: 0300   Tainted: G        W      ------------ T  (3.10.0-1127.18.2.el7.ppc64le)
[8003269.395040] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 48002828  XER: 20000000
[8003269.395383] CFAR: c0000000000093ec DAR: 0000000002000058 DSISR: 40000000 SOFTE: 0 
                 GPR00: c000000000131c98 c00000ffff9cb910 c00000000147ba00 0000000002000000 
                 GPR04: 00000000000000c6 0000000000000001 0000000000000001 0000000000000001 
                 GPR08: 0000000000000218 c000010f9af03dd8 c000000001750a80 00000000ffffffff 
                 GPR12: 0000000000002200 c000000007b9f600 c000017f4d19bf90 0000000000000000 
                 GPR16: 0000000000000000 c00000000002d0d8 c000000000eec980 0000000000000000 
                 GPR20: c00000000174cd88 0000000000000000 c0000001261a0c00 0000000000000000 
                 GPR24: c00000017841b164 0000000000000000 00000000000000c6 00000001252a0000 
                 GPR28: c0000000014c2360 c000000000f00c00 c0000001261a0c00 00000000000000c6 
[8003269.397723] NIP [c000000000131c9c] wq_worker_waking_up+0x2c/0xa0
[8003269.397936] LR [c000000000131c98] wq_worker_waking_up+0x28/0xa0
[8003269.398121] Call Trace:
[8003269.398259] [c00000ffff9cb910] [c0000000014c2360] __per_cpu_offset+0x0/0x4000 (unreliable)
[8003269.398571] [c00000ffff9cb940] [c000000000159f3c] try_to_wake_up+0x29c/0x630
[8003269.398849] [c00000ffff9cb9d0] [c00000000012c87c] __queue_work+0x44c/0x6f0
[8003269.399070] [c00000ffff9cbaa0] [c00000000012cbe4] queue_work_on+0xc4/0xd0
[8003269.399391] [c00000ffff9cbad0] [d0000000c8adb7b4] xfs_buf_bio_end_io+0xb4/0xe0 [xfs]
[8003269.399710] [c00000ffff9cbb00] [c000000000408ba0] bio_endio+0x160/0x230
[8003269.399993] [c00000ffff9cbb50] [c000000000546fb4] blk_update_request+0xf4/0x4b0
[8003269.400262] [c00000ffff9cbc00] [c000000000558084] blk_mq_end_request+0x34/0xd0
[8003269.400508] [c00000ffff9cbc30] [d0000000c5dc2448] nvme_complete_rq+0x38/0x150 [nvme_core]
[8003269.400809] [c00000ffff9cbc50] [d0000000c6063438] nvme_pci_complete_rq+0x1d8/0x260 [nvme]
[8003269.401078] [c00000ffff9cbcb0] [c000000000556340] __blk_mq_complete_request_remote+0x30/0x50
[8003269.401378] [c00000ffff9cbcd0] [c0000000001b19a0] flush_smp_call_function_queue+0xb0/0x1b0
[8003269.401513] [c00000ffff9cbd50] [c000000000056a4c] smp_ipi_demux+0xac/0x180
[8003269.401914] [c00000ffff9cbd90] [c00000000008b0dc] icp_native_ipi_action+0x6c/0x80
[8003269.402198] [c00000ffff9cbdb0] [c000000000205e70] __handle_irq_event_percpu+0xa0/0x2f0
[8003269.402484] [c00000ffff9cbe80] [c0000000002060f8] handle_irq_event_percpu+0x38/0x90
[8003269.402822] [c00000ffff9cbec0] [c00000000020c0a8] handle_percpu_irq+0x88/0xd0
[8003269.403085] [c00000ffff9cbef0] [c000000000204f98] generic_handle_irq+0x58/0x90
[8003269.403377] [c00000ffff9cbf20] [c000000000018280] __do_irq+0x90/0x1a0
[8003269.403601] [c00000ffff9cbf90] [c00000000002d8a0] call_do_irq+0x14/0x24
[8003269.403895] [c000017f4d19baa0] [c00000000001842c] do_IRQ+0x9c/0x110
[8003269.404155] [c000017f4d19baf0] [c000000000002a94] hardware_interrupt_common+0x114/0x180
[8003269.404472] --- Exception: 501 at arch_local_irq_restore+0x74/0x150
                     LR = cpuidle_idle_call+0x140/0x410
[8003269.404907] [c000017f4d19bde0] [c00000012619fb88] 0xc00000012619fb88 (unreliable)
[8003269.405207] [c000017f4d19be00] [c0000000008a9450] cpuidle_idle_call+0x140/0x410
[8003269.405542] [c000017f4d19be70] [c00000000008cb90] powernv_idle+0x20/0x50
[8003269.405793] [c000017f4d19be90] [c00000000001f8c0] arch_cpu_idle+0x70/0x160
[8003269.406031] [c000017f4d19bec0] [c00000000018d800] cpu_startup_entry+0x190/0x210
[8003269.406274] [c000017f4d19bf20] [c000000000057b30] start_secondary+0x310/0x340
[8003269.406560] [c000017f4d19bf90] [c000000000009b6c] start_secondary_prolog+0x10/0x14
[8003269.406850] Instruction dump:
[8003269.406978] 60420000 3c4c0135 38429d90 7c0802a6 60000000 7c0802a6 fbe1fff8 f8010010 
[8003269.407310] f821ffd1 7c9f2378 48009acd 60000000 <81230058> 712801c8 40820028 e9230048 
[8003269.414615] ---[ end trace 120546e37ec2f896 ]---
[8003269.483632] 
[8003269.483728] Sending IPI to other CPUs
[8003269.485129] IPI complete