A VM is getting stuck upon shutting down. pool->lock spinlock and rq.lock spinlock are deadlocked.

Issue

• A VM was getting stuck upon shutting down. The customer injected NMI to capture a dump.

[  278.323387] 3:Shutdown timed out. 0, pid=1879
[  278.323408] 3:Shutdown timed out. 0, pid=6260
[  278.323426] 3:Shutdown timed out. 0, pid=49
[  278.323428] 3:Shutdown timed out. 0, pid=0
[  278.323466] 3:Shutdown timed out. 0, pid=6062
[  278.323468] 2:Shutdown timed out. 3, pid=1883
[  278.323493] 3:Shutdown timed out. 0, pid=1883
[  278.323892] 3:Shutdown timed out. 0, pid=680
[  278.323908] 3:Shutdown timed out. 0, pid=13
[  278.324112] 1:Shutdown timed out. ec, pid=0
[  278.324115] 1:Shutdown timed out. ec, pid=0
[  278.324119] 1:Shutdown timed out. ec, pid=0
[  338.325663] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  357.629428] Uhhuh. NMI received for unknown reason 30 on CPU 0.
[  357.629431] Do you have a strange power saving mode enabled?
[  357.629432] Kernel panic - not syncing: NMI: Not continuing
[  357.629432] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Tainted: G           OE    --------- -  - 4.18.0-425.19.2.el8_7.x86_64 #1
[  357.629433] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[  357.629434] Call Trace:
[  357.629434]  <NMI>
[  357.629435]  dump_stack+0x41/0x60
[  357.629435]  panic+0xe7/0x2ac
[  357.629436]  ? printk+0x58/0x73
[  357.629436]  nmi_panic.cold.11+0xc/0xc
[  357.629437]  unknown_nmi_error.cold.13+0x47/0x55
[  357.629437]  do_nmi+0x1af/0x220
[  357.629438]  end_repeat_nmi+0x16/0x69
[  357.629438] RIP: 0010:__pv_queued_spin_lock_slowpath+0x76/0x2a0
[  357.629438] Code: 00 00 c7 45 08 00 00 00 00 31 ff 48 c7 45 00 00 00 00 00 c6 45 14 00 65 8b 05 76 9e 4b 7d 89 45 10 eb 07 80 e6 ff 75 1f f3 90 <8b> 13 66 85 d2 75 0e 89 f8 f0 0f b0 0b 84 c0 0f 84 b0 00 00 00 f7
[  357.629439] RSP: 0018:ffffb08c40003ee0 EFLAGS: 00000046
[  357.629440] RAX: 0000000000000000 RBX: ffff8a963bc6af40 RCX: 0000000000000001
[  357.629441] RDX: 0000000000080003 RSI: 0000000000000000 RDI: 0000000000000000
[  357.629441] RBP: ffff8a963bc2bcc0 R08: 0000666da583aace R09: 0000000000000000
[  357.629442] R10: 0000000000000000 R11: 0000000000000000 R12: 00000001000095e9
[  357.629442] R13: 0000000000000002 R14: ffff8a963bc6af40 R15: 0000000000000000
[  357.629442]  ? __pv_queued_spin_lock_slowpath+0x76/0x2a0
[  357.629443]  ? __pv_queued_spin_lock_slowpath+0x76/0x2a0
[  357.629443]  </NMI>
[  357.629443]  <IRQ>
[  357.629444]  _raw_spin_lock_irqsave+0x34/0x40
[  357.629444]  run_rebalance_domains+0x17f/0x330
[  357.629444]  ? sched_clock_cpu+0xc/0xb0
[  357.629445]  __do_softirq+0xdc/0x2cf
[  357.629445]  irq_exit_rcu+0xd5/0xe0
[  357.629445]  irq_exit+0xa/0x10
[  357.629446]  call_function_single_interrupt+0xf/0x20
[  357.629446]  </IRQ>
[  357.629446] RIP: 0010:native_safe_halt+0xe/0x20
[  357.629447] Code: 00 f0 80 48 02 20 48 8b 00 a8 08 75 c0 e9 79 ff ff ff 90 90 90 90 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d 46 96 41 00 fb f4 <e9> 5d 09 21 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
[  357.629448] RSP: 0018:ffffffff84203e78 EFLAGS: 00000212 ORIG_RAX: ffffffffffffff04
[  357.629449] RAX: ffffffff833f09b0 RBX: 0000000000000000 RCX: 7fffffbf2f8e983f
[  357.629449] RDX: 00000000000b7d7e RSI: 0000000000000000 RDI: ffff8a963bc1d580
[  357.629449] RBP: 0000000000000000 R08: 0000666da188a3c0 R09: 0000000000000001
[  357.629450] R10: 0000000000000004 R11: 0000000000000091 R12: 0000000000000000
[  357.629450] R13: 0000000000000000 R14: ffffffffffffffff R15: ffffffff84218840
[  357.629451]  ? ldsem_down_write+0x270/0x270
[  357.629454]  default_idle+0xa/0x20
[  357.629454]  default_idle_call+0x44/0xf0
[  357.629455]  do_idle+0x213/0x2d0
[  357.629455]  cpu_startup_entry+0x6f/0x80
[  357.629456]  start_kernel+0x522/0x546
[  357.629456]  secondary_startup_64_no_verify+0xc2/0xcb