/var/log/message showing call traces "percpu ref (blkg_release) <= 0 (-*) after switching to atomic"

Solution Verified - Updated -

Issue

  • /var/log/messages showing the below call trace messages .
[  650.772713] FS-Cache: Loaded
[  650.805951] FS-Cache: Netfs 'nfs' registered for caching
[  698.783263] ------------[ cut here ]------------
[  698.783265] percpu ref (blkg_release) <= 0 (-5) after switching to atomic
[  698.783281] WARNING: CPU: 22 PID: 0 at lib/percpu-refcount.c:161 percpu_ref_switch_to_atomic_rcu+0x135/0x140
[  698.783290] Modules linked in: nfs lockd grace fscache ext4 mbcache jbd2 dmpjbod(POE) dmpap(POE) dmpvmax(POE) dmpaa(POE) ktap_112810(OE) vxfen(POE) vxglm(POE) vxodm(POE) gab(POE) vxspec(POE) vxio(POE) vxdmp(POE) amf(POE) vxcafs(POE) vxportal(POE) fdd(POE) llt(POE) vxfs(POE) veki(POE) intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl intel_cstate intel_uncore pcspkr bonding tls vfat fat ses enclosure scsi_transport_sas rpcrdma rdma_ucm ib_srpt ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm ib_cm iTCO_wdt iTCO_vendor_support ipmi_ssif qedr mgag200 ib_uverbs i2c_algo_bit drm_kms_helper ib_core syscopyarea sysfillrect sysimgblt fb_sys_fops mei_me ioatdma drm joydev mii mei i2c_i801 lpc_ich dca wmi ipmi_si acpi_power_meter acpi_pad binfmt_misc auth_rpcgss sunrpc xfs dm_thin_pool dm_persistent_data
[  698.783384]  dm_bio_prison dm_bufio libcrc32c sd_mod sg qede lpfc qed nvmet_fc nvmet crc32c_intel nvme_fc nvme_fabrics ahci nvme_core megaraid_sas libahci libata t10_pi scsi_transport_fc crc8 dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler
[  698.783409] CPU: 22 PID: 0 Comm: swapper/22 Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-372.32.1.el8_6.x86_64 #1
[  698.783413] Hardware name: Lenovo ThinkSystem SR650 -[7X06CTO1WW]-/-[7X06CTO1WW]-, BIOS -[IVE174G-3.11]- 10/18/2021
[  698.783415] RIP: 0010:percpu_ref_switch_to_atomic_rcu+0x135/0x140
[  698.783420] Code: 80 3d eb 1e 89 01 00 0f 85 53 ff ff ff 49 8b 54 24 d8 49 8b 74 24 e8 48 c7 c7 08 5f 52 9e c6 05 cd 1e 89 01 01 e8 18 62 c2 ff <0f> 0b e9 2f ff ff ff 0f 1f 40 00 41 54 49 89 f4 55 53 48 89 fb 48
[  698.783422] RSP: 0018:ffffc0f9192e0eb0 EFLAGS: 00010282
[  698.783424] RAX: 0000000000000000 RBX: 7ffffffffffffffa RCX: 0000000000000007
[  698.783426] RDX: 0000000000000007 RSI: 0000000000000002 RDI: ffff9cb020196750
[  698.783428] RBP: 00004400d300da60 R08: ffffffff9f87df80 R09: 0000000000000000
[  698.783429] R10: ffff9cf93fbd1abe R11: ffff9cf93fbd1aa2 R12: ffff9cf29f1f9868
[  698.783431] R13: ffffffff9f1adfe0 R14: 0000000000000000 R15: 0000000000000001
[  698.783432] FS:  0000000000000000(0000) GS:ffff9cb020180000(0000) knlGS:0000000000000000
[  698.783434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  698.783436] CR2: 00005631c51422e9 CR3: 0000002c5b410006 CR4: 00000000007706e0
[  698.783438] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  698.783439] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  698.783440] PKRU: 55555554
[  698.783442] Call Trace:
[  698.783444]  <IRQ>
[  698.783447]  rcu_do_batch+0x1c5/0x470
[  698.783456]  rcu_core+0x16a/0x230
[  698.783460]  __do_softirq+0xd7/0x2c8
[  698.783470]  irq_exit_rcu+0xd7/0xe0
[  698.783475]  irq_exit+0xa/0x10
[  698.783477]  smp_apic_timer_interrupt+0x74/0x130
[  698.783483]  apic_timer_interrupt+0xf/0x20
[  698.783487]  </IRQ>
[  698.783488] RIP: 0010:cpuidle_enter_state+0xda/0x3d0
[  698.783499] Code: e8 7b 52 9a ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 aa 02 00 00 31 ff e8 5d 4d a1 ff fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 29 01 00 00 49 63 d6 48 8b 4c 24 10 48 2b 0c 24 48
[  698.783502] RSP: 0018:ffffc0f90030be58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  698.783505] RAX: ffff9cb0201aae40 RBX: ffffffff9f0b8528 RCX: 000000000000001f
[  698.783506] RDX: 000000a2b2ba2cf5 RSI: 00000000281864db RDI: 0000000000000000
[  698.783508] RBP: ffffe0b104580000 R08: 0000000000000002 R09: 000000000002a680
[  698.783509] R10: 00737b0ae3d386bc R11: ffff9cb0201a9b44 R12: 0000000000000003
[  698.783511] R13: ffffffff9f0b83c0 R14: 0000000000000003 R15: 0000000000000003
[  698.783515]  ? cpuidle_enter_state+0xb5/0x3d0
[  698.783520]  cpuidle_enter+0x2c/0x40
[  698.783525]  do_idle+0x268/0x2d0
[  698.783532]  cpu_startup_entry+0x6f/0x80
[  698.783535]  start_secondary+0x1a6/0x1f0
[  698.783547]  secondary_startup_64_no_verify+0xc2/0xcb
[  698.783556] ---[ end trace 770538e484206b65 ]---
[  821.087989] usb 1-1.1: USB disconnect, device number 3

Environment

  • Red Hat Enterprise Linux 8
  • Veritas Volume Manager (VxVM)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content