RHEL8: Soft lockup occurred during __page_mapcount and split_huge_page_to_list

Solution Verified - Updated -

Issue

  • soft lockup occurred with logs:
[241888.165144] watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [kswapd1:514]
[241888.173282] Modules linked in: mptcp_diag xsk_diag raw_diag unix_diag af_packet_diag netlink_diag udp_diag tcp_diag inet_diag rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache bonding tls intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal mgag200 intel_powerclamp coretemp irqbypass i2c_algo_bit crct10dif_pclmul crc32_pclmul drm_shmem_helper drm_kms_helper ghash_clmulni_intel ipmi_ssif syscopyarea rapl sysfillrect sysimgblt fb_sys_fops iTCO_wdt intel_cstate ioatdma iTCO_vendor_support usbnet mei_me mii i2c_i801 mxm_wmi drm lpc_ich intel_uncore mei pcspkr dca ipmi_si ipmi_devintf acpi_pad ipmi_msghandler wmi acpi_power_meter vfat fat binfmt_misc auth_rpcgss sunrpc xfs libcrc32c sd_mod t10_pi sg crc32c_intel tg3 megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[241888.251280] CPU: 45 PID: 514 Comm: kswapd1 Kdump: loaded Not tainted 4.18.0-425.10.1.el8_7.x86_64 #1
[241888.261696] Hardware name: LENOVO Lenovo NeXtScale nx360 M5: -[5465AC1]-/00YE752, BIOS -[THE130I-2.40]- 04/05/2017
[241888.273473] RIP: 0010:native_queued_spin_lock_slowpath+0x61/0x1c0
[241888.280502] Code: 0f ba 2f 08 0f 92 c0 0f b6 c0 c1 e0 08 89 c2 8b 07 30 e4 09 d0 a9 00 01 ff ff 75 4b 85 c0 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 e9 1e b7 aa 00 8b 37 81 fe 00
[241888.301845] RSP: 0018:ffff9c810847f998 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13
[241888.310540] RAX: 0000000000000101 RBX: ffff9c810847f9d8 RCX: 000ffffffffff000
[241888.318752] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffe0d80e488f28
[241888.326968] RBP: ffff8c2736d34e00 R08: 0000000000000067 R09: ffffe0d80e518000
[241888.335188] R10: ffffe0d80e518048 R11: ffffe0d80e518008 R12: ffffe0d80e518000
[241888.343421] R13: 03ffffffffffffff R14: 0000039fc6a00000 R15: 7c00000000000000
[241888.351638] FS:  0000000000000000(0000) GS:ffff8c2dffa40000(0000) knlGS:0000000000000000
[241888.360938] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[241888.367616] CR2: 000003dd36840080 CR3: 0000001712a10003 CR4: 00000000003706e0
[241888.375850] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[241888.384087] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[241888.392318] Call Trace:
[241888.395316]  _raw_spin_lock+0x1e/0x30
[241888.399678]  page_vma_mapped_walk+0x3b8/0x8b0
[241888.404817]  remove_migration_pte+0x7f/0x420
[241888.409859]  rmap_walk_anon+0xe7/0x2a0
[241888.414320]  remove_migration_ptes+0x53/0x80
[241888.419362]  ? migrate_vma_check_page+0xb0/0xb0
[241888.424698]  remap_page+0x4a/0x70
[241888.428676]  split_huge_page_to_list+0x894/0xa50
[241888.434113]  ? free_unref_page_commit+0x91/0x110
[241888.439553]  deferred_split_scan+0x1ca/0x310
[241888.444606]  do_shrink_slab+0x11d/0x330
[241888.449176]  shrink_slab+0x249/0x2f0
[241888.453454]  shrink_node+0x246/0x700
[241888.457730]  balance_pgdat+0x2d7/0x550
[241888.462206]  kswapd+0x201/0x3c0
[241888.465999]  ? finish_wait+0x80/0x80
[241888.470285]  ? balance_pgdat+0x550/0x550
[241888.474958]  kthread+0x10b/0x130
[241888.478857]  ? set_kthread_struct+0x50/0x50
[241888.483822]  ret_from_fork+0x35/0x40
  • Another logs:
[73592.755861] watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [kernel_server:2399822]
[73592.831735] CPU: 28 PID: 2329307 Comm: kernel_server Kdump: loaded Not tainted 4.18.0-425.10.1.el8_7.x86_64 #1
[73592.852000] Hardware name: LENOVO Lenovo NeXtScale nx360 M5: -[5465AC1]-/00YE752, BIOS -[THE130I-2.40]- 04/05/2017
[73592.855525] RIP: 0010:__page_mapcount+0x98/0xa0
[73593.024082] Call Trace:
[73592.855525] RIP: 0010:__page_mapcount+0x98/0xa0
[73593.036206]  smaps_account+0xb9/0x230
[73593.042702]  smaps_pte_range+0x343/0x600
[73593.050760]  walk_pgd_range+0x46f/0xa20
[73593.058043]  __walk_page_range+0x57/0x180
[73593.067342]  walk_page_vma+0x66/0x90
[73593.077020]  smap_gather_stats+0x42/0xa0
[73593.086289]  show_smap+0x42/0x1c0
[73593.095065]  seq_read+0x303/0x420
[73593.101299]  vfs_read+0x91/0x150
[73593.107999]  ksys_read+0x4f/0xb0
[73593.115663]  do_syscall_64+0x5b/0x1b0
[73593.121939]  entry_SYSCALL_64_after_hwframe+0x61/0xc6

Environment

  • Red Hat Enterprise Linux 8.7
  • Red Hat Enterprise Linux 8.6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content