A server frequently crashes in vol_ru_verification_data_unpack() due to severe stack overflows/corruption and slab corruption caused by Veritas VxVM (vxio) issue

Solution Verified - Updated -

Issue

  • A server frequently crashes in vol_ru_verification_data_unpack() due to Veritas VxVM (vxio) issue.
[Fri May 16 23:47:53 UTC 2025] VxVM VVR vxio V-5-0-645 Failing reads on rvg aial03_rvg. Rlink  rlk_aial03p_aial03_rvg is inconsistent. :
[Fri May 16 23:47:53 UTC 2025] VxVM VVR vxio V-5-0-645 Failing reads on rvg aial03_rvg. Rlink  rlk_aial03p_aial03_rvg is inconsistent. :
[Fri May 16 23:47:53 UTC 2025] VxVM VVR vxio V-5-0-645 Failing reads on rvg aial03_rvg. Rlink  rlk_aial03p_aial03_rvg is inconsistent. :
[Fri May 16 23:47:53 UTC 2025] VxVM VVR vxio V-5-0-645 Failing reads on rvg aial03_rvg. Rlink  rlk_aial03p_aial03_rvg is inconsistent. :
[Fri May 16 23:47:53 UTC 2025] VxVM VVR vxio V-5-0-645 Failing reads on rvg aial03_rvg. Rlink  rlk_aial03p_aial03_rvg is inconsistent. :
[Fri May 16 23:47:53 UTC 2025] VxVM VVR vxio V-5-0-645 Failing reads on rvg aial03_rvg. Rlink  rlk_aial03p_aial03_rvg is inconsistent. :
[Fri May 16 23:55:05 UTC 2025] VxVM VVR vxio V-5-0-80 Cannot send out a message
[Sat May 17 00:26:03 UTC 2025] VxVM VVR vxio V-5-0-266 Rlink rlk_aial03p_aial03_rvg disconnected from remote :
[Sat May 17 00:26:07 UTC 2025] VxVM VVR vxio V-5-0-1406 Node 10.32.181.177 disconnected from node 10.214.180.120
[Sat May 17 00:26:10 UTC 2025] VxVM VVR vxio V-5-0-1402 Connected from node 10.32.181.177 to node 10.214.180.120
[Sat May 17 00:26:10 UTC 2025] VxVM VVR vxio V-5-0-265 Rlink rlk_aial03p_aial03_rvg connected to remote :
[Sat May 17 00:26:10 UTC 2025] VxVM VVR vxio V-5-0-1449 Disabling checksum for rlink rlk_aial03p_aial03_rvg
[Sat May 17 01:10:46 UTC 2025] BUG: unable to handle kernel paging request at ffff914a58287000
[Sat May 17 01:10:46 UTC 2025] PGD 9f802067 P4D 9f802067 PUD 9f803067 PMD 17631063 PTE 8000000018287061
[Sat May 17 01:10:46 UTC 2025] Oops: 0003 [#1] SMP PTI
[Sat May 17 01:10:46 UTC 2025] CPU: 3 PID: 5706 Comm: vxiod Kdump: loaded Tainted: P           OE     -------- -  - 4.18.0-553.50.1.el8_10.x86_64 #1
[Sat May 17 01:10:46 UTC 2025] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[Sat May 17 01:10:46 UTC 2025] RIP: 0010:vol_ru_verification_data_unpack+0x8e/0xe0 [vxio]
[Sat May 17 01:10:46 UTC 2025] Code: 0c 24 48 83 e0 f8 85 f6 7e 43 83 ee 01 48 8d 9a 90 00 00 00 48 8d ac b2 94 00 00 00 eb 14 8b 10 48 83 c3 04 48 83 c0 04 0f ca <89> 53 fc 48 39 dd 74 1b 48 39 c8 72 e7 48 89 e7 e8 0d dc ea ff 48
[Sat May 17 01:10:46 UTC 2025] RSP: 0018:ffffbbd1839f7d68 EFLAGS: 00010282
[Sat May 17 01:10:46 UTC 2025] RAX: ffff914c4f422fdc RBX: ffff914a58287004 RCX: ffff918c8f62c000
[Sat May 17 01:10:46 UTC 2025] RDX: 00000000d02f424f RSI: ffff914c4f240000 RDI: ffffbbd1839f7d68
[Sat May 17 01:10:46 UTC 2025] RBP: ffff914ad888c110 R08: ffffffffffff8000 R09: 0000000000000000
[Sat May 17 01:10:46 UTC 2025] R10: ffff914bbdd2d400 R11: 00002ef2efe92f80 R12: ffff914bbdd2d400
[Sat May 17 01:10:46 UTC 2025] R13: 0000000046203000 R14: ffff914c2a030000 R15: ffff914a58084000
[Sat May 17 01:10:46 UTC 2025] FS:  0000000000000000(0000) GS:ffff914e6fd80000(0000) knlGS:0000000000000000
[Sat May 17 01:10:46 UTC 2025] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sat May 17 01:10:46 UTC 2025] CR2: ffff914a58287000 CR3: 000000009de10005 CR4: 00000000001706e0
[Sat May 17 01:10:46 UTC 2025] Call Trace:
[Sat May 17 01:10:46 UTC 2025]  ? __die_body+0x1a/0x60
[Sat May 17 01:10:46 UTC 2025]  ? no_context+0x1ba/0x3f0
[Sat May 17 01:10:46 UTC 2025]  ? __bad_area_nosemaphore+0x157/0x180
[Sat May 17 01:10:46 UTC 2025]  ? spurious_kernel_fault+0x1ed/0x250
[Sat May 17 01:10:46 UTC 2025]  ? do_page_fault+0x37/0x12d
[Sat May 17 01:10:46 UTC 2025]  ? page_fault+0x1e/0x30
[Sat May 17 01:10:46 UTC 2025]  ? vol_ru_verification_data_unpack+0x8e/0xe0 [vxio]
[Sat May 17 01:10:46 UTC 2025]  ? vol_ru_verification_data_unpack+0xa3/0xe0 [vxio]
[Sat May 17 01:10:46 UTC 2025]  vol_ru_verify+0x9d/0x110 [vxio]
[Sat May 17 01:10:46 UTC 2025]  volrv_seclog_bulk_cleanup_verification+0x107/0x1a0 [vxio]
[Sat May 17 01:10:46 UTC 2025]  volrv_seclog_write1_done+0xbf/0xd0 [vxio]
[Sat May 17 01:10:46 UTC 2025]  voliod_iohandle+0x21f/0x390 [vxio]
[Sat May 17 01:10:46 UTC 2025]  voliod_loop+0xc2/0x340 [vxio]
[Sat May 17 01:10:46 UTC 2025]  ? voliod_iohandle+0x390/0x390 [vxio]
[Sat May 17 01:10:46 UTC 2025]  kthread+0x134/0x150
[Sat May 17 01:10:46 UTC 2025]  ? set_kthread_struct+0x50/0x50
[Sat May 17 01:10:46 UTC 2025]  ret_from_fork+0x35/0x40
[Sat May 17 01:10:46 UTC 2025] Modules linked in: ib_core bluetooth ecdh_generic rfkill mptcp_diag xsk_diag vsock_diag raw_diag af_packet_diag netlink_diag falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17607(E) xfs nfsv3 nfs_acl nfs lockd grace fscache unix_diag nf_conntrack_netlink udp_diag nft_queue tcp_diag inet_diag nft_counter nft_log nft_ct nf_tables_set xt_LOG nf_log_syslog nf_conntrack_tftp nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 dmpaa(POE) vxspec(POE) vxio(POE) vxdmp(POE) vxodm(POE) amf(POE) vxcafs(POE) vxportal(POE) fdd(POE) vxfs(POE) storageapi(OE) veki(POE) nf_tables libcrc32c nfnetlink_queue nfnetlink_log nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr vmw_balloon intel_rapl_common sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl joydev pcspkr vmw_vmci i2c_piix4 binfmt_misc ext4 mbcache jbd2 sr_mod cdrom ata_generic sd_mod t10_pi sg vmwgfx drm_ttm_helper ttm
[Sat May 17 01:10:46 UTC 2025]  drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt serio_raw drm ata_piix libata vmxnet3 vmw_pvscsi dm_mod fuse
[Sat May 17 01:10:46 UTC 2025] Red Hat flags: eBPF/event eBPF/rawtrace
[Sat May 17 01:10:46 UTC 2025] CR2: ffff914a58287000

Environment

  • Red Hat Enterprise Linux.
  • Third-Party Module [vxio]

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content