System crashed with error fs/fs-writeback.c:1620 __writeback_single_inode+0x23c/0x300

Solution Unverified - Updated -

Issue

  • One of the nodes in Veritas cluster got crashed with following errors and call traces:

    VxVM vxdmp V-5-3-0 memory allocation failed for size =0x303000
    VxVM vxdmp V-5-0-2031 Cannot allocate mem of size 4194304 bytes
    VxVM vxdmp V-5-3-0 memory allocation failed for size =0x303000
    VxVM vxdmp V-5-0-2031 Cannot allocate mem of size 4194304 bytes
    VxVM vxdmp V-5-3-0 memory allocation failed for size =0x303000
    VxVM vxdmp V-5-0-2031 Cannot allocate mem of size 4194304 bytes
    VxVM vxdmp V-5-3-0 memory allocation failed for size =0x303000
    vxfs: msgcnt 965 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/testdg/testvol file system full (256 block extent)
    vxfs: msgcnt 966 mesg 001: V-2-1: vx_nospace - /dev/vx/dsk/testdg/testvol file system full (256 block extent)
    WARNING: CPU: 67 PID: 374266 at fs/fs-writeback.c:1620 __writeback_single_inode+0x23c/0x300
    Modules linked in: xsk_diag unix_diag af_packet_diag netlink_diag mptcp_diag raw_diag vxfen(POE) vxodm(POE) vxgms(POE) vxglm(POE) gab(POE) [...]
    [...]
    Red Hat flags: eBPF/event eBPF/rawtrace
    CPU: 67 PID: 374266 Comm: kworker/u324:0 Kdump: loaded Tainted: P           OE     -
    Hardware name: HPE ProLiant DL580 Gen10/ProLiant DL580 Gen10, BIOS U34 07/31/2024
    Workqueue: writeback wb_workfn (flush-199:14006)
    RIP: 0010:__writeback_single_inode+0x23c/0x300
    Code: bd 01 4d 85 e4 74 1e 49 8b 04 24 49 8b 7c 24 08 49 83 c4 18 48 89 de e8 72 0e [...]
    RSP: 0018:ffffb441fc257be8 EFLAGS: 00010246
    RAX: 0000000000000015 RBX: ffff979e2e4e64a8 RCX: ffff978da31ab548
    RDX: 0000000000000400 RSI: ffffb441fc257c50 RDI: ffff979e2e4e64a8
    RBP: ffffb441fc257c50 R08: ffff978354539598 R09: ffffb441fc257b18
    R10: ffff97c161477a04 R11: ffff97c8ef3f1dc4 R12: ffff973a58628000
    R13: 0000000000000400 R14: ffff979e2e4e6598 R15: ffff979e2e4e6620
    FS:  0000000000000000(0000) GS:ffff97c8ef3c0000(0000) knlGS:0000000000000000
    CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    CR2: 00007fda4b4bc220 CR3: 000000545c810001 CR4: 00000000007706e0
    PKRU: 55555554
    Call Trace:
     ? __warn+0x94/0xe0
     ? __writeback_single_inode+0x23c/0x300
     ? __writeback_single_inode+0x23c/0x300
     ? report_bug+0xb1/0xe0
     ? do_error_trap+0x9e/0xd0
     ? do_invalid_op+0x36/0x40
     ? __writeback_single_inode+0x23c/0x300
     ? invalid_op+0x14/0x20
     ? __writeback_single_inode+0x23c/0x300
     writeback_sb_inodes+0x1ea/0x450
     __writeback_inodes_wb+0x5f/0xd0
     wb_writeback+0x24c/0x2e0
     wb_workfn+0x35c/0x4e0
     process_one_work+0x1d3/0x390
     worker_thread+0x30/0x390
     ? process_one_work+0x390/0x390
     kthread+0x134/0x150
     ? set_kthread_struct+0x50/0x50
     ret_from_fork+0x1f/0x40
    ---[ end trace 8c6d29fa0e299903 ]---
    BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
    

Environment

  • Red Hat Enterprise Linux 8.10
  • Veritas Filesystem

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content