gfs2: kernel panic after WARNING at kernel/workqueue.c:1632 __queue_delayed_work
Issue
- gfs2: kernel panic after WARNING at kernel/workqueue.c:1632 __queue_delayed_work
[ 114.243370] dlm: data01: dlm_recover 1 generation 3 done: 109 ms
[ 114.294117] FINAL_REJECT: IN=ens192 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:89:dd:96:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[ 114.325594] dlm: data02: dlm_recover 3
[ 114.325628] dlm: data02: add member 2
[ 114.326315] dlm: data02: dlm_recover_members 3 nodes
[ 114.390328] dlm: data02: generation 3 slots 3 1:1 2:3 3:2
[ 114.390330] dlm: data02: dlm_recover_directory
[ 114.391232] dlm: data02: dlm_recover_directory 0 in 0 new
[ 114.413319] dlm: data02: dlm_recover_directory 0 out 2 messages
[ 114.415292] dlm: lvm_vg_workspace: dlm_recover 5 generation 3 done: 213 ms
[ 114.541326] dlm: data02: dlm_recover 3 generation 3 done: 151 ms
[ 114.787513] FINAL_REJECT: IN=ens192 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:89:dd:96:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[ 115.363694] FINAL_REJECT: IN=ens192 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:89:e9:90:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[ 116.487940] FINAL_REJECT: IN=ens192 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:89:e9:90:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[ 117.185215] FINAL_REJECT: IN=ens192 OUT= MAC=ff:ff:ff:ff:ff:ff:00:50:56:89:dd:96:08:00 SRC=0.0.0.0 DST=255.255.255.255 LEN=328 TOS=0x10 PREC=0x00 TTL=128 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=308
[ 117.695752] gfs2: fsid=cluster8:data03: Trying to join cluster "lock_dlm", "cluster8:data03"
[ 117.696122] dlm: data03: joining the lockspace group...
[ 117.698996] gfs2: fsid=cluster8:eps04: Trying to join cluster "lock_dlm", "cluster8:eps04"
[ 117.702218] dlm: data03: dlm_recover 1
[ 117.702236] dlm: data03: group event done 0 0
[ 117.702274] dlm: data03: add member 3
[ 117.702280] dlm: data03: dlm_recover_members 1 nodes
[ 117.702282] dlm: data03: generation 1 slots 1 1:3
z[ 117.702283] dlm: data03: dlm_recover_directory
[ 117.702284] dlm: data03: dlm_recover_directory 0 in 0 new
[ 117.702284] dlm: data03: dlm_recover_directory 0 out 0 messages
[ 117.702297] dlm: data03: dlm_recover 1 generation 1 done: 0 ms
[ 117.702305] WARNING: CPU: 18 PID: 9890 at kernel/workqueue.c:1632 __queue_delayed_work+0x6c/0x80
[ 117.702306] dlm: data03: join complete
[ 117.702306] Modules linked in: gfs2(+) dlm nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_log_syslog nft_log nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic vmwgfx sd_mod t10_pi drm_kms_helper sg syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel ahci drm libahci ata_piix serio_raw libata vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse
[ 117.702331] CPU: 18 PID: 9890 Comm: dlm_recoverd Kdump: loaded Not tainted 4.18.0-348.el8.x86_64 #1
[ 117.702332] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 117.702334] RIP: 0010:__queue_delayed_work+0x6c/0x80
[ 117.702335] Code: 00 00 44 89 82 90 00 00 00 48 01 c8 48 89 42 50 41 81 f8 00 20 00 00 75 1d e9 60 24 07 00 44 89 c7 e9 98 f6 ff ff 0f 0b eb c5 <0f> 0b eb a1 0f 0b eb a7 0f 0b eb ac 44 89 c6 e9 40 23 07 00 0f 1f
[ 117.702336] RSP: 0018:ffff9db790d7be10 EFLAGS: 00010046
[ 117.702337] RAX: 0000000000000246 RBX: 0000000000000246 RCX: 0000000000000000
[ 117.702338] RDX: ffff8bc7413244b0 RSI: 0000000000000000 RDI: ffff8bc7413244f0
[ 117.702339] RBP: ffff8bc741324380 R08: 0000000000002000 R09: 0000000000000000
[ 117.702339] R10: ffff8bc7270cdda8 R11: 0000000000000001 R12: 0000000000000001
[ 117.702340] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8bc70e2df7d8
[ 117.702341] FS: 0000000000000000(0000) GS:ffff8bddde280000(0000) knlGS:0000000000000000
[ 117.702341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 117.702342] CR2: 00007f1491713ff0 CR3: 0000000987e10003 CR4: 00000000003706e0
[ 117.702366] Call Trace:
[ 117.702369] queue_delayed_work_on+0x36/0x40
[ 117.702390] gdlm_recover_done+0xcf/0xe0 [gfs2]
[ 117.702400] dlm_lsop_recover_done+0xce/0xe0 [dlm]
[ 117.702405] ls_recover+0x4be/0x540 [dlm]
[ 117.702410] dlm_recoverd+0x193/0x1c0 [dlm]
[ 117.702414] ? ls_recover+0x540/0x540 [dlm]
[ 117.702416] kthread+0x116/0x130
[ 117.702417] ? kthread_flush_work_fn+0x10/0x10
[ 117.702419] ret_from_fork+0x35/0x40
[ 117.702421] ---[ end trace 5bbf9d15d7ba4db7 ]---
[ 117.702426] BUG: unable to handle kernel NULL pointer dereference at 0000000000000102
[ 117.702457] PGD 0 P4D 0
[ 117.702468] Oops: 0000 [#1] SMP PTI
[ 117.702480] CPU: 18 PID: 9890 Comm: dlm_recoverd Kdump: loaded Tainted: G W --------- - - 4.18.0-348.el8.x86_64 #1
[ 117.702512] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[ 117.702542] RIP: 0010:__queue_work+0x1f/0x3e0
[ 117.702556] Code: 0f 94 c3 e9 4a ff ff ff 0f 1f 00 0f 1f 44 00 00 41 57 49 89 d7 41 56 49 89 f6 41 55 41 89 fd 41 54 41 89 fc 55 53 48 83 ec 10 <f6> 86 02 01 00 00 01 0f 85 e5 02 00 00 48 bd eb 83 b5 80 46 86 c8
[ 117.702589] dlm: eps04: joining the lockspace group...
[ 117.702605] RSP: 0018:ffff9db790d7bdd0 EFLAGS: 00010082
[ 117.702607] RAX: ffff8bc7413244b8 RBX: 0000000000000246 RCX: 0000000000000000
[ 117.702662] RDX: ffff8bc7413244b0 RSI: 0000000000000000 RDI: 0000000000002000
[ 117.702682] RBP: ffff8bc741324380 R08: 0000000000002000 R09: ffff8bc7413244b8
[ 117.702703] R10: ffff8bc7270cdda8 R11: 0000000000000001 R12: 0000000000002000
[ 117.702725] R13: 0000000000002000 R14: 0000000000000000 R15: ffff8bc7413244b0
[ 117.702746] FS: 0000000000000000(0000) GS:ffff8bddde280000(0000) knlGS:0000000000000000
[ 117.702769] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 117.702786] CR2: 0000000000000102 CR3: 0000000987e10003 CR4: 00000000003706e0
[ 117.702843] Call Trace:
[ 117.702857] queue_delayed_work_on+0x36/0x40
[ 117.702884] gdlm_recover_done+0xcf/0xe0 [gfs2]
[ 117.702904] dlm_lsop_recover_done+0xce/0xe0 [dlm]
[ 117.702922] ls_recover+0x4be/0x540 [dlm]
[ 117.702940] dlm_recoverd+0x193/0x1c0 [dlm]
[ 117.702956] ? ls_recover+0x540/0x540 [dlm]
[ 117.702971] kthread+0x116/0x130
[ 117.702983] ? kthread_flush_work_fn+0x10/0x10
[ 117.703000] ret_from_fork+0x35/0x40
[ 117.703015] Modules linked in: gfs2(+) dlm nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nf_log_syslog nft_log nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink vsock_loopback vmw_vsock_virtio_transport_common vmw_vsock_vmci_transport vsock sunrpc intel_rapl_msr intel_rapl_common sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel rapl vmw_balloon joydev pcspkr vmw_vmci i2c_piix4 binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic vmwgfx sd_mod t10_pi drm_kms_helper sg syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel ahci drm libahci ata_piix serio_raw libata vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod fuse
[ 117.703214] CR2: 0000000000000102
Environment
- Red Hat Enterprise Linux Server 8, 9 (with the High Availability Add On and Resilient Storage Add Ons)
- A Global Filesystem 2(
gfs2
)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.