The system with Oracle ACFS crashes or has soft lockups in :oracleacfs:OfsWaitIfTransInProgress / .text.lock.spinlock
Issue
The system with Oracle ACFS crashes or has soft lockups contending a spinlock in .text.lock.spinlock
called from OfsWaitIfTransInProgress()
function in [oracleacfs]
kernel module. The system has the following call trace in the kernel log:
BUG: soft lockup - CPU#1 stuck for 60s! [sp_ordr:28174]
CPU 1:
Modules linked in: loop oracleacfs(PU) oracleadvm(PU) oracleoks(PU) mptctl mptbase oracleasm(U) autofs4 lockd sunrpc bonding
rdma_ucm(U) rds_tcp(U) rds_rdma(U) rds(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_srp(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U)
ib_sa(U) ipv6 xfrm_nalgo crypto_api ib_uverbs(U) ib_umad(U) iw_nes(U) ib_qib(U) dca mlx4_ib(U) ib_mthca(U) ib_mad(U) ib_core(U)
emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video backlight sbs power_meter
hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev st mlx4_en(U) sg
shpchp mlx4_core(U) i7core_edac edac_mc be2net(U) pcspkr tpm_tis tpm serio_raw tpm_bios hpilo 8021q dm_raid45 dm_message
dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx scsi_transport_fc cciss sd_mod
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 28174, comm: sp_ordr Tainted: P ---- 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff80064bda>] [<ffffffff80064bda>] .text.lock.spinlock+0x0/0x30
RSP: 0018:ffff81041f8c5cd0 EFLAGS: 00000282
RAX: ffff81023150f7e0 RBX: ffff810bf35d2a40 RCX: 0000000000000000
RDX: ffff810bf35d2a40 RSI: 0000000000000850 RDI: ffff8110bc34e780
RBP: ffff810128928e50 R08: 0000000000000000 R09: 000000000000008d
R10: ffffffff88ce227d R11: ffffffff88ce227e R12: ffff81010c32dc48
R13: ffff81010e761510 R14: ffff81010e3e59c0 R15: ffff810107d25be0
FS: 00002ad90558edc0(0000) GS:ffff810c0fca5ac0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000f7fec000 CR3: 00000003b9321000 CR4: 00000000000006a0
Call Trace:
[<ffffffff88c7bc17>] :oracleacfs:OfsStartTrans+0x144/0x241
[<ffffffff88bec365>] :oracleacfs:OfsDLMWaitForDelayedBAST+0xa1/0xd6
[<ffffffff88bf68f0>] :oracleacfs:OfsFEtoDRSL+0x202/0x77b
[<ffffffff88ca0a7c>] :oracleacfs:ofs_delete_inode+0x248/0x314
[<ffffffff88ca0834>] :oracleacfs:ofs_delete_inode+0x0/0x314
[<ffffffff8002f0d5>] generic_delete_inode+0xc8/0x147
[<ffffffff88c8af8a>] :oracleacfs:ofs_d_iput+0x5a/0x5e
[<ffffffff8000d61b>] dput+0xf6/0x114
[<ffffffff80036a89>] sys_renameat+0x18a/0x1eb
[<ffffffff800ba703>] audit_syscall_entry+0x1a8/0x1d3
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
Kernel panic - not syncing: softlockup: hung tasks
BUG: soft lockup - CPU#15 stuck for 60s! [acfsds3:22831]
CPU 15:
Modules linked in: loop oracleacfs(PU) oracleadvm(PU) oracleoks(PU) mptctl mptbase oracleasm(U) autofs4 lockd sunrpc bonding
rdma_ucm(U) rds_tcp(U) rds_rdma(U) rds(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_srp(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U)
ib_sa(U) ipv6 xfrm_nalgo crypto_api ib_uverbs(U) ib_umad(U) iw_nes(U) ib_qib(U) dca mlx4_ib(U) ib_mthca(U) ib_mad(U) ib_core(U)
emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video backlight sbs power_meter
hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev st mlx4_en(U) sg
shpchp mlx4_core(U) i7core_edac edac_mc be2net(U) pcspkr tpm_tis tpm serio_raw tpm_bios hpilo 8021q dm_raid45 dm_message
dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx scsi_transport_fc cciss sd_mod
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 22831, comm: acfsds3 Tainted: P ---- 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff80064bdc>] [<ffffffff80064bdc>] .text.lock.spinlock+0x2/0x30
RSP: 0018:ffff8110bff63d28 EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8110bc34e000 RCX: ffff8110c1c7c570
RDX: ffff8110bff63cd8 RSI: 0000000000000202 RDI: ffff8110bc34e780
RBP: ffff8110bff63d30 R08: ffff811179c37820 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8110bc34ecd0
R13: ffff8110bc34e7e8 R14: ffff8110bc34e000 R15: ffff8110bff63cc0
FS: 00002b1fae4ef780(0000) GS:ffff810c0f826d40(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b7c885bd000 CR3: 00000011b1b6d000 CR4: 00000000000006a0
Call Trace:
[<ffffffff88be21e2>] :oracleacfs:OfsWaitIfTransInProgress+0x4d/0x92
[<ffffffff88be49f8>] :oracleacfs:OfsCommitCache+0x27e/0x906
[<ffffffff88bc8d7c>] :oracleacfs:OfsDRSLScan+0xdf2/0x15df
[<ffffffff88bcff7b>] :oracleacfs:OfsKernelThread+0x0/0x534
[<ffffffff88bd0404>] :oracleacfs:OfsKernelThread+0x489/0x534
[<ffffffff8002df0f>] __wake_up+0x38/0x4f
[<ffffffff88bcff7b>] :oracleacfs:OfsKernelThread+0x0/0x534
[<ffffffff88b31b34>] :oracleoks:KsKthreadRun+0x6c/0x85
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff88b31ac8>] :oracleoks:KsKthreadRun+0x0/0x85
[<ffffffff8005dfa7>] child_rip+0x0/0x11
Kernel panic - not syncing: softlockup: hung tasks
Environment
-
Red Hat Enterprise Linux 5
- kernel 2.6.18-308.el5
-
Oracle Automatic Storage Management Cluster File System (Oracle ACFS)
- Oracle Grid Infrastructure 11.2.0.3.0
- Grid Infrastructure Patch Set Update 11.2.0.3.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.