The system with Oracle ACFS crashes or has soft lockups in :oracleacfs:OfsWaitIfTransInProgress / .text.lock.spinlock

Solution Unverified - Updated -

Issue

The system with Oracle ACFS crashes or has soft lockups contending a spinlock in .text.lock.spinlock called from OfsWaitIfTransInProgress() function in [oracleacfs] kernel module. The system has the following call trace in the kernel log:

BUG: soft lockup - CPU#1 stuck for 60s! [sp_ordr:28174]
CPU 1:
Modules linked in: loop oracleacfs(PU) oracleadvm(PU) oracleoks(PU) mptctl mptbase oracleasm(U) autofs4 lockd sunrpc bonding
rdma_ucm(U) rds_tcp(U) rds_rdma(U) rds(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_srp(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U)
ib_sa(U) ipv6 xfrm_nalgo crypto_api ib_uverbs(U) ib_umad(U) iw_nes(U) ib_qib(U) dca mlx4_ib(U) ib_mthca(U) ib_mad(U) ib_core(U)
emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video backlight sbs power_meter
hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev st mlx4_en(U) sg
shpchp mlx4_core(U) i7core_edac edac_mc be2net(U) pcspkr tpm_tis tpm serio_raw tpm_bios hpilo 8021q dm_raid45 dm_message
dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx scsi_transport_fc cciss sd_mod
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 28174, comm: sp_ordr Tainted: P     ---- 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff80064bda>]  [<ffffffff80064bda>] .text.lock.spinlock+0x0/0x30
RSP: 0018:ffff81041f8c5cd0  EFLAGS: 00000282
RAX: ffff81023150f7e0 RBX: ffff810bf35d2a40 RCX: 0000000000000000
RDX: ffff810bf35d2a40 RSI: 0000000000000850 RDI: ffff8110bc34e780
RBP: ffff810128928e50 R08: 0000000000000000 R09: 000000000000008d
R10: ffffffff88ce227d R11: ffffffff88ce227e R12: ffff81010c32dc48
R13: ffff81010e761510 R14: ffff81010e3e59c0 R15: ffff810107d25be0
FS:  00002ad90558edc0(0000) GS:ffff810c0fca5ac0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000f7fec000 CR3: 00000003b9321000 CR4: 00000000000006a0
Call Trace:
 [<ffffffff88c7bc17>] :oracleacfs:OfsStartTrans+0x144/0x241
 [<ffffffff88bec365>] :oracleacfs:OfsDLMWaitForDelayedBAST+0xa1/0xd6
 [<ffffffff88bf68f0>] :oracleacfs:OfsFEtoDRSL+0x202/0x77b
 [<ffffffff88ca0a7c>] :oracleacfs:ofs_delete_inode+0x248/0x314
 [<ffffffff88ca0834>] :oracleacfs:ofs_delete_inode+0x0/0x314
 [<ffffffff8002f0d5>] generic_delete_inode+0xc8/0x147
 [<ffffffff88c8af8a>] :oracleacfs:ofs_d_iput+0x5a/0x5e
 [<ffffffff8000d61b>] dput+0xf6/0x114
 [<ffffffff80036a89>] sys_renameat+0x18a/0x1eb
 [<ffffffff800ba703>] audit_syscall_entry+0x1a8/0x1d3
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Kernel panic - not syncing: softlockup: hung tasks

BUG: soft lockup - CPU#15 stuck for 60s! [acfsds3:22831]
CPU 15:
Modules linked in: loop oracleacfs(PU) oracleadvm(PU) oracleoks(PU) mptctl mptbase oracleasm(U) autofs4 lockd sunrpc bonding
rdma_ucm(U) rds_tcp(U) rds_rdma(U) rds(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_srp(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U)
ib_sa(U) ipv6 xfrm_nalgo crypto_api ib_uverbs(U) ib_umad(U) iw_nes(U) ib_qib(U) dca mlx4_ib(U) ib_mthca(U) ib_mad(U) ib_core(U)
emcpvlumd(PU) emcpxcrypt(PU) emcpdm(PU) emcpgpx(PU) emcpmpx(PU) emcp(PU) dm_multipath scsi_dh video backlight sbs power_meter
hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev st mlx4_en(U) sg
shpchp mlx4_core(U) i7core_edac edac_mc be2net(U) pcspkr tpm_tis tpm serio_raw tpm_bios hpilo 8021q dm_raid45 dm_message
dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx scsi_transport_fc cciss sd_mod
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 22831, comm: acfsds3 Tainted: P     ---- 2.6.18-308.el5 #1
RIP: 0010:[<ffffffff80064bdc>]  [<ffffffff80064bdc>] .text.lock.spinlock+0x2/0x30
RSP: 0018:ffff8110bff63d28  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8110bc34e000 RCX: ffff8110c1c7c570
RDX: ffff8110bff63cd8 RSI: 0000000000000202 RDI: ffff8110bc34e780
RBP: ffff8110bff63d30 R08: ffff811179c37820 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8110bc34ecd0
R13: ffff8110bc34e7e8 R14: ffff8110bc34e000 R15: ffff8110bff63cc0
FS:  00002b1fae4ef780(0000) GS:ffff810c0f826d40(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b7c885bd000 CR3: 00000011b1b6d000 CR4: 00000000000006a0
Call Trace:
 [<ffffffff88be21e2>] :oracleacfs:OfsWaitIfTransInProgress+0x4d/0x92
 [<ffffffff88be49f8>] :oracleacfs:OfsCommitCache+0x27e/0x906
 [<ffffffff88bc8d7c>] :oracleacfs:OfsDRSLScan+0xdf2/0x15df
 [<ffffffff88bcff7b>] :oracleacfs:OfsKernelThread+0x0/0x534
 [<ffffffff88bd0404>] :oracleacfs:OfsKernelThread+0x489/0x534
 [<ffffffff8002df0f>] __wake_up+0x38/0x4f
 [<ffffffff88bcff7b>] :oracleacfs:OfsKernelThread+0x0/0x534
 [<ffffffff88b31b34>] :oracleoks:KsKthreadRun+0x6c/0x85
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff88b31ac8>] :oracleoks:KsKthreadRun+0x0/0x85
 [<ffffffff8005dfa7>] child_rip+0x0/0x11
Kernel panic - not syncing: softlockup: hung tasks

Environment

  • Red Hat Enterprise Linux 5

    • kernel 2.6.18-308.el5
  • Oracle Automatic Storage Management Cluster File System (Oracle ACFS)

    • Oracle Grid Infrastructure 11.2.0.3.0
    • Grid Infrastructure Patch Set Update 11.2.0.3.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content