Oracle RAC node got rebooted with ocfs2 filesystem error.
Issue
- Oracle RAC node rebooted with following log messages.
kernel: (dlm_thread,13530,11):dlm_drop_lockres_ref:2216 ERROR: while dropping ref on FF8014A0C9FF473F894861CD0F355995:N00000000020bbca8 (master=2) got -22.
kernel: (dlm_thread,13530,11):dlm_drop_lockres_ref:2216 ERROR: while dropping ref on FF8014A0C9FF473F894861CD0F355995:N00000000020bbca8 (master=2) got -22.
kernel: lockres: N00000000020bbca801bdc3f6, owner=2, state=64 kernel: lockres: N00000000020bbca801bdc3f6, owner=2, state=64
kernel: last used: 4842290988, refcnt: 3, on purge list: yes
kernel: last used: 4842290988, refcnt: 3, on purge list: yes
kernel: on dirty list: no, on reco list: no, migrating pending: no
kernel: on dirty list: no, on reco list: no, migrating pending: no
kernel: inflight locks: 0, asts reserved: 0
kernel: inflight locks: 0, asts reserved: 0
kernel: refmap nodes: [ ], inflight=0
kernel: refmap nodes: [ ], inflight=0
kernel: granted queue:
kernel: granted queue:
kernel: converting queue:
kernel: converting queue:
kernel: blocked queue:
kernel: blocked queue:
kernel: ----------- [cut here ] --------- [please bite here ] ---------
kernel: ----------- [cut here ] --------- [please bite here ] ---------
kernel: Kernel BUG at ...xiaowei/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmmaster.c:2218
kernel: Kernel BUG at ...xiaowei/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmmaster.c:2218
kernel: invalid opcode: 0000 [1] SMP kernel: invalid opcode: 0000 [1] SMP kernel: last sysfs file: /block/sda/sda1/stat kernel: last sysfs file: /block/sda/sda1/stat kernel: CPU 11
- Kernel panic with following call traces.
Kernel BUG at ...mushran/BUILD/ocfs2-1.4.2/fs/ocfs2/dlm/dlmmaster.c:2218
invalid opcode: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 17
Modules linked in: hangcheck_timer ipmi_si(U) ipmi_devintf(U) ipmi_msghandler(U) oracleasm(U) autofs4 ocfs2(U) nfs lockd fscache nfs_acl ocfs2_dlmfs(U) ocfs2_dlm(U) ocfs2_nodemanager(U) configfs symap_rh_ES_5_2.6.18_128.el5_x86_64(PU) symev_rh_ES_5_2.6.18_128.el5_x86_64(U) ipv6 xfrm_nalgo crypto_api sunrpc bonding dm_round_robin dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sg e1000e(U) hpilo pcspkr shpchp serio_raw bnx2(U) dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc scsi_transport_fc ata_piix libata cciss(U) sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 18135, comm: dlm_thread Tainted: P 2.6.18-128.el5 #1
RIP: 0010:[<ffffffff8851f8c5>] [<ffffffff8851f8c5>] :ocfs2_dlm:dlm_drop_lockres_ref+0x1db/0x1f4
RSP: 0018:ffff8140195f1da0 EFLAGS: 00010282
RAX: ffff810b3c2ef748 RBX: 0000000000000000 RCX: 0000000000000282
RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffffffff802f7adc
RBP: 000000000000001f R08: 0000000000000000 R09: 0000000000000020
R10: ffff813fbf431880 R11: 0000000000000000 R12: ffff810b3c2ef700
R13: ffff814020c48400 R14: ffff810a8921cba0 R15: 0000000000003fa0
FS: 0000000000000000(0000) GS:ffff81402fd4fac0(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000c88e8004 CR3: 0000003f9c0c1000 CR4: 00000000000006e0
Process dlm_thread (pid: 18135, threadinfo ffff8140195f0000, task ffff81401af8d040)
Stack: 1f02000000000000 303030303030304d 3030303030303030 3032633936663235
0030303030303030 0000000000000000 0000000000000000 0000000000000000
0000000000000000 ffffffea00100100 ffff814020c48428 ffff810b3c2ef700
Call Trace:
[<ffffffff8852db65>] :ocfs2_dlm:dlm_run_purge_list+0x28a/0x487
[<ffffffff8852df14>] :ocfs2_dlm:dlm_thread+0x83/0xdcc
[<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
[<ffffffff8009d909>] keventd_create_kthread+0x0/0xc4
[<ffffffff8852de91>] :ocfs2_dlm:dlm_thread+0x0/0xdcc
[<ffffffff8009d909>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032360>] kthread+0xfe/0x132
[<ffffffff8005dfb1>] child_rip+0xa/0x11
[<ffffffff8009d909>] keventd_create_kthread+0x0/0xc4
[<ffffffff80032262>] kthread+0x0/0x132
[<ffffffff8005dfa7>] child_rip+0x0/0x11
Code: 0f 0b 68 d4 49 53 88 c2 aa 08 48 83 c4 50 89 d8 5b 5d 41 5c
RIP [<ffffffff8851f8c5>] :ocfs2_dlm:dlm_drop_lockres_ref+0x1db/0x1f4
RSP <ffff8140195f1da0>
Environment
- Red Hat Enterprise Linux 5
- Oracle Clusterware
- ocfs2 filesystem
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
