Oracle RAC node got rebooted with ocfs2 filesystem error.

Issue

• Oracle RAC node rebooted with following log messages.

kernel: (dlm_thread,13530,11):dlm_drop_lockres_ref:2216 ERROR: while dropping ref on FF8014A0C9FF473F894861CD0F355995:N00000000020bbca8 (master=2) got -22.
kernel: (dlm_thread,13530,11):dlm_drop_lockres_ref:2216 ERROR: while dropping ref on FF8014A0C9FF473F894861CD0F355995:N00000000020bbca8 (master=2) got -22.
kernel: lockres: N00000000020bbca801bdc3f6, owner=2, state=64  kernel: lockres: N00000000020bbca801bdc3f6, owner=2, state=64
kernel:   last used: 4842290988, refcnt: 3, on purge list: yes
kernel:   last used: 4842290988, refcnt: 3, on purge list: yes
kernel:   on dirty list: no, on reco list: no, migrating pending: no
kernel:   on dirty list: no, on reco list: no, migrating pending: no
kernel:   inflight locks: 0, asts reserved: 0
kernel:   inflight locks: 0, asts reserved: 0
kernel:   refmap nodes: [ ], inflight=0
kernel:   refmap nodes: [ ], inflight=0
kernel:   granted queue:
kernel:   granted queue:
kernel:   converting queue:
kernel:   converting queue:
kernel:   blocked queue:
kernel:   blocked queue:
kernel: ----------- [cut here ] --------- [please bite here ] ---------  
kernel: ----------- [cut here ] --------- [please bite here ] ---------  
kernel: Kernel BUG at ...xiaowei/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmmaster.c:2218
kernel: Kernel BUG at ...xiaowei/BUILD/ocfs2-1.4.7/fs/ocfs2/dlm/dlmmaster.c:2218
kernel: invalid opcode: 0000 [1] SMP  kernel: invalid opcode: 0000 [1] SMP  kernel: last sysfs file: /block/sda/sda1/stat  kernel: last sysfs file: /block/sda/sda1/stat  kernel: CPU 11 

• Kernel panic with following call traces.

Kernel BUG at ...mushran/BUILD/ocfs2-1.4.2/fs/ocfs2/dlm/dlmmaster.c:2218
invalid opcode: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 17 
Modules linked in: hangcheck_timer ipmi_si(U) ipmi_devintf(U) ipmi_msghandler(U) oracleasm(U) autofs4 ocfs2(U) nfs lockd fscache nfs_acl ocfs2_dlmfs(U) ocfs2_dlm(U) ocfs2_nodemanager(U) configfs symap_rh_ES_5_2.6.18_128.el5_x86_64(PU) symev_rh_ES_5_2.6.18_128.el5_x86_64(U) ipv6 xfrm_nalgo crypto_api sunrpc bonding dm_round_robin dm_multipath scsi_dh video hwmon backlight sbs i2c_ec i2c_core button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sg e1000e(U) hpilo pcspkr shpchp serio_raw bnx2(U) dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc scsi_transport_fc ata_piix libata cciss(U) sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 18135, comm: dlm_thread Tainted: P      2.6.18-128.el5 #1
RIP: 0010:[<ffffffff8851f8c5>]  [<ffffffff8851f8c5>] :ocfs2_dlm:dlm_drop_lockres_ref+0x1db/0x1f4
RSP: 0018:ffff8140195f1da0  EFLAGS: 00010282
RAX: ffff810b3c2ef748 RBX: 0000000000000000 RCX: 0000000000000282
RDX: 00000000ffffffff RSI: 0000000000000000 RDI: ffffffff802f7adc
RBP: 000000000000001f R08: 0000000000000000 R09: 0000000000000020
R10: ffff813fbf431880 R11: 0000000000000000 R12: ffff810b3c2ef700
R13: ffff814020c48400 R14: ffff810a8921cba0 R15: 0000000000003fa0
FS:  0000000000000000(0000) GS:ffff81402fd4fac0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000c88e8004 CR3: 0000003f9c0c1000 CR4: 00000000000006e0
Process dlm_thread (pid: 18135, threadinfo ffff8140195f0000, task ffff81401af8d040)
Stack:  1f02000000000000 303030303030304d 3030303030303030 3032633936663235
 0030303030303030 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 ffffffea00100100 ffff814020c48428 ffff810b3c2ef700
Call Trace:
 [<ffffffff8852db65>] :ocfs2_dlm:dlm_run_purge_list+0x28a/0x487
 [<ffffffff8852df14>] :ocfs2_dlm:dlm_thread+0x83/0xdcc
 [<ffffffff8009db21>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8009d909>] keventd_create_kthread+0x0/0xc4
 [<ffffffff8852de91>] :ocfs2_dlm:dlm_thread+0x0/0xdcc
 [<ffffffff8009d909>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032360>] kthread+0xfe/0x132
 [<ffffffff8005dfb1>] child_rip+0xa/0x11
 [<ffffffff8009d909>] keventd_create_kthread+0x0/0xc4
 [<ffffffff80032262>] kthread+0x0/0x132
 [<ffffffff8005dfa7>] child_rip+0x0/0x11

Code: 0f 0b 68 d4 49 53 88 c2 aa 08 48 83 c4 50 89 d8 5b 5d 41 5c 
RIP  [<ffffffff8851f8c5>] :ocfs2_dlm:dlm_drop_lockres_ref+0x1db/0x1f4
 RSP <ffff8140195f1da0>