kernel reports soft lockup messages in the `lin_tape` module on a RHEL 5 host

Issue

• System hangs with soft lockup messages in lin_taped
• A cluster node was fenced immediately after reporting a soft-lockup in IBM's lin_taped
• Why is my system showing soft lockups in the lin_taped module?

Aug 10 00:19:53 node1 kernel: BUG: soft lockup - CPU#9 stuck for 60s! [lin_taped:18931]
Aug 10 00:19:53 node1 kernel: CPU 9:
Aug 10 00:19:53 node1 kernel: Modules linked in: nfs nfs_acl hidp rfcomm l2cap bluetooth lock_dlm gfs2(U) dlm configfs lin_tape(U) lockd sunrpc bonding be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod st sg i7300_edac e1000e edac_mc pcspkr hpilo bnx2 tpm_tis tpm ide_cd tpm_bios serio_raw cdrom dm_raid45 dm_message dm_region_hash dm_mem_cache dm_round_robin dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_dh dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata cciss shpchp qla2xxx scsi_transport_fc sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 10 00:19:53 node1 kernel: Pid: 18931, comm: lin_taped Tainted: G     ---- 2.6.18-308.1.1.el5 #1
Aug 10 00:19:53 node1 kernel: RIP: 0010:[<ffffffff800621ad>]  [<ffffffff800621ad>] __read_lock_failed+0x5/0x14
Aug 10 00:19:53 node1 kernel: RSP: 0018:ffff81200c02fde8  EFLAGS: 00000297
Aug 10 00:19:53 node1 kernel: RAX: ffff81202f14a000 RBX: ffff81017c56e800 RCX: ffff81200c02fe5c
Aug 10 00:19:53 node1 kernel: RDX: 0000000000000296 RSI: ffff81017c56e800 RDI: ffff811fc010cbd8
Aug 10 00:19:53 node1 kernel: RBP: ffff811fdb6d9228 R08: 0000000000000000 R09: 0000000000000000
Aug 10 00:19:53 node1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000002de3c
Aug 10 00:19:53 node1 kernel: R13: 0004694b4e1de53c R14: ffff810170b42100 R15: ffff811fdb6d9040
Aug 10 00:19:53 node1 kernel: FS:  00002aaaaaac06e0(0000) GS:ffff810170b311c0(0000) knlGS:0000000000000000
Aug 10 00:19:53 node1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 10 00:19:53 node1 kernel: CR2: 00002aaabf00000a CR3: 0000001ff2114000 CR4: 00000000000006e0
Aug 10 00:19:53 node1 kernel: 
Aug 10 00:19:53 node1 kernel: Call Trace:
Aug 10 00:19:53 node1 kernel:  [<ffffffff800649f1>] __down+0x99/0xd8
Aug 10 00:19:53 node1 kernel:  [<ffffffff80064b55>] _read_lock+0xb/0xc
Aug 10 00:19:53 node1 kernel:  [<ffffffff88829108>] :lin_tape:lin_tape_poll_trace_drive+0x6f/0x2f1
Aug 10 00:19:53 node1 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Aug 10 00:19:53 node1 kernel:  [<ffffffff8882bbb7>] :lin_tape:lin_tape_poll_trace+0x17e/0x1f0
Aug 10 00:19:53 node1 kernel:  [<ffffffff88824741>] :lin_tape:lin_tape_ioctl+0x4e/0xba
Aug 10 00:19:53 node1 kernel:  [<ffffffff80041ea7>] do_ioctl+0x55/0x6b
Aug 10 00:19:53 node1 kernel:  [<ffffffff8002ff00>] vfs_ioctl+0x457/0x4b9
Aug 10 00:19:53 node1 kernel:  [<ffffffff800ba767>] audit_syscall_entry+0x1a8/0x1d3
Aug 10 00:19:53 node1 kernel:  [<ffffffff8004c23f>] sys_ioctl+0x59/0x78
Aug 10 00:19:53 node1 kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Aug 10 00:19:53 node1 kernel: 

May 18 10:41:40 Host1 kernel: BUG: soft lockup - CPU#8 stuck for 60s! [dsmserv:28732]
May 18 10:41:40 Host1 kernel: CPU 8:
May 18 10:41:40 Host1 kernel: Modules linked in: lin_tape(U) bonding ip_conntrack_netbios_ns ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi ext4 jbd2 crc16 dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport shpchp joydev st sg ide_cd tpm_tis tpm bnx2 cdrom tpm_bios e1000e pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc scsi_transport_fc ata_piix libata megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
May 18 10:41:40 Host1 kernel: Pid: 28732, comm: dsmserv Tainted: G     ---- 2.6.18-308.16.1.el5 #1
May 18 10:41:40 Host1 kernel: RIP: 0010:[<ffffffff80062191>]  [<ffffffff80062191>] __write_lock_failed+0x9/0x20
May 18 10:41:40 Host1 kernel: RSP: 0018:ffff810e6d34bc98  EFLAGS: 00000287
May 18 10:41:40 Host1 kernel: RAX: 000000000000000a RBX: 00000000fffffff4 RCX: 0000000000000016
May 18 10:41:40 Host1 kernel: RDX: 000000000000000a RSI: ffff810e6d34bcf0 RDI: ffff811c764a8ba8
May 18 10:41:40 Host1 kernel: RBP: ffff811c75ccade0 R08: 0000000000000001 R09: 0000000000000000
May 18 10:41:40 Host1 kernel: R10: 0000000000000000 R11: ffff811c7c7b8000 R12: ffff811c75cca8c0
May 18 10:41:40 Host1 kernel: R13: 0000000000000004 R14: ffff811c764a8800 R15: 00000000fffffff4
May 18 10:41:40 Host1 kernel: FS:  00002b7b30a4c940(0000) GS:ffff810163ef99c0(0000) knlGS:0000000000000000
May 18 10:41:40 Host1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 18 10:41:40 Host1 kernel: CR2: 00000000006bb544 CR3: 0000001c58eba000 CR4: 00000000000006a0
May 18 10:41:40 Host1 kernel:
May 18 10:41:40 Host1 kernel: Call Trace:
May 18 10:41:40 Host1 kernel: [<ffffffff80064a7d>] _write_lock+0xe/0xf
May 18 10:41:40 Host1 kernel: [<ffffffff887b8a93>] :lin_tape:erp_read_buffer+0x18d/0x5c4
May 18 10:41:40 Host1 kernel: [<ffffffff887b8ff7>] :lin_tape:tape_send_erp_cmd+0x12d/0x1d5
May 18 10:41:40 Host1 kernel: [<ffffffff887b99a9>] :lin_tape:read_dump+0x22/0xdd
May 18 10:41:40 Host1 kernel: [<ffffffff887b9c56>] :lin_tape:tape_check_simmim_dump_logsense+0x2c/0x40
May 18 10:41:40 Host1 kernel: [<ffffffff887ba220>] :lin_tape:tape_send_scsi_cmd+0xe0/0x220
May 18 10:41:40 Host1 kernel: [<ffffffff887a95b0>] :lin_tape:lin_tape_perform_read+0x10e/0x1b5
May 18 10:41:40 Host1 kernel: [<ffffffff887a507c>] :lin_tape:set_drive_busy+0x1b/0x3a
May 18 10:41:40 Host1 kernel: [<ffffffff887b1482>] :lin_tape:lin_tape_drive_read+0x1be/0x2e5
May 18 10:41:40 Host1 kernel: [<ffffffff887a39cd>] :lin_tape:lin_tape_read+0x220/0x273
May 18 10:41:40 Host1 kernel: [<ffffffff8000b735>] vfs_read+0xcb/0x171
May 18 10:41:40 Host1 kernel: [<ffffffff80011d8a>] sys_read+0x45/0x6e
May 18 10:41:40 Host1 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
May 18 10:41:40 Host1 kernel:
May 18 10:42:17 Host1 lin_taped[22502]: lin_taped terminated.