kernel reports soft lockup messages in the `lin_tape` module on a RHEL 5 host

Solution Unverified - Updated -

Issue

  • System hangs with soft lockup messages in lin_taped
  • A cluster node was fenced immediately after reporting a soft-lockup in IBM's lin_taped
  • Why is my system showing soft lockups in the lin_taped module?
Aug 10 00:19:53 node1 kernel: BUG: soft lockup - CPU#9 stuck for 60s! [lin_taped:18931]
Aug 10 00:19:53 node1 kernel: CPU 9:
Aug 10 00:19:53 node1 kernel: Modules linked in: nfs nfs_acl hidp rfcomm l2cap bluetooth lock_dlm gfs2(U) dlm configfs lin_tape(U) lockd sunrpc bonding be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod st sg i7300_edac e1000e edac_mc pcspkr hpilo bnx2 tpm_tis tpm ide_cd tpm_bios serio_raw cdrom dm_raid45 dm_message dm_region_hash dm_mem_cache dm_round_robin dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_dh dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage ata_piix libata cciss shpchp qla2xxx scsi_transport_fc sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Aug 10 00:19:53 node1 kernel: Pid: 18931, comm: lin_taped Tainted: G     ---- 2.6.18-308.1.1.el5 #1
Aug 10 00:19:53 node1 kernel: RIP: 0010:[<ffffffff800621ad>]  [<ffffffff800621ad>] __read_lock_failed+0x5/0x14
Aug 10 00:19:53 node1 kernel: RSP: 0018:ffff81200c02fde8  EFLAGS: 00000297
Aug 10 00:19:53 node1 kernel: RAX: ffff81202f14a000 RBX: ffff81017c56e800 RCX: ffff81200c02fe5c
Aug 10 00:19:53 node1 kernel: RDX: 0000000000000296 RSI: ffff81017c56e800 RDI: ffff811fc010cbd8
Aug 10 00:19:53 node1 kernel: RBP: ffff811fdb6d9228 R08: 0000000000000000 R09: 0000000000000000
Aug 10 00:19:53 node1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000000000002de3c
Aug 10 00:19:53 node1 kernel: R13: 0004694b4e1de53c R14: ffff810170b42100 R15: ffff811fdb6d9040
Aug 10 00:19:53 node1 kernel: FS:  00002aaaaaac06e0(0000) GS:ffff810170b311c0(0000) knlGS:0000000000000000
Aug 10 00:19:53 node1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 10 00:19:53 node1 kernel: CR2: 00002aaabf00000a CR3: 0000001ff2114000 CR4: 00000000000006e0
Aug 10 00:19:53 node1 kernel: 
Aug 10 00:19:53 node1 kernel: Call Trace:
Aug 10 00:19:53 node1 kernel:  [<ffffffff800649f1>] __down+0x99/0xd8
Aug 10 00:19:53 node1 kernel:  [<ffffffff80064b55>] _read_lock+0xb/0xc
Aug 10 00:19:53 node1 kernel:  [<ffffffff88829108>] :lin_tape:lin_tape_poll_trace_drive+0x6f/0x2f1
Aug 10 00:19:53 node1 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Aug 10 00:19:53 node1 kernel:  [<ffffffff8882bbb7>] :lin_tape:lin_tape_poll_trace+0x17e/0x1f0
Aug 10 00:19:53 node1 kernel:  [<ffffffff88824741>] :lin_tape:lin_tape_ioctl+0x4e/0xba
Aug 10 00:19:53 node1 kernel:  [<ffffffff80041ea7>] do_ioctl+0x55/0x6b
Aug 10 00:19:53 node1 kernel:  [<ffffffff8002ff00>] vfs_ioctl+0x457/0x4b9
Aug 10 00:19:53 node1 kernel:  [<ffffffff800ba767>] audit_syscall_entry+0x1a8/0x1d3
Aug 10 00:19:53 node1 kernel:  [<ffffffff8004c23f>] sys_ioctl+0x59/0x78
Aug 10 00:19:53 node1 kernel:  [<ffffffff8005d28d>] tracesys+0xd5/0xe0
Aug 10 00:19:53 node1 kernel: 
May 18 10:41:40 Host1 kernel: BUG: soft lockup - CPU#8 stuck for 60s! [dsmserv:28732]
May 18 10:41:40 Host1 kernel: CPU 8:
May 18 10:41:40 Host1 kernel: Modules linked in: lin_tape(U) bonding ip_conntrack_netbios_ns ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi ext4 jbd2 crc16 dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport shpchp joydev st sg ide_cd tpm_tis tpm bnx2 cdrom tpm_bios e1000e pcspkr dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage lpfc scsi_transport_fc ata_piix libata megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
May 18 10:41:40 Host1 kernel: Pid: 28732, comm: dsmserv Tainted: G     ---- 2.6.18-308.16.1.el5 #1
May 18 10:41:40 Host1 kernel: RIP: 0010:[<ffffffff80062191>]  [<ffffffff80062191>] __write_lock_failed+0x9/0x20
May 18 10:41:40 Host1 kernel: RSP: 0018:ffff810e6d34bc98  EFLAGS: 00000287
May 18 10:41:40 Host1 kernel: RAX: 000000000000000a RBX: 00000000fffffff4 RCX: 0000000000000016
May 18 10:41:40 Host1 kernel: RDX: 000000000000000a RSI: ffff810e6d34bcf0 RDI: ffff811c764a8ba8
May 18 10:41:40 Host1 kernel: RBP: ffff811c75ccade0 R08: 0000000000000001 R09: 0000000000000000
May 18 10:41:40 Host1 kernel: R10: 0000000000000000 R11: ffff811c7c7b8000 R12: ffff811c75cca8c0
May 18 10:41:40 Host1 kernel: R13: 0000000000000004 R14: ffff811c764a8800 R15: 00000000fffffff4
May 18 10:41:40 Host1 kernel: FS:  00002b7b30a4c940(0000) GS:ffff810163ef99c0(0000) knlGS:0000000000000000
May 18 10:41:40 Host1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May 18 10:41:40 Host1 kernel: CR2: 00000000006bb544 CR3: 0000001c58eba000 CR4: 00000000000006a0
May 18 10:41:40 Host1 kernel:
May 18 10:41:40 Host1 kernel: Call Trace:
May 18 10:41:40 Host1 kernel: [<ffffffff80064a7d>] _write_lock+0xe/0xf
May 18 10:41:40 Host1 kernel: [<ffffffff887b8a93>] :lin_tape:erp_read_buffer+0x18d/0x5c4
May 18 10:41:40 Host1 kernel: [<ffffffff887b8ff7>] :lin_tape:tape_send_erp_cmd+0x12d/0x1d5
May 18 10:41:40 Host1 kernel: [<ffffffff887b99a9>] :lin_tape:read_dump+0x22/0xdd
May 18 10:41:40 Host1 kernel: [<ffffffff887b9c56>] :lin_tape:tape_check_simmim_dump_logsense+0x2c/0x40
May 18 10:41:40 Host1 kernel: [<ffffffff887ba220>] :lin_tape:tape_send_scsi_cmd+0xe0/0x220
May 18 10:41:40 Host1 kernel: [<ffffffff887a95b0>] :lin_tape:lin_tape_perform_read+0x10e/0x1b5
May 18 10:41:40 Host1 kernel: [<ffffffff887a507c>] :lin_tape:set_drive_busy+0x1b/0x3a
May 18 10:41:40 Host1 kernel: [<ffffffff887b1482>] :lin_tape:lin_tape_drive_read+0x1be/0x2e5
May 18 10:41:40 Host1 kernel: [<ffffffff887a39cd>] :lin_tape:lin_tape_read+0x220/0x273
May 18 10:41:40 Host1 kernel: [<ffffffff8000b735>] vfs_read+0xcb/0x171
May 18 10:41:40 Host1 kernel: [<ffffffff80011d8a>] sys_read+0x45/0x6e
May 18 10:41:40 Host1 kernel: [<ffffffff8005d28d>] tracesys+0xd5/0xe0
May 18 10:41:40 Host1 kernel:
May 18 10:42:17 Host1 lin_taped[22502]: lin_taped terminated.

Environment

  • Red Hat Enterprise Linux (RHEL) 5
  • IBM lin_tape installed

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.