tgtd segfault in abort_task_set when disk failed in RAID as part DRBD mirror

Solution Verified - Updated -

Issue

  • tgtd segfaults on both nodes of a DRBD mirror with the following backtrace:
     (gdb) bt
#0  0x0000000000415ebf in abort_task_set (mreq=0x11911e0, target=0x111d550, itn_id=1001, tag=98, lun=, all=0)
    at target.c:1013
#1  0x000000000041623d in target_mgmt_request (tid=, itn_id=1001, req_id=18764304, function=13,
    lun_buf=, tag=98, host_no=0) at target.c:1060
#2  0x0000000000404f9e in iscsi_tm_execute (task=0x11e5210) at iscsi/iscsid.c:1395
#3  iscsi_task_execute (task=0x11e5210) at iscsi/iscsid.c:1440
#4  0x0000000000407648 in iscsi_task_queue (conn=0x119ffe8) at iscsi/iscsid.c:1517
#5  iscsi_task_rx_done (conn=0x119ffe8) at iscsi/iscsid.c:1658
#6  iscsi_rx_handler (conn=0x119ffe8) at iscsi/iscsid.c:2045
#7  0x000000000040c330 in iscsi_tcp_event_handler (fd=, events=1, data=0x119ffe8) at iscsi/iscsi_tcp.c:158
#8  0x0000000000411db9 in event_loop () at tgtd.c:279
  • Single physical disk failure in a RAID 1 causes a failure of both nodes in the iSCSI target over DRBD based mirror.
  • Messages seen involving disk failure and tgtd segfault were as follows:  
    • On the first node that encountered the disk failure:
      Mar 27 12:19:58 bs kernel: raid1: Disk failure on sdb1, disabling device.
Mar 27 12:19:58 bs kernel: raid1: Operation continuing on 3 devices.
Mar 27 12:19:58 bs kernel: end_request: I/O error, dev sdb, sector 0
Mar 27 12:19:58 bs kernel: sd 0:0:1:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
Mar 27 12:19:58 bs kernel: tgtd[5900]: segfault at 200000000 ip 0000000000415ebf sp 00007fff320fb450 error 4 in tgtd[400000+2f000]
  • On the second node (no disk failure):
    Mar 27 12:19:01 ts kernel: tgtd[5882]: segfault at 0 ip 0000000000415ebf sp 00007fff22f349d0 error 4 in tgtd[400000+2f000]
Mar 27 12:19:01 ts tgtd: abort_task_set(1008) found 3 0
Mar 27 12:19:58 ts abrt[30396]: saved core dump of pid 5882 (/usr/sbin/tgtd) to /var/spool/abrt/ccpp-1301242798-5882.new/coredump (107663360 bytes)

Environment

  • Red Hat Enterprise Linux 6             
    • kernel-2.6.32-71.14.1.el6.x86_64
    • scsi-target-utils-1.0.4-3.ps.el6.x86_64
  • Two machines with drbd mirroring configured with tgtd running an iSCSI target as follows:
    • iscsi target --> drbd devices --> crypt dev --> LV --> software raid10 (kernel / md driver) --> LSI mptsas --> OCZ Vertex SSD
  • Each machine has 4 SSDs in mdadm RAID 10 (data partitions) and RAID 1 (root/boot).

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content