System gets crashed with error - kernel BUG at block/blk-core.c:2601!

Issue

• While executing the following test with RHEL 7.2 kernel-3.10.0-327.el7, system gets crashed with error - kernel BUG at block/blk-core.c:2601!

  • Configured a system with 2 Qlogic FC HBAs.
  • Configured 249 NPIV ports on each FC port.
  • Assigned the SAN devices to all 249 NPIV ports and started the IO.
  • When some of the target FC ports were manually disabled, SCSI error handling process was triggered, but soon the NPIV host got crashed with followin error messages:

Errors observed at the time of crash:

    [40941.482609] sd 318:0:2:1: rejecting I/O to offline device
    [40941.482612] sd 318:0:2:1: rejecting I/O to offline device
    [40941.482696] qla2xxx [0000:04:00.0]-3821:15: Port to be marked lost on fcport=010200, current port state= LOST.
    [40941.482699] qla2xxx [0000:04:00.0]-3822:15: FCP command status: 0x29-0x0 (0xe0000) nexus=1:2:1 portid=010200 oxid=0x0 cdb=2800087bd42000000800 len=0x1000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880425f0a800 cp=ffff8800a13a8700.
    [40941.482702] qla2xxx [0000:04:00.0]-3821:15: Port to be marked lost on fcport=010200, current port state= LOST.
    [40941.482704] qla2xxx [0000:04:00.0]-3822:15: FCP command status: 0x29-0x0 (0xe0000) nexus=1:2:1 portid=010200 oxid=0x0 cdb=2a0001a7cb5600000800 len=0x1000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff88036391f0c0 cp=ffff880096dc0000.
    [40941.482867] ------------[ cut here ]------------
    [40941.482868] kernel BUG at block/blk-core.c:2601!
    [40941.482870] invalid opcode: 0000 [#1] SMP 
    [...]
    [40941.482921]  i2c_core pps_core megaraid_sas scsi_tgt dca
    [40941.482924] CPU: 0 PID: 545 Comm: kworker/0:1H Not tainted 3.10.0-327.el7.x86_64 #1
    [40941.482924] Hardware name:    /01W23F, BIOS 2.2.2 01/16/2014
    [40941.482931] Workqueue: kblockd blk_delay_work
    [40941.482932] task: ffff880424d50b80 ti: ffff880424f9c000 task.ti: ffff880424f9c000
    [40941.482936] RIP: 0010:[<ffffffff812cae60>]  [<ffffffff812cae60>] blk_finish_request+0xf0/0x100
    [40941.482938] RSP: 0018:ffff88042fa03d58  EFLAGS: 00010006
    [40941.482939] RAX: 0000000000000246 RBX: ffff8803889cb780 RCX: 000000000000223c
    [40941.482940] RDX: 00000000223c223c RSI: 0000000000000000 RDI: ffff8803889cb780
    [40941.482941] RBP: ffff88042fa03d68 R08: 0000000000000246 R09: 0000000000000150
    [40941.482941] R10: ffff88042fa1028c R11: 0000000000001000 R12: 0000000000000000
    [40941.482942] R13: 0000000000000000 R14: ffff8800aea4b0f0 R15: ffff8803f1fac800
    [40941.482943] FS:  0000000000000000(0000) GS:ffff88042fa00000(0000) knlGS:0000000000000000
    [40941.482944] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [40941.482944] CR2: 00007f9d6bca9000 CR3: 000000000194a000 CR4: 00000000001407f0
    [40941.482945] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    [40941.482946] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    [40941.482946] Stack:
    [40941.482951]  ffff8803889cb780 ffff880324d6c700 ffff88042fa03db0 ffffffff81420e9b
    [40941.482953]  0000000000000246 0000000029662540 0000000000000000 0000000000000000
    [40941.482955]  ffff8800aea4b0f0 ffff880324d6c700 ffff8803889cb780 ffff88042fa03e10
    [40941.482955] Call Trace:
    [40941.482957]  <IRQ> 
    [40941.482961]  [<ffffffff81420e9b>] scsi_end_request+0x12b/0x1d0
    [40941.482964]  [<ffffffff814210a3>] scsi_io_completion+0x103/0x600
    [40941.482966]  [<ffffffff814163e5>] scsi_finish_command+0xd5/0x130
    [40941.482969]  [<ffffffff8142057a>] scsi_softirq_done+0x12a/0x150
    [40941.482973]  [<ffffffff812d16a0>] blk_done_softirq+0x90/0xc0
    [40941.482978]  [<ffffffff81084b0f>] __do_softirq+0xef/0x280
    [40941.482983]  [<ffffffff8164721c>] call_softirq+0x1c/0x30
    [40941.482988]  [<ffffffff81016fc5>] do_softirq+0x65/0xa0
    [40941.482990]  [<ffffffff81084ea5>] irq_exit+0x115/0x120
    [40941.482992]  [<ffffffff81647db8>] do_IRQ+0x58/0xf0
    [40941.482996]  [<ffffffff8163d0ed>] common_interrupt+0x6d/0x6d
    [40941.482997]  <EOI> 
    [40941.482999]  [<ffffffff814206cf>] ? scsi_request_fn+0x12f/0x770
    [40941.483003]  [<ffffffff812c7465>] ? blk_run_queue+0x35/0x40
    [40941.483005]  [<ffffffff812c73e3>] __blk_run_queue+0x33/0x40
    [40941.483008]  [<ffffffff812c9515>] blk_delay_work+0x25/0x60
    [40941.483012]  [<ffffffff8109d5fb>] process_one_work+0x17b/0x470
    [40941.483013]  [<ffffffff8109e3cb>] worker_thread+0x11b/0x400
    [40941.483016]  [<ffffffff8109e2b0>] ? rescuer_thread+0x400/0x400
    [40941.483019]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
    [40941.483022]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
    [40941.483025]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
    [40941.483028]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
    [40941.483052] Code: 89 de e8 64 13 00 00 e9 4b ff ff ff 83 7b 50 01 0f 85 58 ff ff ff 48 8b 43 38 48 8d b8 90 01 00 00 e8 75 a8 ea ff e9 43 ff ff ff <0f> 0b 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 
    [40941.483054] RIP  [<ffffffff812cae60>] blk_finish_request+0xf0/0x100
    [40941.483056]  RSP <ffff88042fa03d58>