rport-1:0-0: blocked FC remote port time out: saving binding

Issue

• System /var/log/messages, /var/log/dmesg, or vmcore contain the following logged events indicating FC path lost between host's HBA and storage:

  {RHEL6,7}
      Jun  5 23:46:53 HOST kernel:  rport-1:0-0: blocked FC remote port time out: removing target and saving binding
      Jun  5 23:46:53 HOST kernel:  rport-1:0-1: blocked FC remote port time out: removing target and saving binding
  
  {RHEL5}
      Jun  5 23:46:53 HOST kernel:  rport-1:0-0: blocked FC remote port time out: saving binding
      Jun  5 23:46:53 HOST kernel:  rport-1:0-1: blocked FC remote port time out: saving binding
  
     -or if option fc_remove_on_dev_loss set-
      Jun  5 23:46:53 HOST kernel:  rport-1:0-0: blocked FC remote port time out: removing target and saving binding
      Jun  5 23:46:53 HOST kernel:  rport-1:0-1: blocked FC remote port time out: removing target and saving binding
  

• Device mapper multipath paths going down and multipath -ll shows the path down:

  360060e80056204000000620400000085 dm-1 HP,OPEN-V
  [size=30G][features=1 queue_if_no_path][hwhandler=0][rw]
   \_ round-robin 0 [prio=2][active]
   \_ 2:0:0:1 sdb 8:16  [active][ready]
   \_ 3:0:0:1 sdd 8:48  [failed][ready]   <---
                         ^^^^^^
  

• Found following messages logged in /var/log/messages, what could be possible cause and solution of this issue:

  Jan 20 15:54:20 HOST kernel: qla2xxx 0000:0f:00.1: LOOP DOWN detected (2 3 0).
  Jan 20 15:54:36 HOST kernel:  rport-3:0-0: blocked FC remote port time out: saving binding
  Jan 20 15:54:36 HOST kernel: sd 3:0:0:1: SCSI error: return code = 0x00010000
  Jan 20 15:54:36 HOST kernel: end_request: I/O error, dev sdd, sector 593496
  Jan 20 15:54:36 HOST kernel: device-mapper: multipath: Failing path 8:48.
  

• Ext4 filesystem mount goes read-only and cause system hang, observed blocked FC remote port time out: saving binding, in messages.

• Server kernel panic and vmcore contain the blocked FC remote port error message, what does cause this issue?