IO operations on mdraid raid5 array getting stuck during heavy IO load

Issue

• IO operations on mdraid raid5 array getting stuck during heavy IO load

• At the time of issue, the file systems created over mdraid array become unusable and processes accessing those file systems get hung:

  kernel: INFO: task kworker/u97:5:311 blocked for more than 120 seconds.
  kernel:      Not tainted 4.18.0-477.15.1.el8_8.x86_64 #1
  kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  kernel: task:kworker/u97:5   state:D stack:    0 pid:  311 ppid:     2 flags:0x80004000
  kernel: Workqueue: writeback wb_workfn (flush-9:126)
  kernel: Call Trace:
  kernel: __schedule+0x2d1/0x870
  kernel: ? blk_flush_plug_list+0xc2/0x100
  kernel: schedule+0x55/0xf0
  kernel: raid5_get_active_stripe+0x21e/0x4e0 [raid456]
  kernel: ? finish_wait+0x80/0x80
  kernel: raid5_make_request+0x20c/0xe00 [raid456]
  kernel: ? do_wait_intr_irq+0xa0/0xa0
  kernel: ? finish_wait+0x80/0x80
  kernel: md_handle_request+0x128/0x1b0
  kernel: md_make_request+0x5b/0xb0
  kernel: generic_make_request_no_check+0x202/0x330
  kernel: submit_bio+0x3c/0x160
  kernel: iomap_submit_ioend.isra.47+0x4a/0x80
  kernel: xfs_vm_writepages+0x7e/0xb0 [xfs]
  kernel: do_writepages+0xc6/0x1d0
  kernel: __writeback_single_inode+0x39/0x300
  kernel: writeback_sb_inodes+0x1ea/0x450
  kernel: __writeback_inodes_wb+0x5f/0xd0
  kernel: wb_writeback+0x24c/0x2e0
  kernel: wb_workfn+0x34a/0x4d0
  kernel: process_one_work+0x1a7/0x360
  kernel: worker_thread+0x30/0x390
  kernel: ? create_worker+0x1a0/0x1a0
  kernel: kthread+0x134/0x150
  kernel: ? set_kthread_struct+0x50/0x50
  kernel: ret_from_fork+0x35/0x40
  

• Rolling back to a previous kernel e.g. kernel-4.18.0-425.19.2.el8_7.x86_64 solves the issue.
  So it looks to be a regression in mdraid raid5 code in RHEL 8.8 kernel