RHEL6.5: soft lockups with HP perfd / scopeux in on a machine with a lot of disks
Issue
- Seeing periodic soft lockup backtraces which eventually clear, all look similar to the below, with
disk_seqf_start -> class_dev_iter_next -> klist_next -> _spin_lock
Aug 20 20:19:22 localhost kernel: <EOI>
Aug 20 20:19:22 localhost kernel: [<ffffffff811ae790>] ? seq_read+0x0/0x400
Aug 20 20:19:22 localhost kernel: [<ffffffff8150dcc0>] ? klist_release+0x0/0xd0
Aug 20 20:19:22 localhost kernel: [<ffffffff8152bcc1>] ? _spin_lock+0x21/0x30
Aug 20 20:19:22 localhost kernel: [<ffffffff8150db0f>] ? klist_next+0x2f/0xf0
Aug 20 20:19:22 localhost kernel: [<ffffffff8136f610>] ? class_dev_iter_next+0x20/0x60
Aug 20 20:19:22 localhost kernel: [<ffffffff8127140c>] ? disk_seqf_start+0x8c/0xb0
Aug 20 20:19:22 localhost kernel: [<ffffffff811ae826>] ? seq_read+0x96/0x400
Aug 20 20:19:22 localhost kernel: [<ffffffff8105af90>] ? __dequeue_entity+0x30/0x50
Aug 20 20:19:22 localhost kernel: [<ffffffff81069290>] ? pick_next_task_fair+0xd0/0x130
Aug 20 20:19:22 localhost kernel: [<ffffffff811f3f8e>] ? proc_reg_read+0x7e/0xc0
Aug 20 20:19:22 localhost kernel: [<ffffffff81189715>] ? vfs_read+0xb5/0x1a0
Aug 20 20:19:22 localhost kernel: [<ffffffff81189851>] ? sys_read+0x51/0x90
Aug 20 20:19:22 localhost kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Environment
- Red Hat Enterprise Linux 6.5
- seen on kernel 2.6.32-431.63.1.el6
- HP performance monitoring tools (perfd, scopeux, etc)
- thousands of disks
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.