Performance degradation after "BUG: soft lockup - CPU#24 stuck for 67s! [xfsdatad/24:22292]" messages.

Issue

• CPU soft lockup messages are logged in /var/log/messages and performance is degraded

2013-08-07T02:32:01.070399-04:00 example kernel: BUG: soft lockup - CPU#24 stuck for 67s! [xfsdatad/24:22292]
2013-08-07T02:32:01.070452-04:00 example kernel: Modules linked in: gfs2 dlm configfs bnx2fc cnic uio fcoe libfcoe libfc sunrpc acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 ipt_LOG xt_limit ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_multiport xt_iprange iptable_filter ip_tables xfs exportfs vfat fat microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support cdc_ether usbnet mii shpchp ioatdma dca i7core_edac edac_core e1000e sg bnx2 ext4 mbcache jbd2 dm_round_robin scsi_dh_alua sr_mod cdrom sd_mod crc_t10dif ocz10xx(P)(U) pata_acpi ata_generic ata_piix lpfc scsi_transport_fc scsi_tgt megaraid_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
2013-08-07T02:32:01.070456-04:00 example kernel: CPU 24 
2013-08-07T02:32:01.070466-04:00 example kernel: Modules linked in: gfs2 dlm configfs bnx2fc cnic uio fcoe libfcoe libfc sunrpc acpi_cpufreq freq_table mperf bonding 8021q garp stp llc ipv6 ipt_LOG xt_limit ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_multiport xt_iprange iptable_filter ip_tables xfs exportfs vfat fat microcode serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support cdc_ether usbnet mii shpchp ioatdma dca i7core_edac edac_core e1000e sg bnx2 ext4 mbcache jbd2 dm_round_robin scsi_dh_alua sr_mod cdrom sd_mod crc_t10dif ocz10xx(P)(U) pata_acpi ata_generic ata_piix lpfc scsi_transport_fc scsi_tgt megaraid_sas wmi dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
2013-08-07T02:32:01.070469-04:00 example kernel: 
2013-08-07T02:32:01.070473-04:00 example kernel: Pid: 22292, comm: xfsdatad/24 Tainted: P           ---------------    2.6.32-279.22.1.el6.x86_64 #1 IBM System x3850 X5 -[7143AC1]-/Node 1, Processor Card
2013-08-07T02:32:01.070478-04:00 example kernel: RIP: 0010:[<ffffffffa0146916>]  [<ffffffffa0146916>] List_Del+0x26/0x170 [ocz10xx]
2013-08-07T02:32:01.070482-04:00 example kernel: RSP: 0018:ffff8848d4a03ae0  EFLAGS: 00000246
2013-08-07T02:32:01.075832-04:00 example kernel: RAX: ffff8831b8b2a010 RBX: ffff8848d4a03b00 RCX: ffffc90052142000
2013-08-07T02:32:01.075881-04:00 example kernel: RDX: ffff8848d4a03b30 RSI: 000000000000002f RDI: ffff8831b8b22010
2013-08-07T02:32:01.075886-04:00 example kernel: RBP: ffffffff8100bb93 R08: 0000000000000000 R09: ffffffffa018f248
2013-08-07T02:32:01.075888-04:00 example kernel: R10: ffff8831b8b12010 R11: 0000000000000000 R12: ffff8848d4a03a60
2013-08-07T02:32:01.075891-04:00 example kernel: R13: ffff8831b8b22000 R14: ffff8831b8b22010 R15: ffffffff814f235b
2013-08-07T02:32:01.075894-04:00 example kernel: FS:  0000000000000000(0000) GS:ffff8848d4a00000(0000) knlGS:0000000000000000
2013-08-07T02:32:01.075896-04:00 example kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
2013-08-07T02:32:01.075904-04:00 example kernel: CR2: 0000003657672750 CR3: 0000005dc1721000 CR4: 00000000000006e0
2013-08-07T02:32:01.075907-04:00 example kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2013-08-07T02:32:01.075910-04:00 example kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2013-08-07T02:32:01.075913-04:00 example kernel: Process xfsdatad/24 (pid: 22292, threadinfo ffff8860375ac000, task ffff8860375ab500)
2013-08-07T02:32:01.075915-04:00 example kernel: Stack:
2013-08-07T02:32:01.075945-04:00 example kernel: ffff8831b8b12000 ffffc90052abb000 ffff8831b8b06010 ffffc90052abb938
2013-08-07T02:32:01.075949-04:00 example kernel: <d> ffff8848d4a03b70 ffffffffa0147f23 ffff884841745200 0000000000000282
2013-08-07T02:32:01.075951-04:00 example kernel: <d> 0000000000000011 ffffc90052abb310 ffff884ad45c6010 ffff8831b8b12010
2013-08-07T02:32:01.076013-04:00 example kernel: Call Trace:
2013-08-07T02:32:01.076018-04:00 example kernel: <IRQ> 
2013-08-07T02:32:01.076022-04:00 example kernel: [<ffffffffa0147f23>] ? io_chip_handle_waiting_queue+0x53/0x5b0 [ocz10xx]
2013-08-07T02:32:01.076025-04:00 example kernel: [<ffffffffa01484ac>] ? core_handle_waiting_queue+0x2c/0x40 [ocz10xx]
2013-08-07T02:32:01.076028-04:00 example kernel: [<ffffffffa0148510>] ? core_push_queues+0x50/0x90 [ocz10xx]
2013-08-07T02:32:01.076031-04:00 example kernel: [<ffffffffa012a4e1>] ? mv_linux_queue_command+0x2f1/0x6d0 [ocz10xx]
2013-08-07T02:32:01.076034-04:00 example kernel: [<ffffffff81355941>] ? scsi_dispatch_cmd+0x101/0x360
2013-08-07T02:32:01.076040-04:00 example kernel: [<ffffffff8135d68d>] ? scsi_request_fn+0x5ed/0x7a0
2013-08-07T02:32:01.076043-04:00 example kernel: [<ffffffff8107cc1d>] ? del_timer+0x7d/0xe0
2013-08-07T02:32:01.076046-04:00 example kernel: [<ffffffff8124d041>] ? __blk_run_queue+0x31/0x40
2013-08-07T02:32:01.076049-04:00 example kernel: [<ffffffff8124d190>] ? blk_run_queue+0x30/0x50
2013-08-07T02:32:01.076051-04:00 example kernel: [<ffffffff8135c8a7>] ? scsi_run_queue+0xc7/0x280
2013-08-07T02:32:01.076055-04:00 example kernel: [<ffffffff813562a0>] ? __scsi_put_command+0x60/0xa0
2013-08-07T02:32:01.076058-04:00 example kernel: [<ffffffff8135da52>] ? scsi_next_command+0x42/0x60
2013-08-07T02:32:01.076061-04:00 example kernel: [<ffffffff8135e7ce>] ? scsi_io_completion+0x2ae/0x6c0
2013-08-07T02:32:01.076067-04:00 example kernel: [<ffffffff81355642>] ? scsi_finish_command+0xc2/0x130
2013-08-07T02:32:01.076070-04:00 example kernel: [<ffffffff8135ed45>] ? scsi_softirq_done+0x145/0x170
2013-08-07T02:32:01.076073-04:00 example kernel: [<ffffffff81255105>] ? blk_done_softirq+0x85/0xa0
2013-08-07T02:32:01.076075-04:00 example kernel: [<ffffffff81072ac1>] ? __do_softirq+0xc1/0x1e0
2013-08-07T02:32:01.076078-04:00 example kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
2013-08-07T02:32:01.076081-04:00 example kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
2013-08-07T02:32:01.076083-04:00 example kernel: [<ffffffff810728a5>] ? irq_exit+0x85/0x90
2013-08-07T02:32:01.083032-04:00 example kernel: [<ffffffff8102a2a5>] ? smp_call_function_single_interrupt+0x35/0x40
2013-08-07T02:32:01.083060-04:00 example kernel: [<ffffffff8100bd33>] ? call_function_single_interrupt+0x13/0x20
2013-08-07T02:32:01.083062-04:00 example kernel: <EOI> 
2013-08-07T02:32:01.083152-04:00 example kernel: [<ffffffff811a9466>] ? end_buffer_async_write+0x46/0x190
2013-08-07T02:32:01.083154-04:00 example kernel: [<ffffffffa03b4bd4>] ? xfs_destroy_ioend+0x44/0xa0 [xfs]
2013-08-07T02:32:01.083156-04:00 example kernel: [<ffffffffa03b4d60>] ? xfs_end_io+0x0/0xb0 [xfs]
2013-08-07T02:32:01.083159-04:00 example kernel: [<ffffffffa03b4da8>] ? xfs_end_io+0x48/0xb0 [xfs]
2013-08-07T02:32:01.083160-04:00 example kernel: [<ffffffff8108b370>] ? worker_thread+0x170/0x2a0
2013-08-07T02:32:01.083166-04:00 example kernel: [<ffffffff81090be0>] ? autoremove_wake_function+0x0/0x40
2013-08-07T02:32:01.083168-04:00 example kernel: [<ffffffff8108b200>] ? worker_thread+0x0/0x2a0
2013-08-07T02:32:01.083170-04:00 example kernel: [<ffffffff81090876>] ? kthread+0x96/0xa0
2013-08-07T02:32:01.083171-04:00 example kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
2013-08-07T02:32:01.083173-04:00 example kernel: [<ffffffff810907e0>] ? kthread+0x0/0xa0
2013-08-07T02:32:01.083174-04:00 example kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
2013-08-07T02:32:01.083177-04:00 example kernel: Code: 00 00 00 00 00 55 48 89 e5 53 48 83 ec 18 0f 1f 44 00 00 48 8b 07 48 89 fb 48 85 c0 74 37 48 3b 78 08 75 39 48 8b 43 08 48 3b 18 <48> 89 c2 0f 85 a9 00 00 00 48 8b 13 48 89 10 48 89 42 08 48 c7 
2013-08-07T02:32:01.083180-04:00 example kernel: Call Trace:
2013-08-07T02:32:01.083183-04:00 example kernel: <IRQ>  [<ffffffffa0147f23>] ? io_chip_handle_waiting_queue+0x53/0x5b0 [ocz10xx]
2013-08-07T02:32:01.083185-04:00 example kernel: [<ffffffffa01484ac>] ? core_handle_waiting_queue+0x2c/0x40 [ocz10xx]
2013-08-07T02:32:01.083186-04:00 example kernel: [<ffffffffa0148510>] ? core_push_queues+0x50/0x90 [ocz10xx]
2013-08-07T02:32:01.083188-04:00 example kernel: [<ffffffffa012a4e1>] ? mv_linux_queue_command+0x2f1/0x6d0 [ocz10xx]
2013-08-07T02:32:01.083190-04:00 example kernel: [<ffffffff81355941>] ? scsi_dispatch_cmd+0x101/0x360
2013-08-07T02:32:01.083191-04:00 example kernel: [<ffffffff8135d68d>] ? scsi_request_fn+0x5ed/0x7a0
2013-08-07T02:32:01.083195-04:00 example kernel: [<ffffffff8107cc1d>] ? del_timer+0x7d/0xe0
2013-08-07T02:32:01.083197-04:00 example kernel: [<ffffffff8124d041>] ? __blk_run_queue+0x31/0x40
2013-08-07T02:32:01.083199-04:00 example kernel: [<ffffffff8124d190>] ? blk_run_queue+0x30/0x50
2013-08-07T02:32:01.083200-04:00 example kernel: [<ffffffff8135c8a7>] ? scsi_run_queue+0xc7/0x280
2013-08-07T02:32:01.083202-04:00 example kernel: [<ffffffff813562a0>] ? __scsi_put_command+0x60/0xa0
2013-08-07T02:32:01.083204-04:00 example kernel: [<ffffffff8135da52>] ? scsi_next_command+0x42/0x60
2013-08-07T02:32:01.083205-04:00 example kernel: [<ffffffff8135e7ce>] ? scsi_io_completion+0x2ae/0x6c0
2013-08-07T02:32:01.083207-04:00 example kernel: [<ffffffff81355642>] ? scsi_finish_command+0xc2/0x130
2013-08-07T02:32:01.083210-04:00 example kernel: [<ffffffff8135ed45>] ? scsi_softirq_done+0x145/0x170
2013-08-07T02:32:01.083212-04:00 example kernel: [<ffffffff81255105>] ? blk_done_softirq+0x85/0xa0
2013-08-07T02:32:01.083214-04:00 example kernel: [<ffffffff81072ac1>] ? __do_softirq+0xc1/0x1e0
2013-08-07T02:32:01.083216-04:00 example kernel: [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
2013-08-07T02:32:01.083217-04:00 example kernel: [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
2013-08-07T02:32:01.083218-04:00 example kernel: [<ffffffff810728a5>] ? irq_exit+0x85/0x90
2013-08-07T02:32:01.083220-04:00 example kernel: [<ffffffff8102a2a5>] ? smp_call_function_single_interrupt+0x35/0x40
2013-08-07T02:32:01.083223-04:00 example kernel: [<ffffffff8100bd33>] ? call_function_single_interrupt+0x13/0x20
2013-08-07T02:32:01.083225-04:00 example kernel: <EOI>  [<ffffffff811a9466>] ? end_buffer_async_write+0x46/0x190
2013-08-07T02:32:01.083227-04:00 example kernel: [<ffffffffa03b4bd4>] ? xfs_destroy_ioend+0x44/0xa0 [xfs]
2013-08-07T02:32:01.083229-04:00 example kernel: [<ffffffffa03b4d60>] ? xfs_end_io+0x0/0xb0 [xfs]
2013-08-07T02:32:01.083231-04:00 example kernel: [<ffffffffa03b4da8>] ? xfs_end_io+0x48/0xb0 [xfs]
2013-08-07T02:32:01.083232-04:00 example kernel: [<ffffffff8108b370>] ? worker_thread+0x170/0x2a0
2013-08-07T02:32:01.083234-04:00 example kernel: [<ffffffff81090be0>] ? autoremove_wake_function+0x0/0x40
2013-08-07T02:32:01.083235-04:00 example kernel: [<ffffffff8108b200>] ? worker_thread+0x0/0x2a0
2013-08-07T02:32:01.083238-04:00 example kernel: [<ffffffff81090876>] ? kthread+0x96/0xa0
2013-08-07T02:32:01.083240-04:00 example kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
2013-08-07T02:32:01.083242-04:00 example kernel: [<ffffffff810907e0>] ? kthread+0x0/0xa0
2013-08-07T02:32:01.083243-04:00 example kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
2013-08-07T02:32:02.138142-04:00 example abrt-dump-oops: Reported 1 kernel oopses to Abrt