kernel: ata2.00: qc timeout (cmd 0xa0)

Latest response

Physical machine：HP DL385 G7 CPU:2*8 16core Mem:256GB ESXI5.5
VMware Virtual Platform Red Hat Enterprise Linux Server release 6.2
Kernel version：Linux 2.6.32-279.el6.x86_64 #1
Problem description:The same physical machine running multiple virtual machines, a day of continuous problems.CPU utilization close to 100%, IO busy, Ping packet loss serious, SSH login system can not operate normally.
The first virtual machine restart to resume normal; second virtual machine migration to other physical machine to restore normal; The third virtual machine problem can not Ping pass, migration can not solve the problem, restore normal after power off.
Physical machine normal, storage no bad disk.

View log information as follows:
The first virtual machine：
Dec 21 08:07:07 first ntpd[1901]: can't open /etc/ntp.drift.TEMP: Permission denied
Dec 21 08:34:03 first kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 08:34:03 first kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:34:03 first kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 08:34:03 first kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 08:34:03 first kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 08:34:03 first kernel: ata2.00: status: { DRDY ERR }
Dec 21 08:34:03 first kernel: ata2: soft resetting link
Dec 21 08:34:04 first kernel: ata2.00: configured for UDMA/33
Dec 21 08:34:04 first kernel: ata2: EH complete
Dec 21 08:39:28 first kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 08:39:28 first kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:39:28 first kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 08:39:28 first kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 08:39:28 first kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 08:39:28 first kernel: ata2.00: status: { DRDY ERR }
Dec 21 08:39:28 first kernel: ata2: soft resetting link
Dec 21 08:39:36 first kernel: ata2.00: qc timeout (cmd 0xa1)
Dec 21 08:39:36 first kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Dec 21 08:39:36 first kernel: ata2.00: revalidation failed (errno=-5)
Dec 21 08:39:36 first kernel: ata2: soft resetting link
Dec 21 08:39:48 first kernel: ata2.00: qc timeout (cmd 0xa1)
Dec 21 08:39:48 first kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Dec 21 08:39:48 first kernel: ata2.00: revalidation failed (errno=-5)
Dec 21 08:39:48 first kernel: ata2: soft resetting link
Dec 21 08:39:50 first kernel: ata2.00: configured for UDMA/33
Dec 21 08:39:50 first kernel: ata2: EH complete
Dec 21 08:42:22 first kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 08:42:22 first kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:42:22 first kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 08:42:22 first kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 08:42:22 first kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 08:42:22 first kernel: ata2.00: status: { DRDY ERR }
Dec 21 08:42:22 first kernel: ata2: soft resetting link
Dec 21 08:42:25 first kernel: ata2.00: configured for UDMA/33
Dec 21 08:42:25 first kernel: ata2: EH complete
Dec 21 08:43:36 first kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 08:43:36 first kernel: ata2.00: limiting speed to UDMA/25:PIO4
Dec 21 08:43:36 first kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:43:36 first kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 08:43:36 first kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 08:43:36 first kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 08:43:36 first kernel: ata2.00: status: { DRDY ERR }
Dec 21 08:43:36 first kernel: ata2: soft resetting link
Dec 21 08:43:36 first kernel: ata2.00: configured for UDMA/25
Dec 21 08:43:36 first kernel: ata2: EH complete
Dec 21 08:44:21 first kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 08:44:21 first kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:44:21 first kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 08:44:21 first kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 08:44:21 first kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 08:44:21 first kernel: ata2.00: status: { DRDY ERR }
Dec 21 08:44:21 first kernel: ata2: soft resetting link
Dec 21 08:44:26 first kernel: ata2.00: qc timeout (cmd 0xa1)
Dec 21 08:44:26 first kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Dec 21 08:44:26 first kernel: ata2.00: revalidation failed (errno=-5)
Dec 21 08:44:26 first kernel: ata2: soft resetting link
Dec 21 08:44:37 first kernel: ata2.00: qc timeout (cmd 0xa1)
.......

Second virtual machine :
Dec 21 06:47:16 localhost ntpd[1880]: can't open /etc/ntp.drift.TEMP: Permission denied
Dec 21 07:47:16 localhost ntpd[1880]: can't open /etc/ntp.drift.TEMP: Permission denied
Dec 21 08:24:02 localhost kernel: ata2: lost interrupt (Status 0x58)
Dec 21 08:24:02 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:24:02 localhost kernel: sr 1:0:0:0: CDB: Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Dec 21 08:24:02 localhost kernel: ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Dec 21 08:24:02 localhost kernel: res 40/00:03:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Dec 21 08:24:02 localhost kernel: ata2.00: status: { DRDY }
Dec 21 08:24:02 localhost kernel: ata2: soft resetting link
Dec 21 08:24:37 localhost kernel: ata2.00: configured for UDMA/33
Dec 21 08:24:37 localhost kernel: ata2: EH complete
Dec 21 08:24:37 localhost kernel: sr0: CDROM (ioctl) error, command: Get event status notification 4a 01 00 00 10 00 00 00 08 00
Dec 21 08:24:37 localhost kernel: sr: Sense Key : Aborted Command [current] [descriptor]
Dec 21 08:24:37 localhost kernel: sr: Add. Sense: No additional sense information
Dec 21 08:27:26 localhost kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 08:27:26 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 08:27:26 localhost kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 08:27:26 localhost kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 08:27:26 localhost kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 08:27:26 localhost kernel: ata2.00: status: { DRDY ERR }
Dec 21 08:27:26 localhost kernel: ata2: soft resetting link
Dec 21 08:27:27 localhost kernel: ata2.00: configured for UDMA/33
Dec 21 08:27:28 localhost kernel: ata2: EH complete
Dec 21 08:36:25 localhost abrt[27085]: saved core dump of pid 27084 (/usr/libexec/fprintd) to /var/spool/abrt/ccpp-2016-12-21-08:36:25-
27084.new/coredump (778240 bytes)
Dec 21 11:06:13 localhost kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 11:06:13 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 11:06:13 localhost kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 11:06:13 localhost kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 11:06:13 localhost kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 11:06:13 localhost kernel: ata2.00: status: { DRDY ERR }
Dec 21 11:06:13 localhost kernel: ata2: soft resetting link
Dec 21 11:06:14 localhost kernel: ata2.00: configured for UDMA/33
Dec 21 11:06:14 localhost kernel: ata2: EH complete
Dec 21 11:51:29 localhost kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 21 11:51:29 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 21 11:51:29 localhost kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 21 11:51:29 localhost kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 21 11:51:29 localhost kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 21 11:51:29 localhost kernel: ata2.00: status: { DRDY ERR }
Dec 21 11:51:29 localhost kernel: ata2: soft resetting link
Dec 21 11:51:35 localhost kernel: ata2.00: qc timeout (cmd 0xa1)
Dec 21 11:51:35 localhost kernel: ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Dec 21 11:51:35 localhost kernel: ata2.00: revalidation failed (errno=-5)
Dec 21 11:51:35 localhost kernel: ata2: soft resetting link
Dec 21 11:51:38 localhost kernel: ata2.00: configured for UDMA/33
Dec 21 11:51:38 localhost kernel: ata2: EH complete
.....

The third virtual machine:
Dec 22 05:18:23 localhost ntpd[2199]: can't open /etc/ntp.drift.TEMP: Permission denied
Dec 22 06:18:23 localhost ntpd[2199]: can't open /etc/ntp.drift.TEMP: Permission denied
Dec 22 06:32:25 localhost kernel: ------------[ cut here ]------------
Dec 22 06:32:25 localhost kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26d/0x280() (Not tainted)
Dec 22 06:32:25 localhost kernel: Hardware name: VMware Virtual Platform
Dec 22 06:32:25 localhost kernel: ata2.00: qc timeout (cmd 0xa0)
Dec 22 06:32:25 localhost kernel: NETDEV WATCHDOG: eth1 (e1000): transmit queue 0 timed out
Dec 22 06:32:25 localhost kernel: Modules linked in: fuse iptable_filter ip_tables bridge target_core_iblock target_core_file target_co
re_pscsi
Dec 22 06:32:25 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 22 06:32:25 localhost kernel: target_core_mod configfs bnx2fc cnic uio
Dec 22 06:32:25 localhost kernel: sr 1:0:0:0: CDB: Test Unit Ready: 00 00 00 00 00 00
Dec 22 06:32:25 localhost kernel: ata2.00: cmd a0/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Dec 22 06:32:25 localhost kernel: res 51/20:03:00:00:00/00:00:00:00:00/a0 Emask 0x5 (timeout)
Dec 22 06:32:25 localhost kernel: ata2.00: status: { DRDY ERR }
Dec 22 06:32:25 localhost kernel: fcoe libfcoe libfc 8021q scsi_transport_fc garp scsi_tgt stp llc vsock(U) ip6t_REJECT nf_conntrack_ip
v6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 uinput ppdev parport_pc parport e1000 sg vmware_balloon microco
de vmci(U) i2c_piix4 i2c_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom vmw_pvscsi pata_acpi ata_generic ata_piix dm_mirr
or dm_region_hash dm_log dm_mod [last unloaded: mperf]
Dec 22 06:32:25 localhost kernel: Pid: 24025, comm: amqrmppa Not tainted 2.6.32-279.el6.x86_64 #1
Dec 22 06:32:25 localhost kernel: Call Trace:
Dec 22 06:32:25 localhost kernel: [] ? warn_slowpath_common+0x87/0xc0
Dec 22 06:32:25 localhost kernel: [] ? warn_slowpath_fmt+0x46/0x50
Dec 22 06:32:25 localhost kernel: [] ? dev_watchdog+0x26d/0x280
Dec 22 06:32:25 localhost kernel: [] ? insert_work+0x6d/0xb0
Dec 22 06:32:25 localhost kernel: [] ? internal_add_timer+0xb5/0x110
Dec 22 06:32:25 localhost kernel: [] ? dev_watchdog+0x0/0x280
Dec 22 06:32:25 localhost kernel: [] ? run_timer_softirq+0x197/0x340
Dec 22 06:32:25 localhost kernel: [] ? rb_insert_color+0x125/0x160
Dec 22 06:32:25 localhost kernel: [] ? lapic_next_event+0x1d/0x30
Dec 22 06:32:25 localhost kernel: [] ? __do_softirq+0xc1/0x1e0
Dec 22 06:32:25 localhost kernel: [] ? hrtimer_interrupt+0x140/0x250
Dec 22 06:32:25 localhost kernel: [] ? call_softirq+0x1c/0x30
Dec 22 06:32:25 localhost kernel: [] ? do_softirq+0x65/0xa0
Dec 22 06:32:25 localhost kernel: [] ? irq_exit+0x85/0x90
Dec 22 06:32:25 localhost kernel: [] ? smp_apic_timer_interrupt+0x70/0x9b
Dec 22 06:32:25 localhost kernel: [] ? apic_timer_interrupt+0x13/0x20
Dec 22 06:32:25 localhost kernel:
Dec 22 06:32:25 localhost kernel: ---[ end trace 0d29e419b1d0e248 ]---
Dec 22 06:32:25 localhost kernel: e1000 0000:02:00.0: eth1: Reset adapter
Dec 22 06:32:25 localhost kernel: ata2: soft resetting link
Dec 22 06:32:25 localhost kernel: ata2.00: configured for UDMA/33
Dec 22 06:32:25 localhost kernel: ata2: EH complete
.....

Also please help look at what is the problem,thanks!

Responses