System was crashed with I/O error in Red Hat Enterprise Linux 5.
Issue
-
This is NOT AN ACTUAL PROBLEM!
-
Run maintenance script with iSCSI device, system crashed suddenly.
iSCSI device is always cut off from online system, it is connected maintenance mode only.
crash 4.0-8.9.1.el5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.18-164.el5PAE/vmlinux
DUMPFILE: vmcore
CPUS: 1
DATE: Mon Jan 18 19:09:51 2010
UPTIME: 01:22:56
LOAD AVERAGE: 3.55, 1.45, 0.56
TASKS: 92
NODENAME:
RELEASE: 2.6.18-164.el5PAE
VERSION: #1 SMP Tue Aug 18 15:59:11 EDT 2009
MACHINE: i686 (2332 Mhz)
MEMORY: 512 MB
PANIC: "Kernel panic - not syncing: EXT3-fs (device sdc1): panic forced after error"
PID: 14995
COMMAND: "ls"
TASK: d65f1aa0 [THREAD_INFO: d55f5000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
>log
.
.
EXT3 FS on sdc1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sdc1, type ext3), uses xattr
connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4546303, last ping 4551303, now 4556304
connection1:0: detected conn error (1011)
session1: session recovery timed out after 120 secs
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9291
Buffer I/O error on device sdc1, logical block 4645
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 1035
Buffer I/O error on device sdc1, logical block 517
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 521
Buffer I/O error on device sdc1, logical block 260
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 261
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 3
Buffer I/O error on device sdc1, logical block 1
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9431
Buffer I/O error on device sdc1, logical block 4715
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9411
Buffer I/O error on device sdc1, logical block 4705
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 4706
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 4707
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 4708
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9427
Aborting journal on device sdc1.
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 1063
EXT3-fs error (device sdc1) in ext3_reserve_inode_write: Journal has aborted
Kernel panic - not syncing: EXT3-fs (device sdc1): panic forced after error
crash> bt
PID: 14995 TASK: d65f1aa0 CPU: 0 COMMAND: "ls"
#0 [d55f5e78] crash_kexec at c0440ecf
#1 [d55f5ebc] panic at c04244b9
#2 [d55f5ed8] ext3_handle_error at e08a77aa
#3 [d55f5eec] __ext3_std_error at e08a7808
#4 [d55f5f18] ext3_reserve_inode_write at e08a074a
#5 [d55f5f30] ext3_mark_inode_dirty at e08a076c
#6 [d55f5f4c] ext3_dirty_inode at e08a325e
#7 [d55f5f5c] __mark_inode_dirty at c049192e
#8 [d55f5f78] vfs_readdir at c0483adb
#9 [d55f5f94] sys_getdents64 at c0483b55
#10 [d55f5fb8] system_call at c0404f10
EAX: ffffffda EBX: 00000003 ECX: 0929637c EDX: 00008000
DS: 007b ESI: 0929637c ES: 007b EDI: 00744ff4
SS: 007b ESP: bf85e704 EBP: bf85e720
CS: 0073 EIP: 00a53410 ERR: 000000dc EFLAGS: 00000212
---
File /etc/fstab:
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
10.209.149.113:/data /data nfs defaults 1 0
10.209.149.108:/extra /extra nfs defaults 1 0
#/dev/sdb1 /sdb ext3 errors=panic 1 2
#/dev/sdc1 /sdb/mp ext3 errors=panic 1 2
Environment
- Red Hat Enterprise Linux 5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
