System was crashed with I/O error in Red Hat Enterprise Linux 5.

Solution Verified - Updated -

Issue

  • This is NOT AN ACTUAL PROBLEM!

  • Run maintenance script with iSCSI device, system crashed suddenly.

iSCSI device is always cut off from online system, it is connected maintenance mode only.

crash 4.0-8.9.1.el5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: /usr/lib/debug/lib/modules/2.6.18-164.el5PAE/vmlinux
    DUMPFILE: vmcore
        CPUS: 1
        DATE: Mon Jan 18 19:09:51 2010
      UPTIME: 01:22:56
LOAD AVERAGE: 3.55, 1.45, 0.56
       TASKS: 92
    NODENAME: 
     RELEASE: 2.6.18-164.el5PAE
     VERSION: #1 SMP Tue Aug 18 15:59:11 EDT 2009
     MACHINE: i686  (2332 Mhz)
      MEMORY: 512 MB
       PANIC: "Kernel panic - not syncing: EXT3-fs (device sdc1): panic forced after error"
         PID: 14995
     COMMAND: "ls"
        TASK: d65f1aa0  [THREAD_INFO: d55f5000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

>log
.
.
EXT3 FS on sdc1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: initialized (dev sdc1, type ext3), uses xattr
 connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4546303, last ping 4551303, now 4556304
 connection1:0: detected conn error (1011)
 session1: session recovery timed out after 120 secs
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9291
Buffer I/O error on device sdc1, logical block 4645
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 1035
Buffer I/O error on device sdc1, logical block 517
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 521
Buffer I/O error on device sdc1, logical block 260
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 261
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 3
Buffer I/O error on device sdc1, logical block 1
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9431
Buffer I/O error on device sdc1, logical block 4715
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9411
Buffer I/O error on device sdc1, logical block 4705
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 4706
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 4707
lost page write due to I/O error on sdc1
Buffer I/O error on device sdc1, logical block 4708
lost page write due to I/O error on sdc1
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 9427
Aborting journal on device sdc1.
sd 1:0:0:1: SCSI error: return code = 0x000f0000
end_request: I/O error, dev sdc, sector 1063
EXT3-fs error (device sdc1) in ext3_reserve_inode_write: Journal has aborted
Kernel panic - not syncing: EXT3-fs (device sdc1): panic forced after error
crash> bt
PID: 14995  TASK: d65f1aa0  CPU: 0   COMMAND: "ls"
 #0 [d55f5e78] crash_kexec at c0440ecf
 #1 [d55f5ebc] panic at c04244b9
 #2 [d55f5ed8] ext3_handle_error at e08a77aa
 #3 [d55f5eec] __ext3_std_error at e08a7808
 #4 [d55f5f18] ext3_reserve_inode_write at e08a074a
 #5 [d55f5f30] ext3_mark_inode_dirty at e08a076c
 #6 [d55f5f4c] ext3_dirty_inode at e08a325e
 #7 [d55f5f5c] __mark_inode_dirty at c049192e
 #8 [d55f5f78] vfs_readdir at c0483adb
 #9 [d55f5f94] sys_getdents64 at c0483b55
#10 [d55f5fb8] system_call at c0404f10
    EAX: ffffffda  EBX: 00000003  ECX: 0929637c  EDX: 00008000 
    DS:  007b      ESI: 0929637c  ES:  007b      EDI: 00744ff4
    SS:  007b      ESP: bf85e704  EBP: bf85e720
    CS:  0073      EIP: 00a53410  ERR: 000000dc  EFLAGS: 00000212 



---
File /etc/fstab:
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0
10.209.149.113:/data     /data               nfs     defaults      1 0
10.209.149.108:/extra      /extra               nfs     defaults      1 0
#/dev/sdb1          /sdb               ext3     errors=panic      1 2
#/dev/sdc1          /sdb/mp               ext3     errors=panic      1 2

Environment

  • Red Hat Enterprise Linux 5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.