EXT4-fs error (device dm-50): ext4_ext_search_left
Issue
- The following EXT4-fs errors are appearing within /var/log/messages:
Jul 27 11:09:24 localhost kernel: EXT4-fs error (device dm-50): ext4_ext_search_left: inode #443416591: (comm oracle) ix (4831232) != EXT_FIRST_INDEX (0) (depth 0)!
Jul 27 11:09:24 localhost kernel: EXT4-fs (dm-50): delayed block allocation failed for inode 443416591 at logical offset 4863944 with max blocks 32 with error -5
Jul 27 11:09:24 localhost kernel:
Jul 27 11:09:24 localhost kernel: This should not happen!! Data will be lost
- DB2 database failed with the following log:
var/log/messages:Jul 27 02:13:51 localhost kernel: EXT4-fs error (device dm-33): ext4_ext_search_left: inode #10485776: (comm db2sysc) ix (33841240) != EXT_FIRST_INDEX (33550664) (depth 2)!
var/log/messages:Jul 27 05:11:15 localhost kernel: EXT4-fs error (device dm-33): ext4_ext_search_left: inode #10485776: (comm db2sysc) ix (33841240) != EXT_FIRST_INDEX (33550664) (depth 2)!
- Eventually, any access into the file system will hang necessitating a reboot.
Environment
- Red Hat Enterprise Linux 5.8
- Red Hat Enterprise Linux 6
- Oracle data files stored on ext4 volume
- DB2 data file stored on ext4 volumes.
- Storage configuration
- ext4 --> LVM striped logical volume --> multipath devices --> NETAPP LUNs
vg-vol dm-50 (253:50) (striped) [/mountpoint ext4 rw,barrier=1,data=ordered]
`- mpath9p1 dm-46 (253:46) (linear)
`- mpath9 dm-37 (253:37) (multipath)
`- sdck (69:128) [on PCI bus 40 device 00 irq 169] [] [5:0:0:20:] [5:0:0:20:]
`- sddr (71:144) [on PCI bus 40 device 01 irq 177] [] [6:0:0:20:] [6:0:0:20:]
`- sdbd (67:112) [on PCI bus 30 device 01 irq 154] [] [4:0:0:20:] [4:0:0:20:]
`- sdw (65:96) [on PCI bus 30 device 00 irq 185] [] [3:0:0:20:] [3:0:0:20:]
`- mpath29p1 dm-45 (253:45) (linear)
`- mpath29 dm-20 (253:20) (multipath)
`- sdam (66:96) [on PCI bus 30 device 01 irq 154] [] [4:0:0:3:] [4:0:0:3:]
`- sdf (8:80) [on PCI bus 30 device 00 irq 185] [] [3:0:0:3:] [3:0:0:3:]
`- sdbt (68:112) [on PCI bus 40 device 00 irq 169] [] [5:0:0:3:] [5:0:0:3:]
`- sdda (70:128) [on PCI bus 40 device 01 irq 177] [] [6:0:0:3:] [6:0:0:3:]
`- mpath27p1 dm-47 (253:47) (linear)
`- mpath27 dm-18 (253:18) (multipath)
`- sdak (66:64) [on PCI bus 30 device 01 irq 154] [] [4:0:0:1:] [4:0:0:1:]
`- sdd (8:48) [on PCI bus 30 device 00 irq 185] [] [3:0:0:1:] [3:0:0:1:]
`- sdbr (68:80) [on PCI bus 40 device 00 irq 169] [] [5:0:0:1:] [5:0:0:1:]
`- sdcy (70:96) [on PCI bus 40 device 01 irq 177] [] [6:0:0:1:] [6:0:0:1:]
`- mpath7p1 dm-49 (253:49) (linear)
`- mpath7 dm-36 (253:36) (multipath)
`- sdcj (69:112) [on PCI bus 40 device 00 irq 169] [] [5:0:0:19:] [5:0:0:19:]
`- sddq (71:128) [on PCI bus 40 device 01 irq 177] [] [6:0:0:19:] [6:0:0:19:]
`- sdbc (67:96) [on PCI bus 30 device 01 irq 154] [] [4:0:0:19:] [4:0:0:19:]
`- sdv (65:80) [on PCI bus 30 device 00 irq 185] [] [3:0:0:19:] [3:0:0:19:]
`- mpath30p1 dm-43 (253:43) (linear)
`- mpath30 dm-21 (253:21) (multipath)
`- sdan (66:112) [on PCI bus 30 device 01 irq 154] [] [4:0:0:4:] [4:0:0:4:]
`- sdg (8:96) [on PCI bus 30 device 00 irq 185] [] [3:0:0:4:] [3:0:0:4:]
`- sdbu (68:128) [on PCI bus 40 device 00 irq 169] [] [5:0:0:4:] [5:0:0:4:]
`- sddb (70:144) [on PCI bus 40 device 01 irq 177] [] [6:0:0:4:] [6:0:0:4:]
`- mpath28p1 dm-48 (253:48) (linear)
`- mpath28 dm-19 (253:19) (multipath)
`- sdal (66:80) [on PCI bus 30 device 01 irq 154] [] [4:0:0:2:] [4:0:0:2:]
`- sde (8:64) [on PCI bus 30 device 00 irq 185] [] [3:0:0:2:] [3:0:0:2:]
`- sdbs (68:96) [on PCI bus 40 device 00 irq 169] [] [5:0:0:2:] [5:0:0:2:]
`- sdcz (70:112) [on PCI bus 40 device 01 irq 177] [] [6:0:0:2:] [6:0:0:2:]
`- mpath5p1 dm-42 (253:42) (linear)
`- mpath5 dm-35 (253:35) (multipath)
`- sdci (69:96) [on PCI bus 40 device 00 irq 169] [] [5:0:0:18:] [5:0:0:18:]
`- sddp (71:112) [on PCI bus 40 device 01 irq 177] [] [6:0:0:18:] [6:0:0:18:]
`- sdbb (67:80) [on PCI bus 30 device 01 irq 154] [] [4:0:0:18:] [4:0:0:18:]
`- sdu (65:64) [on PCI bus 30 device 00 irq 185] [] [3:0:0:18:] [3:0:0:18:]
`- mpath3p1 dm-44 (253:44) (linear)
`- mpath3 dm-34 (253:34) (multipath)
`- sdch (69:80) [on PCI bus 40 device 00 irq 169] [] [5:0:0:17:] [5:0:0:17:]
`- sddo (71:96) [on PCI bus 40 device 01 irq 177] [] [6:0:0:17:] [6:0:0:17:]
`- sdba (67:64) [on PCI bus 30 device 01 irq 154] [] [4:0:0:17:] [4:0:0:17:]
`- sdt (65:48) [on PCI bus 30 device 00 irq 185] [] [3:0:0:17:] [3:0:0:17:]
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
