EXT4-fs error (device dm-50): ext4_ext_search_left

Solution Verified - Updated -

Issue

  • The following EXT4-fs errors are appearing within /var/log/messages:
Jul 27 11:09:24 localhost kernel: EXT4-fs error (device dm-50): ext4_ext_search_left: inode #443416591: (comm oracle) ix (4831232) != EXT_FIRST_INDEX (0) (depth 0)!
Jul 27 11:09:24 localhost kernel: EXT4-fs (dm-50): delayed block allocation failed for inode 443416591 at logical offset 4863944 with max blocks 32 with error -5
Jul 27 11:09:24 localhost kernel:
Jul 27 11:09:24 localhost kernel: This should not happen!!  Data will be lost
  • DB2 database failed with the following log:
var/log/messages:Jul 27 02:13:51 localhost kernel: EXT4-fs error (device dm-33): ext4_ext_search_left: inode #10485776: (comm db2sysc) ix (33841240) != EXT_FIRST_INDEX (33550664) (depth 2)!
var/log/messages:Jul 27 05:11:15 localhost kernel: EXT4-fs error (device dm-33): ext4_ext_search_left: inode #10485776: (comm db2sysc) ix (33841240) != EXT_FIRST_INDEX (33550664) (depth 2)!
  • Eventually, any access into the file system will hang necessitating a reboot.

Environment

  • Red Hat Enterprise Linux 5.8
  • Red Hat Enterprise Linux 6
  • Oracle data files stored on ext4 volume
  • DB2 data file stored on ext4 volumes.
  • Storage configuration
    • ext4 --> LVM striped logical volume --> multipath devices --> NETAPP LUNs
vg-vol dm-50 (253:50) (striped) [/mountpoint ext4 rw,barrier=1,data=ordered]
`- mpath9p1 dm-46 (253:46) (linear)
   `- mpath9 dm-37 (253:37) (multipath)
      `- sdck (69:128) [on PCI bus 40 device 00 irq 169] [] [5:0:0:20:] [5:0:0:20:]
      `- sddr (71:144) [on PCI bus 40 device 01 irq 177] [] [6:0:0:20:] [6:0:0:20:]
      `- sdbd (67:112) [on PCI bus 30 device 01 irq 154] [] [4:0:0:20:] [4:0:0:20:]
      `- sdw (65:96) [on PCI bus 30 device 00 irq 185] [] [3:0:0:20:] [3:0:0:20:]
`- mpath29p1 dm-45 (253:45) (linear)
   `- mpath29 dm-20 (253:20) (multipath)
      `- sdam (66:96) [on PCI bus 30 device 01 irq 154] [] [4:0:0:3:] [4:0:0:3:]
      `- sdf (8:80) [on PCI bus 30 device 00 irq 185] [] [3:0:0:3:] [3:0:0:3:]
      `- sdbt (68:112) [on PCI bus 40 device 00 irq 169] [] [5:0:0:3:] [5:0:0:3:]
      `- sdda (70:128) [on PCI bus 40 device 01 irq 177] [] [6:0:0:3:] [6:0:0:3:]
`- mpath27p1 dm-47 (253:47) (linear)
   `- mpath27 dm-18 (253:18) (multipath)
      `- sdak (66:64) [on PCI bus 30 device 01 irq 154] [] [4:0:0:1:] [4:0:0:1:]
      `- sdd (8:48) [on PCI bus 30 device 00 irq 185] [] [3:0:0:1:] [3:0:0:1:]
      `- sdbr (68:80) [on PCI bus 40 device 00 irq 169] [] [5:0:0:1:] [5:0:0:1:]
      `- sdcy (70:96) [on PCI bus 40 device 01 irq 177] [] [6:0:0:1:] [6:0:0:1:]
`- mpath7p1 dm-49 (253:49) (linear)
   `- mpath7 dm-36 (253:36) (multipath)
      `- sdcj (69:112) [on PCI bus 40 device 00 irq 169] [] [5:0:0:19:] [5:0:0:19:]
      `- sddq (71:128) [on PCI bus 40 device 01 irq 177] [] [6:0:0:19:] [6:0:0:19:]
      `- sdbc (67:96) [on PCI bus 30 device 01 irq 154] [] [4:0:0:19:] [4:0:0:19:]
      `- sdv (65:80) [on PCI bus 30 device 00 irq 185] [] [3:0:0:19:] [3:0:0:19:]
`- mpath30p1 dm-43 (253:43) (linear)
   `- mpath30 dm-21 (253:21) (multipath)
      `- sdan (66:112) [on PCI bus 30 device 01 irq 154] [] [4:0:0:4:] [4:0:0:4:]
      `- sdg (8:96) [on PCI bus 30 device 00 irq 185] [] [3:0:0:4:] [3:0:0:4:]
      `- sdbu (68:128) [on PCI bus 40 device 00 irq 169] [] [5:0:0:4:] [5:0:0:4:]
      `- sddb (70:144) [on PCI bus 40 device 01 irq 177] [] [6:0:0:4:] [6:0:0:4:]
`- mpath28p1 dm-48 (253:48) (linear)
   `- mpath28 dm-19 (253:19) (multipath)
      `- sdal (66:80) [on PCI bus 30 device 01 irq 154] [] [4:0:0:2:] [4:0:0:2:]
      `- sde (8:64) [on PCI bus 30 device 00 irq 185] [] [3:0:0:2:] [3:0:0:2:]
      `- sdbs (68:96) [on PCI bus 40 device 00 irq 169] [] [5:0:0:2:] [5:0:0:2:]
      `- sdcz (70:112) [on PCI bus 40 device 01 irq 177] [] [6:0:0:2:] [6:0:0:2:]
`- mpath5p1 dm-42 (253:42) (linear)
   `- mpath5 dm-35 (253:35) (multipath)
      `- sdci (69:96) [on PCI bus 40 device 00 irq 169] [] [5:0:0:18:] [5:0:0:18:]
      `- sddp (71:112) [on PCI bus 40 device 01 irq 177] [] [6:0:0:18:] [6:0:0:18:]
      `- sdbb (67:80) [on PCI bus 30 device 01 irq 154] [] [4:0:0:18:] [4:0:0:18:]
      `- sdu (65:64) [on PCI bus 30 device 00 irq 185] [] [3:0:0:18:] [3:0:0:18:]
`- mpath3p1 dm-44 (253:44) (linear)
   `- mpath3 dm-34 (253:34) (multipath)
      `- sdch (69:80) [on PCI bus 40 device 00 irq 169] [] [5:0:0:17:] [5:0:0:17:]
      `- sddo (71:96) [on PCI bus 40 device 01 irq 177] [] [6:0:0:17:] [6:0:0:17:]
      `- sdba (67:64) [on PCI bus 30 device 01 irq 154] [] [4:0:0:17:] [4:0:0:17:]
      `- sdt (65:48) [on PCI bus 30 device 00 irq 185] [] [3:0:0:17:] [3:0:0:17:]

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In