ext3_lookup finds an "unlinked inode" and calls ext3_abort which results in a read-only filesystem

Solution Verified - Updated -

Issue

  • The ext3-filesystem went read-only. In /var/log/messages the following errors are logged:
EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 91626760 in dir #19039379
Aborting journal on device dm-4.
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
  • A couple of days later (after restoring with as written in the resolution):
EXT3-fs error (device dm-4): ext3_free_blocks_sb: bit already cleared for block 43951098 
Aborting journal on device dm-4. 
EXT3-fs error (device dm-4) in ext3_free_blocks_sb: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_truncate: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_orphan_del: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_delete_inode: Journal has aborted 
ext3_abort called. 
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal 
Remounting filesystem read-only 
__journal_remove_journal_head: freeing b_committed_data 
  • nfs ext3 file system going read-only on two servers which are nfs cross mounted to each other.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339312 in dir #318520
Aug 16 11:57:17 server1 kernel: Aborting journal on device dm-10.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339315 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339293 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339321 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339323 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339276 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339322 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339291 in dir #318520
Aug 16 11:57:17 server1 kernel: ext3_abort called.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
Aug 16 11:57:17 server1 kernel: Remounting filesystem read-only
Aug 16 12:02:10 server1 multipathd: dm-10: umount map (uevent) 
Aug 16 12:02:11 server1 kernel: ext3_abort called.
Aug 16 12:02:11 server1 kernel: EXT3-fs error (device dm-10): ext3_put_super: Couldn't clean up the journal
Aug 16 12:03:55 server1 multipathd: dm-10: remove map (uevent) 
Aug 16 12:05:01 server1 multipathd: dm-10: add map (uevent) 
Aug 16 12:05:22 server1 kernel: kjournald starting.  Commit interval 5 seconds
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 16 12:05:22 server1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • EXT3 filesystem
  • Optionally exporting the ext3 filesystem over NFS (makes the issue more likely to happen)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content