ext3_lookup finds an "unlinked inode" and calls ext3_abort which results in a read-only filesystem

Solution Verified - Updated -

Issue

  • The ext3-filesystem went read-only. In /var/log/messages the following errors are logged:
EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 91626760 in dir #19039379
Aborting journal on device dm-4.
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
  • A couple of days later (after restoring with as written in the resolution):
EXT3-fs error (device dm-4): ext3_free_blocks_sb: bit already cleared for block 43951098 
Aborting journal on device dm-4. 
EXT3-fs error (device dm-4) in ext3_free_blocks_sb: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_truncate: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_orphan_del: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted 
EXT3-fs error (device dm-4) in ext3_delete_inode: Journal has aborted 
ext3_abort called. 
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal 
Remounting filesystem read-only 
__journal_remove_journal_head: freeing b_committed_data 
  • nfs ext3 file system going read-only on two servers which are nfs cross mounted to each other.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339312 in dir #318520
Aug 16 11:57:17 server1 kernel: Aborting journal on device dm-10.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339315 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339293 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339321 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339323 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339276 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339322 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339291 in dir #318520
Aug 16 11:57:17 server1 kernel: ext3_abort called.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
Aug 16 11:57:17 server1 kernel: Remounting filesystem read-only
Aug 16 12:02:10 server1 multipathd: dm-10: umount map (uevent) 
Aug 16 12:02:11 server1 kernel: ext3_abort called.
Aug 16 12:02:11 server1 kernel: EXT3-fs error (device dm-10): ext3_put_super: Couldn't clean up the journal
Aug 16 12:03:55 server1 multipathd: dm-10: remove map (uevent) 
Aug 16 12:05:01 server1 multipathd: dm-10: add map (uevent) 
Aug 16 12:05:22 server1 kernel: kjournald starting.  Commit interval 5 seconds
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 16 12:05:22 server1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • EXT3 filesystem
  • Optionally exporting the ext3 filesystem over NFS (makes the issue more likely to happen)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In