ext3_lookup finds an "unlinked inode" and calls ext3_abort which results in a read-only filesystem
Issue
- The ext3-filesystem went read-only. In
/var/log/messagesthe following errors are logged:
EXT3-fs error (device dm-4): ext3_lookup: unlinked inode 91626760 in dir #19039379
Aborting journal on device dm-4.
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
- A couple of days later (after restoring with as written in the resolution):
EXT3-fs error (device dm-4): ext3_free_blocks_sb: bit already cleared for block 43951098
Aborting journal on device dm-4.
EXT3-fs error (device dm-4) in ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_truncate: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_orphan_del: Journal has aborted
EXT3-fs error (device dm-4) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-4) in ext3_delete_inode: Journal has aborted
ext3_abort called.
EXT3-fs error (device dm-4): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
__journal_remove_journal_head: freeing b_committed_data
- nfs ext3 file system going read-only on two servers which are nfs cross mounted to each other.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339312 in dir #318520
Aug 16 11:57:17 server1 kernel: Aborting journal on device dm-10.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339315 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339293 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339321 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339323 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339276 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339322 in dir #318520
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_lookup: unlinked inode 2339291 in dir #318520
Aug 16 11:57:17 server1 kernel: ext3_abort called.
Aug 16 11:57:17 server1 kernel: EXT3-fs error (device dm-10): ext3_journal_start_sb: Detected aborted journal
Aug 16 11:57:17 server1 kernel: Remounting filesystem read-only
Aug 16 12:02:10 server1 multipathd: dm-10: umount map (uevent)
Aug 16 12:02:11 server1 kernel: ext3_abort called.
Aug 16 12:02:11 server1 kernel: EXT3-fs error (device dm-10): ext3_put_super: Couldn't clean up the journal
Aug 16 12:03:55 server1 multipathd: dm-10: remove map (uevent)
Aug 16 12:05:01 server1 multipathd: dm-10: add map (uevent)
Aug 16 12:05:22 server1 kernel: kjournald starting. Commit interval 5 seconds
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Aug 16 12:05:22 server1 kernel: EXT3-fs warning (device dm-10): ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug 16 12:05:22 server1 kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended
Environment
- Red Hat Enterprise Linux 5
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 7
- EXT3 filesystem
- Optionally exporting the ext3 filesystem over NFS (makes the issue more likely to happen)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.