How to repair filesystem in rescue environment for Red Hat Enterprise Linux?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 10
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5
  • Local Filesystems (Ext4 and XFS)
  • Rescue Environment

Issue

  • How to repair a local filesystem in the rescue environment for Red Hat Enterprise Linux?
  • A system will not load after boot, and it requests a repair of an OS-dependent filesystem, i.e., typically either / (root) or /var.
  • The / (root) filesystem is going into read-only mode.
  • How to repair the / (root) or /var filesystem (either Ext4 or XFS)?

Resolution

  1. Boot the system from the binary DVD or boot disc1 of the same major release as the system. If possible, download the latest minor release available from the Downloads page at access.redhat.com. For example, it would be better to use RHEL 8.10 as opposed to RHEL 8.0 because the former will likely contain more updates/fixes than the previous minor release.

    Once the system has successfully booted from the ISO image and Red Hat Enterprise Linux boot screen will appear.

  • For RHEL7, 8, 9 and 10: select Troubleshooting, then Rescue a Red Hat Enterprise Linux system. Then go to Step 4. Below example is for RHEL 7 but applicable to RHEL 8, 9 and 10 as well.

    ~~~
Install Red Hat Enterprise Linux 7.0
Test this media & install Red Hat Enterprise Linux 7.0
Troubleshooting                                    <--- Select
~~~

~~~
Install Red Hat Enterprise Linux 7.0 in basic graphics mod
Rescue a Red Hat Enterprise Linux system          <--- Select
Run a memory test

Boot from local drive

Return to main menu
~~~

    • For RHEL6: select Rescue installed system. Then go to Step 2.

      Install or upgrade an existing system
Install system with basic cideo driver
Rescue installed system                            <--- Select
Boot from local drive
Memory test

    • For RHEL 5: Type: linux rescue without the quotes, and hit enter at the prompt. Then go to Step 2.

      [F1-Main] [F2-Options] [F3-General] [F4-Kernel] [F5-Rescue] 

boot: linux rescue

  1. When prompted for language, and keyboard, provide the pertinent information for the system.

  2. When prompted to enable the network devices on the system, select: No

  3. Select: Skip when prompted.

  4. If using Software RAID, Initialize the RAID array first.

    sh-4.2# mdadm --assemble --scan

  5. If using LVM, activate the volumes in order to scan them.

    sh-4.2# lvm vgchange -ay
  1 logical volume(s) in volume group "VolGroup00" now active

  6. Execute the check on the device which contains the filesystem.

    • For XFS filesystem
    sh-4.2# xfs_repair /dev/mapper/<vg>-<lv>
or
sh-4.2# xfs_repair /dev/<sd device>
or
sh-4.2# xfs_repair /dev/<md device>
    • For EXT filesystem
    sh-4.2# e2fsck -fvy /dev/mapper/<vg>-<lv>
or 
sh-4.2# e2fsck -fvy /dev/<sd device>
or
sh-4.2# e2fsck -fvy /dev/<md device>

    NOTE: You may have to recreate the log if xfs_repair will not run. This can be done by running xfs_repair -L.
    NOTE: Please make sure you have a known, good backup of the data present on the impacted filesystem before attempting a repair.

  7. Exit the rescue environment and boot the system normally.

    sh-4.2# exit

Root Cause

  • Filesystems can become corrupted for a variety of reasons, the most notable of which are:
    • Connection failure(s) during write().
    • Bad hardware (intermittent hardware failure).
    • Bad cables/fabric.
    • Power loss.
    • Faulty network connections.
    • Flapping on NIC.
    • Software/firmware bugs,
    • Incorrect file system resize operations, such as logical volume resizing.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments