Filesystems not mounting at boot, getting "login incorrect"
This is a RHEL 5.4 system that's worked perfectly fine for more than 2 years. It's on an HP Proliant DL180G6 system with SmartArray controller and a couple terrabytes of array size.
Upon the last reboot, I tried to login and get the dreaded "login incorrect" before any password prompt, even for root. The login prompt even has the blank parentheses "()" which means the system isn't all there... filesystems such as /usr and /var do not look to be getting mounted.
During the boot, toward the end before it clears the screen for the login prompt, I can see a message that says something to the effect of "no fstab.sys, mounting internal defaults".
I can boot from the RHEL 5.4 install DVD and use the linux rescue mode and that will find and mount all my filesystems just fine. The /etc/fstab file is present and it looks just fine as well. I've run fsck on all partitions/filesystems and they check out fine too.
If I boot the system in single user mode, or boot with "init=/bin/bash" I can then log on as root, but a "df -k" shows all the filesystems apparently actually getting mounted to the root partition on the array since they all show the same exact size as the root partition.
I thought this might be a broken initrd file in /boot, but I recreated that using mkinitrd and instructions from one of the various "rebuild your initrd" websites, but that didn't fix of change anything... still getting same results from new or old initrd file.
I'm at a loss how to proceed, except for reloading the O/S from scratch and restoring the system from backup tapes, which will take all day long, possible two days to finish.
Any ideas how to fix this? I'm pretty sure all my data on all my filesystems is still there and in good shape, just something broken in whatever Linux is doing during bootup and failing to correctly mount all its filesystems onto the proper hardware partitions.
I'm not using LVM at all, just good old fashioned multiple partitions sliced out of the HP SmartArray's disk arrays.
I've thought about trying to do an RHEL version upgrade over the top of the existing installed system (e.g. and in-place upgrade of 5.5 over the existing 5.4) since rescue mode booting from a DVD disk can see all the partitions and their filesystems just fine.
Anybody know if that would work?