Question -> shrinking non-root lvms, then grow root "/" lvm
I have two systems I recently loaded with RHEL 6.5 server. I realized afterwards, I did not provide enough space to the "/" file system, it's currently at 90% usage. I typically never run into this problem; I have to date never had to shrink/grow any LVM except in class because I have (until now) properly measured my disks, measuring twice, cutting once... So on the Windshield-Bug spectrum, I'm currently the bug in this round.
I found three other LVM file systems I could reduce:
/dev/mapper/disk0-tmp "/tmp" reduce by 8gb
/dev/mapper/disk0-var "/var' reduce by 4gb
/var/mapper/disk0-spool "/var/spool" reduce by 4gb
/dev/mapper/disk0-log "/var/log" reduce by 4gb
/dev/mapper/disk0-home "/home" reduce by 8gb
This would reclaim 28GB that I would want to allocate to /dev/mapper/disk0-slash "/"
I found the documentation to shrink the LVMs above at this link
I found the documentation to grow the LVM "/" at this link
And the last link I posted said if I make room elsewhere, I do not have to add a physical volume, but just use the unused space that I created as a result of shrinking the other volumes.
MY QUESTION:
When I grow the "/" LVM, do I have to do that in the single user mode since the system is using "/"
Can I grow the lvm /dev/mapper/disk0-slash while in run level 3? Are there any other steps I should take when growing the LVM for "/"?
Request help please...
Thanks
Responses
Guru
6863 points
PREFACE: when shrinking a filesytem/volume - SHRINK THE FILESYSTEM FIRST!!! Then the volume. (and backup your data, if it's important ;-)
You can grow the filesystem in multiuser if you have the space available in the VG. I, personally, use lvresize, but there is the doc for lvextend
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/lv_extend.html
Guru
6827 points
Thanks much James!
Guru
6863 points
For some reason... I didn't realize you were the OP. Ha!
I guess I forgot to mention that you need to lvresize then run resize2fs to "grow the FS in to the Volume". There is also a command resize4fs (part of e4fsprogs) if you are unable to grow your EXT4 Filesystems (I can't recall the specific error you receive, but it's not obvious that resize4fs would fix the issue.. but it does ;-)
Guru
6435 points
What I've found to be a safe rule of thumb for shrinking filesystems is to shrink the filesystem to slightly below the target size, shrink the volume to the desired size, the do a no arguments resize of the filesystem to grow it to the end of the shrunk volume. That way, you're ensured of leaving no orphan blocks.
Fun part of shrinking volumes in the root volume group is that, if you do it in multi-user, there's a strong chance that there will be writers to the filesystems. If you're in single for the purpose of shrinking other volumes, you may as well stay there for the extend of the volumes you're making room for.
Hi,
if you need to shrink FS and that FS is OS (system) FS based on LVM, then you need to umount those FS 1st.
Because i see that you've got OS FSs, you'll have to do the whole process while booted from boot DVD.
Growing FS based on LVM can be done online, so you can grow your "/" while your OS is booted and running.
From the FSs you have mentioned only /home can be unmounted while OS is up (you need to log off users 1st ofc).
Run fsck on unmounted FSs first before you start shrinking them.
Guru
6435 points
"/tmp" - frequently offline-able, even in full multi-user mode. That said, rather than reducing the size of the volume hosting "tmp", I'd nuke the volume, completely, and convert to using "/tmp" as tmpfs.
"/var" - usually offline-able, even in full multi-user mode.
"/var/spool" - absolutely offline-able, even in full multi-user mode.
"/var/log" - usually offline-able, even in full multi-user mode. Though, depending what hardening you've done, reducing the size of this partition may result in audit-shutdowns if you fill it up.
"/home" - absolutely offline-able, even in full multi-user mode. And, if you're in a multi-node environment, you want to consider offloading "/home" to an NFS (or even CIFS) server.
Only real problem to offlining any of the above filesystems is that you have to figure out what services you'll need to restart, afterwards. Usually, all it takes to offline any of these filesystems is a fuser -ck against the mount point then an unmount to free them up, then umount them (or even just do umount -f). Because of the impact to services while doing the offline operations, the only real reason to try to do your operations in full-multi is that you want to do things over the network rather than the console.
Guru
6827 points
Thanks everyone for the inputs
- Karel, what you say is what I had suspected. I was anticipating doing this while "/" lvm was not mounted, and the fsck against the file system prior seems like a proper idea, thanks
- Tom, I have a graphical interface that serves as doing this procedure at the console, without having to be at the console.
- James, thans for that info again!
Guru
6827 points
Update,
I found the following worked for a few of the file systems:
(I found in the man page for lvreduce that the "-r" argument "resizes the underlying filesystem together with the logical volume using fsadm")
$ echo log in as a user that does not have /home/yyyy as it's home drive
$ echo done on a freshly built virtual system, new file systems as a test
$ become root
# umount /home
# lvreduce -r /dev/disk0/home -L 4096M
# mount /home
#
# echo next one
# umount /tmp
# lvreduce -r /dev/disk0/tmp -L 8192M
# mount /tmp
#
# lvextend -r -l +100%FREE /dev/disk0/slash
# df -PhT | column -t
# echo worked as expected
The remaining /var file systems, I tested on a 'test victim' system and just 'commented out' /var and /var/tmp and /var/spool' and rebooted to init S via grub.
While the system did indeed complain about /var/ not being available, I was able to do the lvreduce and lvextend (lvextend against the same "/dev/disk0/slash" lvm above). I then removed the "#" characters from the /var mount points so it would mount /var/ and everything under it. That test system I did this on took this well enough... but I suspect there's a better way to deal with /var mountpoints for lvreduce and am going to research this more.
Guru
6435 points
Writers to the /var hierarchy will vary by how the system's used. The two most consistent writers into /var are syslog (which keeps /var/log active) and the audit daemon (which keeps /var/log/audit active). If you're running an SMTP service, that will tend to write to /var, but tends to do so without holding the filesystems open (opening and closing files in /var/spool/mail as it deposits new messages). Similarly, if using the RedHat Apache package, it will tend to be active against /var/www. There's a few other packages out there that interact with the /var hierarchy, but not to the degree that syslog and auditd do. The best way to identify what's using your /var (and related subdirs) is to use fuser.
All that aside, running without /var tends to be non-fatal - you just need to stop services so you can unmount /var (and the volumes you have under it).
Gotta say, though: having /var/tmp as it's own filesystem is a bit unusual; having /var/spool as its own filesystem is unusual if you're not running a mail server.
Guru
6863 points
I anticipate that as people start to embrace some of the published security standards you will see more environments using /var/tmp /tmp /var/log, etc... as a separate file systems.
Now this discussion made me wonder how you can unmount var (under any circumstance) if you have auditd enabled, as well as all of the files placed in /var/run/ and /var/cache.
Guru
6435 points
Environments I work in already do /var and /var/log/audit separately. With virtualization, there's enough benefits to running /tmp as a pseudo-filesystem that even that's pretty common, at this point: it's the provisioning-time norm on EL6-based deployments (whether templated or kicked) and the norm for EL5-based systems templated in the last three years.
Also, given the requirements to run A/V on even Linux, having /tmp run as a real filesystem is a major performance-drag.
Guru
6827 points
It is perhaps overkill, yet I've found conditions where it helped avert the /var/ file system being filled. That aside, some of it was security driven.
Guru
6827 points
Tom,
I believe someplace you had mentioned that XFS can not be reduced via 'lvmreduce' command or typical LVM reductions... I came across that fact with a RHEL 7 system very recently. This link and this link addresses it.
The first link above mentions the upstream provider's page on xfs and shrinking, which seems to infer they're at least exploring that as a future option to some degree.
Guru
6827 points
(I am the OP)
So I had to reinstall grub on one of the two systems I performed the LVM resizing operations (via command-line). Then I had to adjust the /boot/grub/grub.conf (more details below. What's odd, is prior to this the /boot/grub/grub.conf worked, but I had to change the (hd1,1 to (hd0,1)
When I finally rebooted this system that I performed this LVM reduction/extension on, it failed to boot. The Master Boot Record (MBR) was no longer present.
At this Red Hat documentation on basic steps to resolve a non-booting system, it mentions in this location: "36.1.1.1. Unable to Boot into Red Hat Enterprise Linux", in the second paragraph:
Another common problem occurs when
using a partitioning tool to resize
a partition...
-- However, the partitioning "tool" I used was command-line LVM commands. Somehow the MBR went away. The paragraph above seems to point to my LVM actions.
The systems came with 16 drive slots, and a pre-configured raid 5 with five disks in the first drive slot positions (I of course first wiped the pre-configured raid array completely). I added two disks as a mirror after that. This mirror is where the operating system resides, and I configured the raid so the 136GB mirror would be the boot device in the raid bios.
I did a grub-reinstall to /dev/sdb (large data raid in first position) and then oddly had to change grub (which was working) from (hd1,1) to (hd0,1). I just checked, and /boot is of course on sdb.
So after I ran (and am 100% positive I ran it as follows):
/sbin/grub-install /dev/sdb
and adjusted the current kernel stanza from (hd1,1) to (hd0,1), the system booted without complaint.
Moral of the story, measure twice, cut once with kickstarts and partition sizes - I typically do this. I've built hundreds of systems, yet this time I missed two. I'll probably also take any pre-built raid that comes from the factory and move it to the second position with an operating system mirrored disk preceding it in order, just for additional measure.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.