Moving /var/log folders to new partitions

Latest response

I had a RHEL 7 server initially installed with separate partitions configured - i.e. /, /home, /boot, /var, /tmp.

In response to a security audit, we were required to add separate partitions for /var/log, /var/tmp, and /var/log/audit.

I added some additional drive space to the server (the server is a VMware virtual machine), created a physical volume and logical group, and divided the logical group into 3 logical volumes, formatted the lv with xfs file system, and added the partitions to /etc/fstab.

When the system came back up, ssh didn't work anymore (the connection would immediately drop after entering the username and password), and much of the logging didn't work. I had assumed the different files would automatically be created (/var/log/messages, /var/log/secure, /var/log/maillog, /var/log/wtmp) but noticed I had to create most of them. Adding wtmp got the "last" command working, adding dmesg got dmesg working, but some logging never started, i.e. messages.

To get the system back to normal, I commented out the 3 lines in /etc/fstab, and ssh started working again, and the previous log entries were still there as well.

So, I'm assuming its possible to successfully add the additional partitions, but I'm missing some important details in the implementation.

Can anyone fill in the blanks here?

Thanks

Paul Greene

Responses

Paul,

You could do this with a cloned system in vmware if needed

In principle, (and short answer), one possible approach could be... Mount the new partitions as /var/log2 /var/tmp2 and /var/log/audit2 NOTE: keeping the original mounts mounted as normal!

Use rsync at rescue.target (or equivalent) to take data from the existing locations to the new locations.

Validate selinux contexts and resolve proper proper contexts for new files/directories, and also permissions etc.

chcon --reference /var/log/audit /var/log/audit2
chown --reference /var/log/audit /var/log/audit2
chmod --reference /var/log/audit /var/log/audit2
echo "and so forth, there are different methods, this is just one"
echo "and remember to do files too"

If you read some security "requirements", they will say they require a bind mount between /var/tmp and /tmp - and I'd recommend you see Tom Jones' input at this Red Hat Discussion https://access.redhat.com/discussions/1369623 first, before considering bind mount.

After the rsync, and other bits above, comment out (not delete) the fstab entries (in fact, make a backup copy of your /etc/fstab first), and then point your new storage partitions to the correct locations --after-- doing the above, and reboot.

Your mileage may vary, if needed, clone the system first and try it on that system with no network, make backups of any config files you change (/etc/fstab).

Write down what you did, if you're unfamiliar with this, get another person to witness the event as you transition as it sometimes helps to have another human around for "bread crumbs" (besides writing down things in a log or someplace).

Let us know how this goes, post back here. someone should answer...

Kind Regards

Sorry it took awhile to get some feedback. I was testing everything offline on a lab server before messing with the production server. The selinux contexts were the issue - I temporarily changed the mode from enforcing to permissive, and ssh worked correctly after that, so I duplicated the security contexts on all the files in /var/log and below, and ssh was fine. I'm still waiting to get feedback from the arcsight guys to make sure they're getting the log feeds they're supposed to get, but I'm assuming it should be good now. Thanks very much for your help. Paul

Glad you got it worked out.. SELINUX has been the cause of many a sleepless night for us all :)

Paul, in addition to what I wrote, see this Red Hat solution at https://access.redhat.com/solutions/67781 which addresses this directly.

ADDED: I highly recommend reading that especially because of the "sparse" file called "/var/log/lastlog" which often appears to be excessively huge, make sure to check that bit out.

second the selinux contexts.. sounds like its creating your headaches.

Good point Will -- Of course, one can [very] temporarily turn off selinux keeping this in mind to see if that was the issue then after proper diagnosis & resolution, turn it back on. (see example above for chcon I mentioned)

Ok, one question comes to mind using rsync (after reading that 67781 article). (I'm re-editing because my original wording seemed very clumsy) Would it be better to rsync just /var, and then all sub-folders would get populated? Or is it better to rsync each sub-folder separately? i.e. rsync /var, rsync /var/log, rsync /var/log/audit, and rsync /var/tmp independently?

I know I'm a bit old-school in how I do things, but for things like moving data around in ways that preserve things like extended attributes, I tend to be a fan of older tools like dump, dd or cpio.

That aside: if you're syncing from (in your case) /var to a set of filesystems that are overlap-mounted in another location (e.g., /mnt/var, /mnt/var/log, /mnt/var/log/audit), doing a single sync will typically put things into the appropriate places.

If you're able to boot off of alternate media (DVD-boot, net-boot, etc.) you'll tend to get more reliable results in your copy-actions (particularly given the nature of a /var-hierarchy on a normally-booted system).

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.