Reasoning for seperating /var from /

Latest response

There is a commonly held wisdom that /var should by default be seperated from the root partition (for example https://access.redhat.com/site/articles/10332).

While reviewing our current kickstart setup (seperate /, /var, /tmp and /home provided by NFS) I came to question why / was seperated from /var.

Asking around the building, the commonly stated reason tended to be that /var could easily be filled by a (possibly misbehaved) application, and that if it wasn't seperate from / the filling of / would cause a kernel would panic. Though, in my (limited) experience filling /, I haven't witnessed this - I have found references to this occuring in years long past in kernel and OS versions long out of use.

Supposing we had to make the choice between a 20GB /, or a 15GB / and 5GB /var, the latter case has the advantage of isolating applications filling / from interferring from /var or vis versa, but increases the chances that a filesystem will fill.

The seriousness of a filled /var is already quite substantial - systems will not function properly, I imagine / filling at the time would only be a small incremental danger. So it becomes a tradeoff, one big partition for / and /var lowers the chance that /var will fill due to applications writing to /var but increases the consequence of /var filling (being that / will fill at the same time.)

Is there some additional risk to / filling that I haven't observed? Is a full / any more likely to cause a kernel panic than a full /var? (If there is any kernel panic? All that came to mind was if device nodes couldn't be created, some issues could manifest - but if that were the only means of this rumoured kernel panic then perhaps we should be putting /dev on the seperate filesystem).

Naturally there are case specific reasons to split off /var (or a subset of /var) into its on file system, though in the general case I'm wondering what the reasons are.

Responses

In my recent experience I agree. Filling / does not cause a panic in 9 out of 10 cases. I think a sensitive application that comes to a screeching halt may inturn cause a kernel panaic. Just the other day a system filled / because something that writes to /var/log/spool/ created a mess. The remedy required removing the offending data and restarting an applications services and things were back to business as usual. Like I stated above it would depened on the sensitivity of the applications or services which this system or systems are running.

I believe you need to either have a set standard that you roll out. Which would combined or seperate filesystems. Or you have a partitioning scheme for critical servers in order to mitigate any potential downtime caused by anything writing to /var/log and the like. Proper log rotation can help greatly with this issue.

My 2cents..

David above has good things to say and I agree with pretty much everything.

I have not seen a system panic from a full rootfs in quite a while. I've never seen a system panic from a full /var (but as David says, it's all about the applications in use).

That said, var is much more susceptible to filling up -- by accident or by attack.

  • You have the various logging happening to /var/log/

    • Even in 2013 it's still common to see support cases where logs spiral out of control
  • There's also the oft-forgotten /var/tmp/ directory, which OFFICIALLY1 is expected to be world-writeable (sticky-bit'd) persistent storage -- i.e., storage available between reboots (in contrast to /tmp/ which doesn't have that requirement; indeed, in many modern Linux distros /tmp/ is an in-RAM tmpfs).

In short ....

"Is there some additional risk to / filling that I haven't observed?"

From what you said, no I don't think sooo, but that kind of depends on what you've observed.

"Is a full / any more likely to cause a kernel panic than a full /var?"

Absolutely I would say it's more likely that a full rootfs would cause a panic than a full var. The likelihood probably depends on what the kernel is doing though. :)

Our practice is to isolate /var/tmp & /var/log to independent LVs.

/var/tmp for reasons Ryan mentioned, and /var/log is actually to protect /var/log from running out of space due to something else in /var running wild. If we can't write to /var/log nothing works (such as logins), so we're protecting that.

Some additional advantages of a separate /var
you may deal with journaling differently, you may want to set nosuid - but the most important reason I like to separate file systems is so I don't get paged for a filesystem that is typically not our problem. /var is a bit different.

"Is there some additional risk to / filling that I haven't observed?"
If you fill up /var (versus a /var that is part of /), things that require files to be written to /tmp are not affected. I think we lose focus of how many things throw files in temp locations, until we run out of space ;-)

Primary benefit to partitionitis (as promulgated in things like the DISA STIGs), is the ability to set different (mostly security-related) mount options on each partition. Also means you can set different newfs options to better support a given file-allocation pattern (generally more beneficial to bitmapped filesystems than extent-based filesystems). Similarly, it means that you can use different fs-types optimized to a given partition's workload.

Partitionitis is a pain in the ass when you're using relatively static FS-types like are most-frequently available in Commercial Linux distros. Sucks to have to rebuild your root drive (or even just manually fix things when booting from rescue-mode) because your partitioning layout wasn't well-suited to the actual file distribution patterns of you partitioning-scheme.

Agree with all these points.

Ideally you should be mounting /var with noexec and nosuid flags.

We do the same in our environment with noexec and nosuid at kickstart

# add nosuid,nodev to fstab
/bin/cp /etc/fstab /etc/fstab.original
/bin/cat /etc/fstab | /usr/bin/perl -ne 's/defaults/nodev,nosuid/ if (/var|home|boot/); print;' > /tmp/fstab.new
/bin/mv /tmp/fstab.new /etc/fstab
/bin/chmod 644 /etc/fstab

Why bother with a %post action for that? Use "fsoptions", instead.

Good point

Agree with Tom re: fsoptions, and if you want to do it at command line (perhaps after the fact) this should achieve the same using only sed:

sed -i.original -r '/var|home|boot/ s/defaults/nodev,nosuid/' /etc/fstab

Thanks

A lot of good points above...

My two cents... In our environment, we make separate logical volumes for /tmp/ /var/ /var/log and /var/log/audit . probably overkill, and I'm good with that. And Tom Jones' point about complying with security principles is quite salient. I also put a separate partition on /tmp/ else I'm near guaranteed that certian groups or people will fill up / quickly.

In highly audited environments, I've seen consternation if /var/log/audit is not segregated from / .. We place /var/log/audit in it's own logical volume especially on tomcat or web servers.

I really disdain having to drop whatever I'm doing and fix a / partition that has been filled up. I build systems so that / is protected. Some servers have additional logical volumes as needed for unique data that might threaten "/".

With the generous space available on current drives we are receiving, it is trivial now to properly set up systems to partition space away from "/" to different logical volumes so that we don't have do firefighting on resolving a filled / partition, or get called in after-hours to resolve this. Another example, Tomcat content is often in /var/lib so sometimes we have needed a separate partition so tomcat is not contending for space with it's contents.

If you're working in a virtualized environment (or one where you're forced to run A/V on your Linux hosts), change /tmp to be tmpfs-based. Gives you your separation, improves performance of your host (and, where relevant, is easier on your virtual infrastructure). Just make sure that if you use tmpfs for /tmp that you set an appropriate "size=" option.

Note: if your users/tenants are used to being able to keep stuff in /tmp forever, there will be whining. And, if anyone's running poorly-written applications (using /tmp rather than /var/tmp for persistent storage of scratch data), there will also be whining.

Nice idea - thanks

No problem: this past year, we were forced to roll out McAfee's product onto every system. It killed our Linux systems until we started converting them to tmpfs. Had the nice side-benefit of also reducing pressure on our ESX-serving SAN arrays.

Thanks for that extra detail - that sounds good to know.

Tom,

Interested to know what performance benefits this is giving you specifically? especially "easier on your virtual infrastructure"

I can understand moving the filesystem off disk will reduce iops on the underlying disk, but I haven't seen a scenario where /tmp usage produces any significant IO... is this a byproduct of your antivirus software specifically (eg. it uses tmp to extract archives?)

In my experience /tmp is more abused than used by:
1. Software vendor installers extracting to, and executing installers from there
2. Users using it to stash useless files

Neither of which cause significant load on the filesystem.

Prior to the A/V mandate, our systems were built with /tmp placed onto an LV of the root disk. Then, last summer, a long-standing "A/V all the things" project came to a head. So, it was pushed out pell-mell.

...and became a problem, particularly on systems that do a lot of disk I/Os. Specifically, the A/V's on-access scanning components became problematic. The on-access scanning does frequent I/Os to the /tmp directory as it operates. When /tmp is disk-based, particularly on systems running applications that do frequent disk I/Os, the scanner processes are pretty much constantly a top-10 to top-5 entry in the process table. Moved to tmpfs, the scanner processes only infrequently crack the top 20. Moved to tmpfs, the amount of overall disk I/Os drop by at least half and traffic to the root devices drops to about what you'd expect for normal security- and audit-logging activities.

Presumably, some of this pain could possibly be addressed through better exclude list capabilities. Unfortunately, the Linux implementation of the A/V is kind of horrid. Doesn't seem to support stackable or client-specific rule-sets. In an environment of thousands of systems, you either have a miles-long exclude-list for each and every system or you have exclude-lists that are uselessly generic. The /tmp as tmpfs makes the symptoms of the A/Vs activities mostly disappear.

Thanks for the additional info.

Luckily the closest I have come to this pain is a mandate to install A/V on any Linux servers hosting Windows shares.

The fact this thing is even hitting tmp is a bit of a concern and appreciate your summary and details of workaround. If nothing else, this has convinced me that Linux A/V is just as horrid as it was last time I touched it.

How did you separate it? I have been trying to, but after running an install with the defaults I have only been able to separate /var/log and /var/log/audit from /. If i mount /var separately the splash screen never goes away.

So, I'm resurrecting this thread just to share my current experience. Due to some application requirements, I've got a request to move /home and /tmp onto /. Apparently, the application installer requires hardlinks between files in /tmp, /home and /. It's quite dreadful.

This is the stuff that pings my phone a 2 in the morning to fix... There's no amount of 'I told you so' that makes it ok.

:(

Err... That application is beyond badly designed. I mean, other than a one-off, personal-desktop kind of environment, /home is rarely on the same device as /tmp. Never mind the whole hardlinks in /tmp implying a permanence that's inconsistent with the reason for having /tmp in the first place.

There's a reason that the second verse of the old classic "The C Days of Y2K" is "/var is full".

Great discussion. Too many solutions with root filesystem in one partition. Is there a consensus for a General Purpose server layout? Is there a consensus on the fstab defaults nodev, noexec for /home, /var/*, /tmp/* (and anything exported via NFS) FWIW- had seen /var/cache/yum piling up 4 GB - any reason to tolerate this? https://access.redhat.com/solutions/974373 - seems like its to be allowed in RHEL6 I see same behave in my RHEL7.2 servers. Does satellite obviate this?
Setting up cron to clean seems a bit ugly.

@Dustin Trapani said: Due to some application requirements, I've got a request to move /home and /tmp onto /

I concur with @Tom Jones who said "that application is beyond badly designed". And that is why I've frequently told my engineers to engage me early when they start scoping a new app. If it breaks some basic understanding like this I've been known to show vendor application engineers to the door.

@John Westerdale said: Is there a consensus for a General Purpose server layout?

Apparently not. I work in an organization that run the entire spectrum from "everything on /" to separating out /var /var/db /var/log /var/cache /tmp, and a few I'm not remembering.

I for one still recommend separating out the volatile filesystems and protecting root. It is not always just about a kernel panic. I've had systems where a full / filesystem will hide problems until the next crash or boot, whichever happens first. Remember that there are things going on all the time that might try to rewrite a file in /etc. If / is full, that usually results in a zero byte file. How well will your system fare with a null /etc/passwd or /etc/shadow, or how about /etc/pam.d/system-auth-ac. There are files that have lower impact if they are rewritten to zero bytes, but really, why take the chance?

I put /var and /tmp separate filesystem and likely others depending on the role the machine is playing. I use tmpfs when possible, but when not I use a separate filesystem.

Hi Everyone, My 2 cents too

I think the same things like a lot of here.

I prefere too separate (and far away !) and it seems to be old fashion style but efficient. It's an additional security to production server. Additionnal options for such mount point (more precisely) and better to allow space according to needs. One bad design application doesn't destroy all or the server could become unstable and touch other applications or services for one which take its confort with storage...

Furthermore, When I see this configuration, I am happy, because I know somewhere that the administrator think to be better than a / (with no constraint or reflexion ?) and want to have the best for its system.

I like it with LVM of course...

Ex : /var/cache

Aren't we ignoring monitoring on the filesystems? Monitoring is the major tool in handling a file system filling up. A unitary slash server will have more space available when monitoring detects a problem brewing. Also, for todays small vm servers, partitioning out so many file systems will double your disk needs . ex. a 30 GB server using 10GB for OS and app leaves 20 GB free for the various possibles, whereas giving 5GB each to /tmp /var/tmp /var/tmp/audit /var/log and 10GB to /home (for those killer ISOs) just doubled your server disk and lowered your security buffer from 20 GB to 5GB in most cases. You also doubled your backup sizes on image backups.... all to handle fears about yesterday's monolithic servers. If you are still worrying, add back half the disk you save by going to unitary slash and set your monitoring to a higher percent free. Yes, there are situations separate filesystems pay off but for most cases, monitoring and a good buffer fits the bill: catch it before it happens.

Meh... If you have a modern storage subsystem like ZFS, you can have your separate "filesystem" while still retaining the usage-flexibility of a /+swap layout. Sadly, neither ZFS nor BTRFS never made it in RHEL-land (let alone as root-compatible filesystems).

As to aggressive utilization-monitoring... If you're building ephemeral systems, if the disk fills up and the system falls over, your fault-detector nukes the system and rebuilds it (that's assuming it lives long enough to fill up).

Great point Will on monitoring.

I completely agree with and support (and we are using) monitoring. Some companies, and other entities are mandated to use separate partitions. While we use monitoring (one customer uses zenoss, for example), and it is wonderful to inform the appropriate teams to take action, sometimes things fall in the cracks. Yet another example, with docker being used heavily, some who request servers forget to note their use of docker which fills up /var/lib if not constrained. Yes, that should be taken care of in engineering, agreed. But nothing ever goes wrong in engineering, and all factors are always vetted prior to deployment, right???

Again, we wholeheartedly support and use monitoring of one form or another. Yet separating specific partitions where it is sane (that's the qualifier, and it might differ slightly from place to place) helps a lot.

Regards

RJ

Hi,

Good points were raised by many esteemed colleagues in the forum.

Whenever I can, I insist on having separate file systems, like:

/var /var/tmp /var/log /var/log/audit

So far, it served me well.

a) No need to worry about, say /home, wasting all the space in root file system. By separating file systems, one delegates maintenance duties to relevant teams.

Files in /home are certainly not sysadmin's job to clean up.

b) Separation of file systems gives easy visual presentation of where the files are located and which directory/file system is requiring attention due to shortage of available space.

c) Separation of file systems provides simpler creation and maintenance of LVM snapshots.

d) Separation of file systems enforces rule of "not putting all eggs in one basket".

e) Separation of file systems provides granular maintenance of permissions and ACLs:

noatime nodev nosuid noexec and so on...

f) Separation of file systems provides simpler administration of tape/disk-based backups (easy to exclude files and directories).

g) Separation of file systems ensures less escalations to sysadmins due to space problems.

If there is only a root file system (or similar), any problem in any directory is sent to sysadmins.

Hands up if anyone likes to be waken up at night because of disk issue in, say /home?

h) Inode management in file systems. Some file systems, like ext3 and ext4, require that sysadmin defines inode numbers BEFORE creation of the file systems.

If there are too many files in, say /home, it should not affect root file system.

i) Separation of file systems ensures easier management of hard links (only possible within the same file system).

j) Separation of file systems ensures easier management of file system checks. For example, one could decide not to worry about fsck-ing /var/tmp at boot time...

k) and more :)

Regards,

Dusan Baljevic (amateur radio VK2COT)

Resurrecting this for visibility as of 2021.04.13

Almost 8 years must be a record thread revival.. what are you all still doing here? :D

One of those times I post something like:

Hi RJ!

Yes, good to reactivate it.

I still strongly stand by the statements made before. In critical environments, those rules saved me lot of times.

In fact, when I do health checks of any Linux server (I wrote a script to do it before and after any change is made, or simply to check status of servers). Here is how I pick up on servers that do not comply (example taken from a test server which shows two missing file systems):

Some system files and directories have wrong ownership or permissions  WARNING 
RHEL repositories /etc/yum.repos.d/redhat.repo  enabled - check if they should be enabled permanently  CRITICAL 
SElinux not enabled in enforcing mode  WARNING 
FIPS is not enabled  WARNING 
There are systemd services in failed state  WARNING 
NetworkManager services in not active  WARNING 
Email Services - Postfix process not running  WARNING 
pwck not clear of errors  WARNING 
PAM enabled "nullok" option in /etc/pam.d configs  WARNING 
PAM enabled "log_passwd" option in /etc/pam.d configs  WARNING 
PAM "wheel" group not enabled in /etc/pam.d/su  WARNING 
/boot/efi is not a separate file system  WARNING 
/var/tmp is not a separate file system  WARNING 
Unreferenced files found  WARNING 
SUDO config errors and warnings found  WARNING 
SUDO config /etc/sudoers does not include support for drop-in files (default is /etc/sudoers.d)  WARNING 
Rsyslog has config errors  WARNING 
dmesg errors and warnings logged  WARNING 
/etc/cron.allow not enabled  WARNING 
Some broken symbolic links found  WARNING 
Module signing verification is not enabled in running kernel  WARNING

Regards,

Dusan Baljevic (amateur radio VK2COT)

And thanks for your great updates/inputs Dusan!

Regards
RJ