Reasoning for seperating /var from /

Latest response

There is a commonly held wisdom that /var should by default be seperated from the root partition (for example https://access.redhat.com/site/articles/10332).

While reviewing our current kickstart setup (seperate /, /var, /tmp and /home provided by NFS) I came to question why / was seperated from /var.

Asking around the building, the commonly stated reason tended to be that /var could easily be filled by a (possibly misbehaved) application, and that if it wasn't seperate from / the filling of / would cause a kernel would panic. Though, in my (limited) experience filling /, I haven't witnessed this - I have found references to this occuring in years long past in kernel and OS versions long out of use.

Supposing we had to make the choice between a 20GB /, or a 15GB / and 5GB /var, the latter case has the advantage of isolating applications filling / from interferring from /var or vis versa, but increases the chances that a filesystem will fill.

The seriousness of a filled /var is already quite substantial - systems will not function properly, I imagine / filling at the time would only be a small incremental danger. So it becomes a tradeoff, one big partition for / and /var lowers the chance that /var will fill due to applications writing to /var but increases the consequence of /var filling (being that / will fill at the same time.)

Is there some additional risk to / filling that I haven't observed? Is a full / any more likely to cause a kernel panic than a full /var? (If there is any kernel panic? All that came to mind was if device nodes couldn't be created, some issues could manifest - but if that were the only means of this rumoured kernel panic then perhaps we should be putting /dev on the seperate filesystem).

Naturally there are case specific reasons to split off /var (or a subset of /var) into its on file system, though in the general case I'm wondering what the reasons are.

Responses

In my recent experience I agree. Filling / does not cause a panic in 9 out of 10 cases. I think a sensitive application that comes to a screeching halt may inturn cause a kernel panaic. Just the other day a system filled / because something that writes to /var/log/spool/ created a mess. The remedy required removing the offending data and restarting an applications services and things were back to business as usual. Like I stated above it would depened on the sensitivity of the applications or services which this system or systems are running.

I believe you need to either have a set standard that you roll out. Which would combined or seperate filesystems. Or you have a partitioning scheme for critical servers in order to mitigate any potential downtime caused by anything writing to /var/log and the like. Proper log rotation can help greatly with this issue.

My 2cents..

David above has good things to say and I agree with pretty much everything.

I have not seen a system panic from a full rootfs in quite a while. I've never seen a system panic from a full /var (but as David says, it's all about the applications in use).

That said, var is much more susceptible to filling up -- by accident or by attack.

  • You have the various logging happening to /var/log/

    • Even in 2013 it's still common to see support cases where logs spiral out of control
  • There's also the oft-forgotten /var/tmp/ directory, which OFFICIALLY1 is expected to be world-writeable (sticky-bit'd) persistent storage -- i.e., storage available between reboots (in contrast to /tmp/ which doesn't have that requirement; indeed, in many modern Linux distros /tmp/ is an in-RAM tmpfs).

In short ....

"Is there some additional risk to / filling that I haven't observed?"

From what you said, no I don't think sooo, but that kind of depends on what you've observed.

"Is a full / any more likely to cause a kernel panic than a full /var?"

Absolutely I would say it's more likely that a full rootfs would cause a panic than a full var. The likelihood probably depends on what the kernel is doing though. :)

Our practice is to isolate /var/tmp & /var/log to independent LVs.

/var/tmp for reasons Ryan mentioned, and /var/log is actually to protect /var/log from running out of space due to something else in /var running wild. If we can't write to /var/log nothing works (such as logins), so we're protecting that.

Some additional advantages of a separate /var
you may deal with journaling differently, you may want to set nosuid - but the most important reason I like to separate file systems is so I don't get paged for a filesystem that is typically not our problem. /var is a bit different.

"Is there some additional risk to / filling that I haven't observed?"
If you fill up /var (versus a /var that is part of /), things that require files to be written to /tmp are not affected. I think we lose focus of how many things throw files in temp locations, until we run out of space ;-)

Primary benefit to partitionitis (as promulgated in things like the DISA STIGs), is the ability to set different (mostly security-related) mount options on each partition. Also means you can set different newfs options to better support a given file-allocation pattern (generally more beneficial to bitmapped filesystems than extent-based filesystems). Similarly, it means that you can use different fs-types optimized to a given partition's workload.

Partitionitis is a pain in the ass when you're using relatively static FS-types like are most-frequently available in Commercial Linux distros. Sucks to have to rebuild your root drive (or even just manually fix things when booting from rescue-mode) because your partitioning layout wasn't well-suited to the actual file distribution patterns of you partitioning-scheme.

Agree with all these points.

Ideally you should be mounting /var with noexec and nosuid flags.

We do the same in our environment with noexec and nosuid at kickstart

# add nosuid,nodev to fstab
/bin/cp /etc/fstab /etc/fstab.original
/bin/cat /etc/fstab | /usr/bin/perl -ne 's/defaults/nodev,nosuid/ if (/var|home|boot/); print;' > /tmp/fstab.new
/bin/mv /tmp/fstab.new /etc/fstab
/bin/chmod 644 /etc/fstab

Why bother with a %post action for that? Use "fsoptions", instead.

Good point

Agree with Tom re: fsoptions, and if you want to do it at command line (perhaps after the fact) this should achieve the same using only sed:

sed -i.original -r '/var|home|boot/ s/defaults/nodev,nosuid/' /etc/fstab

Thanks

A lot of good points above...

My two cents... In our environment, we make separate logical volumes for /tmp/ /var/ /var/log and /var/log/audit . probably overkill, and I'm good with that. And Tom Jones' point about complying with security principles is quite salient. I also put a separate partition on /tmp/ else I'm near guaranteed that certian groups or people will fill up / quickly.

In highly audited environments, I've seen consternation if /var/log/audit is not segregated from / .. We place /var/log/audit in it's own logical volume especially on tomcat or web servers.

I really disdain having to drop whatever I'm doing and fix a / partition that has been filled up. I build systems so that / is protected. Some servers have additional logical volumes as needed for unique data that might threaten "/".

With the generous space available on current drives we are receiving, it is trivial now to properly set up systems to partition space away from "/" to different logical volumes so that we don't have do firefighting on resolving a filled / partition, or get called in after-hours to resolve this. Another example, Tomcat content is often in /var/lib so sometimes we have needed a separate partition so tomcat is not contending for space with it's contents.

If you're working in a virtualized environment (or one where you're forced to run A/V on your Linux hosts), change /tmp to be tmpfs-based. Gives you your separation, improves performance of your host (and, where relevant, is easier on your virtual infrastructure). Just make sure that if you use tmpfs for /tmp that you set an appropriate "size=" option.

Note: if your users/tenants are used to being able to keep stuff in /tmp forever, there will be whining. And, if anyone's running poorly-written applications (using /tmp rather than /var/tmp for persistent storage of scratch data), there will also be whining.

Nice idea - thanks

No problem: this past year, we were forced to roll out McAfee's product onto every system. It killed our Linux systems until we started converting them to tmpfs. Had the nice side-benefit of also reducing pressure on our ESX-serving SAN arrays.

Thanks for that extra detail - that sounds good to know.

Tom,

Interested to know what performance benefits this is giving you specifically? especially "easier on your virtual infrastructure"

I can understand moving the filesystem off disk will reduce iops on the underlying disk, but I haven't seen a scenario where /tmp usage produces any significant IO... is this a byproduct of your antivirus software specifically (eg. it uses tmp to extract archives?)

In my experience /tmp is more abused than used by:
1. Software vendor installers extracting to, and executing installers from there
2. Users using it to stash useless files

Neither of which cause significant load on the filesystem.

Prior to the A/V mandate, our systems were built with /tmp placed onto an LV of the root disk. Then, last summer, a long-standing "A/V all the things" project came to a head. So, it was pushed out pell-mell.

...and became a problem, particularly on systems that do a lot of disk I/Os. Specifically, the A/V's on-access scanning components became problematic. The on-access scanning does frequent I/Os to the /tmp directory as it operates. When /tmp is disk-based, particularly on systems running applications that do frequent disk I/Os, the scanner processes are pretty much constantly a top-10 to top-5 entry in the process table. Moved to tmpfs, the scanner processes only infrequently crack the top 20. Moved to tmpfs, the amount of overall disk I/Os drop by at least half and traffic to the root devices drops to about what you'd expect for normal security- and audit-logging activities.

Presumably, some of this pain could possibly be addressed through better exclude list capabilities. Unfortunately, the Linux implementation of the A/V is kind of horrid. Doesn't seem to support stackable or client-specific rule-sets. In an environment of thousands of systems, you either have a miles-long exclude-list for each and every system or you have exclude-lists that are uselessly generic. The /tmp as tmpfs makes the symptoms of the A/Vs activities mostly disappear.

Thanks for the additional info.

Luckily the closest I have come to this pain is a mandate to install A/V on any Linux servers hosting Windows shares.

The fact this thing is even hitting tmp is a bit of a concern and appreciate your summary and details of workaround. If nothing else, this has convinced me that Linux A/V is just as horrid as it was last time I touched it.

How did you separate it? I have been trying to, but after running an install with the defaults I have only been able to separate /var/log and /var/log/audit from /. If i mount /var separately the splash screen never goes away.

So, I'm resurrecting this thread just to share my current experience. Due to some application requirements, I've got a request to move /home and /tmp onto /. Apparently, the application installer requires hardlinks between files in /tmp, /home and /. It's quite dreadful.

This is the stuff that pings my phone a 2 in the morning to fix... There's no amount of 'I told you so' that makes it ok.

:(

Err... That application is beyond badly designed. I mean, other than a one-off, personal-desktop kind of environment, /home is rarely on the same device as /tmp. Never mind the whole hardlinks in /tmp implying a permanence that's inconsistent with the reason for having /tmp in the first place.

There's a reason that the second verse of the old classic "The C Days of Y2K" is "/var is full".

Great discussion. Too many solutions with root filesystem in one partition. Is there a consensus for a General Purpose server layout? Is there a consensus on the fstab defaults nodev, noexec for /home, /var/*, /tmp/* (and anything exported via NFS) FWIW- had seen /var/cache/yum piling up 4 GB - any reason to tolerate this? https://access.redhat.com/solutions/974373 - seems like its to be allowed in RHEL6 I see same behave in my RHEL7.2 servers. Does satellite obviate this?
Setting up cron to clean seems a bit ugly.

@Dustin Trapani said: Due to some application requirements, I've got a request to move /home and /tmp onto /

I concur with @Tom Jones who said "that application is beyond badly designed". And that is why I've frequently told my engineers to engage me early when they start scoping a new app. If it breaks some basic understanding like this I've been known to show vendor application engineers to the door.

@John Westerdale said: Is there a consensus for a General Purpose server layout?

Apparently not. I work in an organization that run the entire spectrum from "everything on /" to separating out /var /var/db /var/log /var/cache /tmp, and a few I'm not remembering.

I for one still recommend separating out the volatile filesystems and protecting root. It is not always just about a kernel panic. I've had systems where a full / filesystem will hide problems until the next crash or boot, whichever happens first. Remember that there are things going on all the time that might try to rewrite a file in /etc. If / is full, that usually results in a zero byte file. How well will your system fare with a null /etc/passwd or /etc/shadow, or how about /etc/pam.d/system-auth-ac. There are files that have lower impact if they are rewritten to zero bytes, but really, why take the chance?

I put /var and /tmp separate filesystem and likely others depending on the role the machine is playing. I use tmpfs when possible, but when not I use a separate filesystem.

Hi Everyone, My 2 cents too

I think the same things like a lot of here.

I prefere too separate (and far away !) and it seems to be old fashion style but efficient. It's an additional security to production server. Additionnal options for such mount point (more precisely) and better to allow space according to needs. One bad design application doesn't destroy all or the server could become unstable and touch other applications or services for one which take its confort with storage...

Furthermore, When I see this configuration, I am happy, because I know somewhere that the administrator think to be better than a / (with no constraint or reflexion ?) and want to have the best for its system.

I like it with LVM of course...

Ex : /var/cache

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.