Recommendations and Best Practices when setting up a Logging Server?

Latest response

I'm going to build a new RHEL7 VM to off load logs from other RHEL VMs and EXSi hosts.

The previous RHEL VM had a directory at the root level called /logging, where I dumped various logs.

Had an issue with the RHEL VM earlier this year and had to open a case with Red Hat Support and they recommended to dump logs under /var/log instead.

Is there a right or wrong way to dump logs to another RHEL VM or a best practice that someone could recommend, that way I can try to set up the best way possible the 1st time?

thanks

Responses

Christopher,

Probably not the best person to answer, but I do have a couple of servers collecting RSyslog logs. I have a folder structure under /var/log/ sorted by year/month/day/system name/log files. I think the biggest reason for the /var/log suggestion is that is keeps the SELinux context straight without too many changes. It also defaults to needing permissions to access the logs. This helps if you have to keep logs for compliance reasons.

I don't really know a "right" way to store logs, I only know how I do it. And I may be doing it completely wrong, but it works for my needs.

Frank

Looking at system layout, I would also recommend that /var and /var/log/ be placed on its own file system. This way if there is a run away logging event the log server would not fold up under file system being too full.

The nice thing about many VM systems is that the file system usage for the VM does not have to be pre-allocated. So it is more than possible to provide the additional mount points and provide some elbow room for unexpected file system growth.

I have used similar layout that Christopher referenced in my last work location and that setup provides a very good method to age out data when necessary.

Hi Christopher,

a) if you want to keep logs locally, then it is always a good idea to separate the roles. I manage lot of critical servers and always insist on three file systems:

/var
/var/log
/var/log/audit

Sometimes even /var/tmp is separate.

b) If you want to centralize rsyslog on remote server, it is best to use a different file system on it and then create structure under it as you wish. For example, you might have a file system on remote rsyslog server named /rsyslog. The name is really arbitrary.

I have done various kinds of centralised rsyslog setups on remote servers. Very easy and useful.

Regards,

Dusan Baljevic (amateur radio VK2COT)

Hi Dusan, you are correct in that when I'm building out a new RHEL VM, I typically have the following partitions:

/var/
/var/log/
/var/log/audit 
/var/tmp (as recommended by my Red Hat Liaison)  

Are you creating the /rsyslog under /var/log or is that under / ?

As Frank Coons pointed out, best to place those logs under /var/log for SELinux reasons and also pointed out by Red Hat Support when I had an issue with the server.

And again, this server will have Rsyslog and LogWatch running on it.

thanks

to extend the comments about file system layout (/var & /var/log): another option is leave /var & /var/log in their "standard" configuration (which can vary - Red Hat devs. really seem to want /var on the same file system as / (root), but may of us for "best practices" reasons from past painful encounters have /var on a dedicated file system) - but reconfigure (r)syslog to write specific files to another mounted file system (I have a centralized syslog server that has rsyslog writing to /mnt/logs/, which is an enterprise-grade NAS serving NFS). Note that in my case, I also had to customize the systemd startup script for rsyslogd to have it require & start after the network-fs service.

Hi Christopher,

As other mentioned, and I confirmed, use separate file system under / so that you do not have dependency on other file systems.

I insist on separating O/S-related file systems from those used specifically for applications and monitoring like centralized rsyslog... This helps me survive in IT business for 34 years :)

Being of Montenegrin background, and supposedly the laziest people in the world, I let computers do my work as much as I can.

There is a rumour that Montenegrins need chair next to the bed in the bedroom. Why? So that they can rest after sleep...

Just a small joke at my own expense for Friday :)

Regards,

Dusan Baljevic (amateur radio VK2COT)