[INFO] What package, script or process is creating /etc/cron.daily/auditd.cron?

Latest response

PREFACE (EDIT): Please keep in mind 2 things before posting, as they have been visited several times and are not part of our issue.

1) Before we posted this, we had already done the following RPM commands: -qf (RPM [what]provides), -q --scripts (RPM scripts), -q --triggers (RPM triggers). We are trying to figure out what process is copying the file /usr/share/doc/audit-*/auditd.cron to /etc/cron.daily. It seems to be a post-installation process not in any RPM, file, script or trigger.

2) We are not using logrotate for audit, period. auditd.cron and logrotate are for different sets of files, as part of a greater solution reduce disk usage for logging and auditing. We use logrotate and its inherent (zip -9) compression for those files in logrotate which, again, does not include auditd's logs. We are also using other scripts -- cron.hourly/zlogs.cron (ORIGINAL) and cron.monthly/arclogs.cron (NEW) -- to rename/compress audit and other log files as well as move them to another file system. The latter, normally run monthly, even gets automatically kicked off, per audit/auditd.conf, when the file system drops to under 0.5GiB free.

LONGER INQUIRY (ORIGINAL): We are building an RPM to modify/rotate/compress logs on RHEL 6.

This includes the logrotate and auditd services. We wish to move /etc/cron.daily/logrotate and /etc/cron.daily/auditd.cron to /etc/cron.hourly/, so it rotates hourly. As part of our RPM SPEC, we are going to build a trigger that will re-move these files to /etc/cron.hourly/. in case they are updated in /etc/cron.daily/.

Although the package "logrotate" includes the /etc/cron.daily/logrotate file, the /etc/cron.daily/auditd.cron file is neither part of the package "audit" nor created as part of the scripts of package "audit". Using "rpm -q --whatprovides /etc/cron.daily/auditd.cron" and "rpm -qa --scripts" we are unable to determine how it is created at RPM install-time, or by any RPM/YUM action.

Searching Bugzilla only results in bz#811588 ( https://bugzilla.redhat.com/show_bug.cgi?id=811588 ), which shows this file is coped from /usr/share/doc/audit-*/, but not when this step occurs. Please advise on how this file is created, and how we may handle whenever it could be updated.

Something always creates /etc/cron.daily/auditd.conf, but it is neither a RPM managed file nor created as part of a RPM script, from what we can tell.

UPDATE: After several Kickstart tests and an overnight burn-in, I cannot track down what is copying this file from /usr/share/doc/audit-*/. I believe it may not be a Red Hat installed process, facility or other solution, but a site-specific process.

Responses

A symlink comes comes to mind;

%> ln -s /etc/cron.daily/logrotate /etc/cron.hourly/logrotate
%> ln -s /etc/cron.daily/auditd.cron /etc/cron.hourly/auditd.cron

That will keep your file up to date with changes on new/upgrades of logrotate &/or auditd.

In regards to your log rotation tool, why not just use logrotated and add your custom logs to the syslog facility?

Jason --

Regarding symlinks ...

I am concerned that it will run twice, hourly at 03:01 [A] and again at 03:05 [B]. If that happens, and the compression takes 4+ minutes, then I have to start adding lock files, calculating time differential from lockfile, etc... It's going to get complicated really quick.

Right now it's taken only 15s/GiB, so we're usually finishing well under 1 minute. And the reason we're doing it every hour is so we're only taking 15-30s/hour to compress, and saving 97% of disk space (gzip -9, just like logrotate offers to do with compression). After 24 hours, we could be seeing 50GiB+, and that will take 15 minutes to compress, not to mention we'll probably fill up our /var/log[/audit] by then.

Regarding why not just use logrotate ...

While we did modify /etc/logrotate.conf to use (logfile).%s, that still doesn't address ... - auditd (most specifically) and a few other programs that work differently and cannot just be added to /etc/logrotate.d/ - programs that don't compress, and generate .1, -2, _3 and similar, and then we cannot compress them because .1.gz, -2.gz, _3.gz already exist. - systems where we are not allowed to modify /etc/logrotate.conf

In that case, dropping in this /etc/cron.hourly/zlogs.cron script is a nice, even if partially redundant in some cases, solution. Understand some of our systems are generating GiBs/hour in audit logs alone. Again, audit != logrotate, and even Red Hat (somehow) copies (from /usr/share/doc/audit-*/) to /etc/cron.daily/auditd.cron file (which is what I'm trying to figure out when and how often -- like on a RPM update).

NOTES:

[A] cron.hourly -- per /etc/crond.d/0hourly, :01 after the hour

[B] cron.daily -- per /etc/anacrontab, START_HOURS_RANGE=3-22, delay in minutes for daily = 5

Hi Bryan,

I checked my systems. I (of course) found the same results as you.

Additionally, I checked my systems for the existence of the file at /etc/cron.daily/auditd.cron by pushing a query from my satellite server. None of my systems have the file named "/etc/cron.daily/auditd.cron"

Is there any chance someone is making this file as the result of a different cron, or a script within a cron? Or maybe someone is putting the file there via another means? I have seen where some ... people ... drop in files under /etc/cron.daily for their own needs, in a less-than-documented way.

Is there any chance someone's pushing this file via some other method that's not the function of a script, such as ansible or puppet or some other means?

Hope you find the culprit function that's creating that file... good luck

R. Hinton --

Do you have auditd installed, configured and, most importantly, with rules? E.g., /etc/audit/audit.rules /etc/audit/rules.d/* (in addition to /etc/audit/auditd.conf), plus chkconfig on?

-- bjs

Bryan, yes

We have a means to put our audit logs to another location for archive, and any required future reference.

BTW, one of the major reasons for this file is that these systems run at remote sites without network access and without local sysadmins. So anything and everything we can do to compress and archive is most ideal, to avoid filling up /var/log[/audit]

Ah, I see the challenge you have...

Indeed. Which is why /etc/cron.hourly/zlogs.cron is a "catch all" that fills in various gaps, as well as adding a few redundant "checks."

E.g., someone updates and/or removes/re-installs logrotate and loses settings. It will catch, rename and compress several files.

Regardless, in the case of audit, it doesn't seem to have a compression setting. Although it does have an "exec" option, especially as the file system is filling up. So we have another script to move already compressed files to another file system as well.

UPDATE: I'm starting to think it might not be a Red Hat package or script, based on timestamps.

%> su -c 'yum whatprovides */auditd.cron'
Password: 
Loaded plugins: product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is receiving updates from RHN Classic or RHN Satellite.
epel/filelists_db                                                                                           | 8.0 MB     00:00     
rhel-x86_64-server-6/filelists                                                                              |  26 MB     00:27     
rhel-x86_64-server-optional-6/filelists                                                                     |  10 MB     00:10     
audit-2.1.3-3.el6.x86_64 : User space tools for 2.6 kernel auditing
Repo        : rhel-x86_64-server-6
Matched from:
Filename    : /usr/share/doc/audit-2.1.3/auditd.cron



audit-2.1-5.el6.x86_64 : User space tools for 2.6 kernel auditing
Repo        : rhel-x86_64-server-6
Matched from:
Filename    : /usr/share/doc/audit-2.1/auditd.cron



audit-2.0.4-1.el6.x86_64 : User space tools for 2.6 kernel auditing
Repo        : rhel-x86_64-server-6
Matched from:
Filename    : /usr/share/doc/audit-2.0.4/auditd.cron



audit-2.2-4.el6_5.x86_64 : User space tools for 2.6 kernel auditing
Repo        : rhel-x86_64-server-6
Matched from:
Filename    : /usr/share/doc/audit-2.2/auditd.cron



audit-2.3.7-5.el6.x86_64 : User space tools for 2.6 kernel auditing
Repo        : rhel-x86_64-server-6
Matched from:
Filename    : /usr/share/doc/audit-2.3.7/auditd.cron



audit-2.2-2.el6.x86_64 : User space tools for 2.6 kernel auditing
Repo        : rhel-x86_64-server-6
Matched from:
Filename    : /usr/share/doc/audit-2.2/auditd.cron



audit-2.3.7-5.el6.x86_64 : User space tools for 2.6 kernel auditing
Repo        : installed
Matched from:
Filename    : /usr/share/doc/audit-2.3.7/auditd.cron

Yep, which is why I'm trying to track down (as I mentioned prior) ...

'... even Red Hat (somehow) copies (from /usr/share/doc/audit-*/) to /etc/cron.daily/auditd.cron file (which is what I'm trying to figure out when and how often -- like on a RPM update) ..." -- 29 March 2016 7:10 PM (GMT) Bryan Smith

I'm now trying to see the differences between a basic Kickstarted system and the fully configured systems we have. Again, I'm starting to think this isn't from a process that is created/generated/managed by a Red Hat utility, policy or other solution.

Partly what I was attempting to suggest in my initial reply... One of my suspicions was someone else dropped it in there.

You can monitor processes with this and examine the strace for possible creation of the auditd.conf after removing it.

#!/bin/sh

# Change 'auditd' for whatever pattern you wish to examine
for pid in $(lsof | grep auditd | awk '$1 ~ /^auditd$/ && $9 ~ /^\/[sbin|lib|var].*/{print $2":"$9}'); do
  file=$(echo "${pid}"|cut -d : -f 2);
  proc=$(echo "${pid}"|cut -d : -f 1);

  if [[ "$(file ${file})" =~ object ]]; then
    # Schedule the killing of strace
    echo "kill -9 $(ps -ef|grep -v grep|grep strace|awk '{print $2}')"|at now + 1 min

    # Watch & log the ${proc}
    strace -fo /var/tmp/${proc}-$(basename ${file})-strace.log -p ${proc}
  fi
done

Just FYI, I'm looking for how "/etc/cron.daily/auditd.cron" is being created. I will run this on a Kickstarted system to see if it is created by the "auditd" process itself.

BTW, the "/etc/audit/auditd.conf" file is part of package "audit", and I've already created the "trigger" in my zlogs.spec file for my RPM to backup/replace that file if the package "auditd" is installed, updated, etc...

...
%files
%defattr(-,root,root,-)
%attr(600,root,root) /etc/audit/auditd.conf_archive
...

%triggerin -- audit
#   Executed if audit is installed, aleady installed or upgraded
# Backup and replace /etc/audit/auditd.conf, and restart auditd
/bin/cp -p /etc/audit/auditd.conf /etc/audit/auditd.conf_`/bin/date +%s`
/bin/cp -f /etc/audit/auditd.conf_archive /etc/audit/auditd.conf
/sbin/service auditd reload
# Move any auditd.cron logrotation script to /etc/cron.hourly
if [ ! -e "/etc/cron.hourly/auditd.cron" ] ; then
    if [ -e "/etc/cron.daily/auditd.cron" ] ; then
        /bin/mv -f "/etc/cron.daily/auditd.cron" "/etc/cron.hourly/" 2>> /dev/null
    elif [ -e "/etc/cron.weekly/auditd.cron" ] ; then
        /bin/mv -f "/etc/cron.weekly/auditd.cron" "/etc/cron.hourly/" 2>> /dev/null
    elif [ -e "/etc/cron.monthly/auditd.cron" ] ; then
        /bin/mv -f "/etc/cron.monthly/auditd.cron" "/etc/cron.hourly/" 2>> /dev/null
    else
        f="/.invalidfile"
        # Look for auditd.cron script in the /usr/share/doc/audit-*/ directory(ies)
        for p in /usr/share/doc/audit-*/auditd.cron ; do f="${p}" ; done
        if [ -r "${f}" ] ; then
            # Copy, instead of move
            /bin/cp -n "${f}" /etc/cron.hourly/ 2>> /dev/null
            /bin/chmod 700 /etc/cron.hourly/auditd.cron 2>> /dev/null
            /sbin/restorecon /etc/cron.hourly/auditd.cron 2>> /dev/null
        fi
    fi
fi
...

I figured from the onset that was your goal, to track down the means "/etc/cron.daily/auditd.cron" was appearing on your system. Nice bit you posted with the rpm spec file you made.

I have heard tacit discussion (but have not found anything documented... yet) for /etc/some.conf.override file (ending in "override"). I'm not personally fully versed in the use of that, but I had thought from the tacit info I read, that could retain a configuration file one may want to keep. Have you heard of this or know of any documentation that covers it (I've not found any yet, still searching)? Of course there are instances where this may not be something someone wants.

When I did my "rpm -ql" queries, I did not see "/etc/cron.daily/auditd.cron" as a resultant file from the audit RPM, but the "rpm -ql audit" of course shows "/etc/audit/auditd.conf". I did some queries as I mentioned (tacitly) and found the same results as you.

I just (now) did a kickstart of rhel 7.2 and (of course) I did not see "/etc/cron.daily/auditd.cron" and of course the /etc/audit/auditd.conf was present as listed and expected.

This is just one way I've approached it, when I always want to override what a RPM package provides. Take package "logrotate" for example, which does provide not only ...

  • /etc/logrotate.conf , but also ... /etc/cron.daily/logrotate

Again. I want to replace both the former and move the latter, provided in the package, to /etc/cron.hourly/ -- including when logrotate is updated.

So I use the following trigger in the same SPEC file for zlogs as well ...

...
%files
%defattr(-,root,root,-)
%attr(644,root,root) /etc/logrotate.conf_hourly
...

%triggerin -- logrotate
#       Executed if logrotate is installed, aleady installed or upgraded
# Backup and replace /etc/audit/logrotate.conf
/bin/cp -p /etc/logrotate.conf /etc/logrotate.conf_`/bin/date +%s`
/bin/cp -f /etc/logrotate.conf_hourly /etc/logrotate.conf
# Move any logrotate script to /etc/cron.hourly
if [ ! -e "/etc/cron.hourly/logrotate" ] ; then
        if [ -e "/etc/cron.daily/logrotate" ] ; then
                /bin/mv -f "/etc/cron.daily/auditd.cron" "/etc/cron.hourly/" 2>> /dev/null
        fi
fi

We also have the Katello, Foreman, Puppet (Pulp, Candlepin, et al.) stack in our lab. We're going to move away from RPM-based configuration and do this proper in Puppet Manifests, especially if and when Satellite 6 (the commercial version of the aforementioned Upstream combo meal) is more stable (hopefully in 6.2 shortly).

Thanks Jason, that script could be useful for other things as well

Usually, when trying to track down where files come from I do something like rpm -qf $(readlink -f /path/to/file).

Neither my EL6 nor EL7 systems have an /etc/cron.daily/auditd.cron file. Then again, it's more typical to have the actual auditd service configured to handle its own log-rotation rather than using logrotate/cron. Looking at rpm -qVf $(readlink -f /etc/audit/auditd.conf) (on a test system), its default configuration has auditd handling its own log roatation:

num_logs = 5
max_log_file = 6
max_log_file_action = ROTATE

Generally, it's bad juju if you have both auditd and logrotate trying to rotate the same set of files.

To reduce revisiting this over and over (since it has happened several times now, not trying to single anyone out), I have added the following to the top of the Discussion ... ;)

PREFACE (EDIT): Please keep in mind 2 things before posting, as they have been visited several times and are not part of our issue.

1) Before we posted this, we had already done the following RPM commands: -qf (RPM [what]provides), -q --scripts (RPM scripts), -q --triggers (RPM triggers). We are trying to figure out what process is copying the file /usr/share/doc/audit-*/auditd.cron to /etc/cron.daily. It seems to be a post-installation process not in any RPM, file, script or trigger.

2) We are not using logrotate for audit, period. auditd.cron and logrotate are for different sets of files, as part of a greater solution reduce disk usage for logging and auditing. We use logrotate and its inherent (zip -9) compression for those files in logrotate which, again, does not include auditd's logs. We are also using other scripts -- cron.hourly/zlogs.cron (ORIGINAL) and cron.monthly/arclogs.cron (NEW) -- to rename/compress audit and other log files as well as move them to another file system. The latter, normally run monthly, even gets automatically kicked off, per audit/auditd.conf, when the file system drops to under 0.5GiB free.

And what I'm saying is that, on the unmodified EL6 and EL7 test systems (AWS is awesome for quickly genning "stock"systems), no auditd.cron scripts get created. The default is to use the functionality built into the auditd service (thus, the posting of the excerpted /etc/auditd/auditd.conf file).

Usually, when I see someone using a cron job, it's because they've specifically disabled auditd's internal log-rotation and then created the cron job via other avenues. Perhaps whoever put together your KickStart (or other standard build) profile opted to incorporate the cron-enabled auditd log-rotation?

We're using the cron job to always rotate and, subsequently, compress by the hour. But if auditing is generating massive amounts -- say 2GiB/hour -- we want to be cutting them faster, if and when that happens. Hence why we put higher limits in the auditd.conf, but they are there to ensure they get cut more often.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.