Logs flooded with systemd messages: Created slice, Starting Session

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7, 8, 9

Issue

  • On my newly installed system I am seeing the following in /var/log/messages and journal all the time

    Jul 24 08:50:01 example.com systemd: Created slice user-0.slice.
    Jul 24 08:50:01 example.com systemd: Starting Session 150 of user root.
    Jul 24 08:50:01 example.com systemd: Started Session 150 of user root.
    Jul 24 09:00:01 example.com systemd: Created slice user-0.slice.
    Jul 24 09:00:02 example.com systemd: Starting Session 151 of user root.
    Jul 24 09:00:02 example.com systemd: Started Session 151 of user root.
    

Resolution

  • To suppress these log entries in /var/log/messages and journal you have following options:

    • enable lingering for the user so systemd --user instance will continue running even when user has no active sessions:

      # loginctl enable-linger testuser
      
    • restrict logging for systemd user manager for the user:

      # mkdir /etc/systemd/system/user@1000.service.d
      # cat > /etc/systemd/system/user@1000.service.d/logging.conf <<EOF
      [Service]
      LogLevelMax=notice
      EOF
      
    • restrict logging for systemd user manager globally:

      # sed -i -e 's/#LogLevel=info/LogLevel=notice/' /etc/systemd/user.conf
      
    • in RHEL 7, create a discard filter with rsyslog, e.g., run the following command:

      echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf
      

      Then restart the rsyslog service:

      systemctl restart rsyslog
      

      Note In case above rule only works for local syslog logging (/var/log/messages) but doesn't work for remote syslog server then we can understand this rule is read after syslog events are sent to syslog server. So in order to avoid this add the rule directly in rsyslog.conf above the line of remote syslog sending configuration. (*.* @@sys-log server)

Root Cause

  • These messages are normal and expected -- they will be seen any time a user logs in or e.g. a cron job is executed for the user

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

19 Comments

The proposed command produces the following error:

rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]

Indeed. (Though to be clear, it's a warning; not an error.) Old habits die hard. I've updated the article. Thanks.

I have updated the filter to: if $programname == "systemd" and ($msg startswith "Starting Session" or $msg startswith "Started Session" or $msg startswith "Created slice" or $msg startswith "Starting user-" or $msg startswith "Removed slice" or $msg startswith "Stopping user-") then stop

This filter will no longer log entries like:

Apr  5 09:53:01 aixtsmprod01 systemd: Removed slice user-1000.slice.
Apr  5 09:53:01 aixtsmprod01 systemd: Stopping user-1000.slice.

I guess the filter should also contain below 2 entries also:

$msg contains "Removed slice"
$msg contains "Stopping user-"

To remove such log entries from messages logs:

Aug 18 10:20:01 test-prov-el7 systemd: Removed slice user-0.slice.
Aug 18 10:20:01 test-prov-el7 systemd: Stopping user-0.slice.

The overall command post adding the changes would be:

echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of" or $msg contains "Removed slice" or $msg contains "Stopping user-") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf

In rhel7u4, rsyslog have update to rsyslog v8 version. And the sample log format have changed in rhel7u4. Thanks for the update

The log is printed by the function session_start_scope()of the systemd.

Dec 6 04:01:01 localhost systemd: Created slice user-0.slice. Dec 6 04:01:01 localhost systemd: Starting Session 63 of user root. Dec 6 04:01:01 localhost systemd: Started Session 63 of user root. Dec 6 05:01:01 localhost systemd: Created slice user-0.slice.

This function is used to create cgroup; The function will be called when there a user login because the OS will need to create cgroup for the just login user (The user could a real user or an app which uses dbus messages to communicate with other users). These log just shows the info when the cgroup is created by systemd, so normally, you might be able to just ignore these logs.

Why tune rsyslog when we might tune systemd? THats where the message is created. If it were in SSH session mgt, then we'd tune sshd_config first. or.. maybe not push it into /var/log/messages.

This statement is misleading: "they will be seen any time a user logs in" An interactive user login is not the only event that will generate this message to rsyslog; other events generate that same message. That statement might suggest otherwise to an unknowing system administrator.

seems like in rhel 7.4 the logger deamon is no longer "systemd" but "systemd-logind" which makes the current rsyslog filter invalid:

Nov 30 08:54:04 db161142 systemd-logind[2445]: New session 2172664 of user dbi_backup. Nov 30 08:54:05 db161142 systemd-logind[2445]: New session 2172665 of user dbi_backup. Nov 30 08:54:05 db161142 systemd-logind[2445]: New session 2172666 of user dbi_backup.

I tried to install cinnamon. I ended with a balcnk screen. I pressed ctrl alt F2. I got a blinking cursor at the upper left of the scree. Then the messages appeared: [Ok] Created slice User Slice of root. [ok] Started session 1 of user root [ok] Removed slice User Slice of root] at the bottom of this the cursor is blinking and I can enter nothing. I see it is continuing [ok]Removed slice User of root [ok]Started session 4 of user root [OK] Removed slice User Slice of root What do I do?

Can I ask what was the logic in having this information go to syslog in the first place?
We have filtering tools to ignore specific types of messages but this make as much sense as having email go to a folder by default so you can delete it in bulk.

Why not solve the problem, not the symptom?

Exactly! Tune the service that creates the log. Since we have a service called systemd-logind, can we tune that? Surely the ability to control logging behavior isn't taken away. Might we add logging controls to pam_systemd? This looks like a good place to apply some logic . https://www.freedesktop.org/software/systemd/man/pam_systemd.html - then we can put rules in for logging cron jobs, ssh sessions, console logins, webmin, .. might even catch unintended sessions from exploits? Since Systemd is subsuming everything, this should be considered.

For the record (years later), if you're getting these every 10 minutes (and only every 10 minutes), it's most likely sar's cronjob firing off that systemd feels the need to flood our collective logs with.

If the sar is cause of this logs, I think the kb needs to modify only to filter out cron related root login session logs.

What about logs like these:

Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "45": "289262471".
Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "42": "16794120".
Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "35": "766617882".
Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "34": "21982869".
Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "32": "1211475599".
Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "20": "320649923".
Oct 12 22:58:41 system teamd_team0[1940]: stats update for hash "18": "268314772".
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "99" (delta 1058683) to port enp5s6f0.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "32" (delta 196490) to port enp5s5f0.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "79" (delta 10360) to port enp4s0f1.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "92" (delta 6684) to port enp3s0f1.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "34" (delta 6398) to port enp3s0f0.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "35" (delta 6326) to port enp3s0f1.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "66" (delta 5876) to port enp4s0f0.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "55" (delta 4555) to port enp4s0f1.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "51" (delta 4530) to port enp4s0f0.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "18" (delta 4134) to port enp3s0f0.
Oct 12 22:58:41 system teamd_team0[1940]: Remapped hash "108" (delta 4134) to port enp3s0f1.
Oct 12 22:58:41 system teamd_team0[1940]: Port enp5s6f0 rebalanced, delta: 1058683
Oct 12 22:58:41 system teamd_team0[1940]: Port enp5s5f0 rebalanced, delta: 196490
Oct 12 22:58:41 system teamd_team0[1940]: Port enp4s0f1 rebalanced, delta: 15311
Oct 12 22:58:41 system teamd_team0[1940]: Port enp4s0f0 rebalanced, delta: 17200
Oct 12 22:58:41 system teamd_team0[1940]: Port enp3s0f1 rebalanced, delta: 17144
Oct 12 22:58:41 system teamd_team0[1940]: Port enp3s0f0 rebalanced, delta: 16930

Did anyone manage to suppress these ?

systemd генерит не информативный спам в логи. Надо чинить systemd, а не делать костыли в правилах rsyslog

Hi team.

I have this problem... the the connection is lost immediately

May 18 21:44:15 CDRServer systemd: Created slice User Slice of SplunkLogs. May 18 21:44:15 CDRServer systemd: Started Session 268482 of user SplunkLogs. May 18 21:44:15 CDRServer systemd-logind: New session 268482 of user SplunkLogs. May 18 21:44:15 CDRServer systemd-logind: Removed session 268482.

Do you know the possibles reasons and how to fix it?

Regards!

Hi all.

I have this problem... the connection is lost immediately

May 18 21:44:15 CDRServer systemd: Created slice User Slice of SplunkLogs. 
May 18 21:44:15 CDRServer systemd: Started Session 268482 of user SplunkLogs. 
May 18 21:44:15 CDRServer systemd-logind: New session 268482 of user SplunkLogs. 
May 18 21:44:15 CDRServer systemd-logind: Removed session 268482.

Do you know the possible reasons and how to fix it?

Regards!

Did you get any solution Carlos Duarte.? If yes , let me know since I'm also facing the same issue.