[SOLVED] - unkown cause for systemd failure state - systemd-logind cannot start org.freedesktop.systemd1 after server reboot
Basic information
Red Hat Enterprise Linux Server release 7.4 (Maipo)
component: systemd
Hardware: x86_64 Linux
[root@scvberpat01 log]# uname -r
3.10.0-693.21.1.el7.x86_64
General description
Hey everyone, I had quite a major issue this weekend resulting in a necessary server restart of our company's production Server.
It all corresponded around the systemd Service going into FAILURE Status and not being able to start the loginmanager org.freedesktop.login1 and org.freedesktop.systemd1 via dbus/systemd-logind again.
It seems that something that I am not able to recognise has managed to kill the systemd-logind Service. We are running a heavy production System with around 1700 Tasks/processes on an average production day simultaniously. Crons and incrons are in use.
In Addition, since the restart the modul org.freedesktop.systemd1 cannot be started by systemd causing a significant delay in logintime and user change time via su.
systemd version
[root@scvberpat01 tmp]# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
Journal entries of the incident starting off the FAILURE state of systemd
ul 14 05:09:40 scvberpat01 sshd[31966]: Accepted password for erp_monitoring from 172.20.0.63 port 58768 ssh2
Jul 14 05:09:40 scvberpat01 systemd-logind: New session 1135476 of user erp_monitoring.
Jul 14 05:09:40 scvberpat01 systemd: Started Session 1135476 of user erp_monitoring.
Jul 14 05:09:40 scvberpat01 systemd: Starting Session 1135476 of user erp_monitoring.
Jul 14 05:09:45 scvberpat01 sshd[32537]: Accepted password for erp_monitoring from 172.20.0.63 port 58801 ssh2
Jul 14 05:09:45 scvberpat01 systemd-logind: New session 1135477 of user erp_monitoring.
Jul 14 05:09:45 scvberpat01 systemd: Started Session 1135477 of user erp_monitoring.
Jul 14 05:09:45 scvberpat01 systemd: Starting Session 1135477 of user erp_monitoring.
Jul 14 05:10:01 scvberpat01 systemd: Started Session 1135478 of user root.
Jul 14 05:10:01 scvberpat01 systemd: Starting Session 1135478 of user root.
Jul 14 05:10:02 scvberpat01 systemd: Started Session 1135479 of user paseb.
Jul 14 05:10:02 scvberpat01 systemd: Starting Session 1135479 of user paseb.
Jul 14 05:10:02 scvberpat01 systemd: Started Session 1135480 of user root.
Jul 14 05:10:02 scvberpat01 systemd: Starting Session 1135480 of user root.
Jul 14 05:10:03 scvberpat01 systemd: Started Session 1135482 of user paseb.
Jul 14 05:10:03 scvberpat01 systemd: Starting Session 1135482 of user paseb.
Jul 14 05:10:03 scvberpat01 systemd: Started Session 1135481 of user paseb.
Jul 14 05:10:03 scvberpat01 systemd: Starting Session 1135481 of user paseb.
Jul 14 05:10:03 scvberpat01 systemd: Started Session 1135484 of user batchparg.
Jul 14 05:10:03 scvberpat01 systemd: Starting Session 1135484 of user batchparg.
Jul 14 05:10:04 scvberpat01 systemd: Started Session 1135485 of user batchparg.
Jul 14 05:10:04 scvberpat01 systemd: Starting Session 1135485 of user batchparg.
Jul 14 05:10:05 scvberpat01 systemd: Started Session 1135483 of user batchparg.
Jul 14 05:10:05 scvberpat01 systemd: Starting Session 1135483 of user batchparg.
Jul 14 05:10:06 scvberpat01 systemd: Started Session 1135486 of user batchparg.
Jul 14 05:10:06 scvberpat01 systemd: Starting Session 1135486 of user batchparg.
Jul 14 05:10:07 scvberpat01 systemd: Started Session 1135487 of user batchparg.
Jul 14 05:10:07 scvberpat01 systemd: Starting Session 1135487 of user batchparg.
Jul 14 05:10:08 scvberpat01 systemd: Started Session 1135488 of user batchparg.
Jul 14 05:10:08 scvberpat01 systemd: Starting Session 1135488 of user batchparg.
Jul 14 05:10:10 scvberpat01 systemd: Started Session 1135489 of user batchparg.
Jul 14 05:10:10 scvberpat01 systemd: Starting Session 1135489 of user batchparg.
Jul 14 05:10:11 scvberpat01 systemd: Started Session 1135490 of user batchparg.
Jul 14 05:10:11 scvberpat01 systemd: Starting Session 1135490 of user batchparg.
Jul 14 05:10:12 scvberpat01 systemd: Started Session 1135491 of user batchparg.
Jul 14 05:10:12 scvberpat01 systemd: Starting Session 1135491 of user batchparg.
Jul 14 05:10:13 scvberpat01 systemd: systemd-logind.service has no holdoff time, scheduling restart.
Jul 14 05:10:13 scvberpat01 systemd: Starting Login Service...
Jul 14 05:10:13 scvberpat01 systemd: Started Login Service.
Jul 14 05:10:13 scvberpat01 systemd-logind: New seat seat0.
Jul 14 05:10:13 scvberpat01 systemd-logind: Failed to read /run/systemd/users/11469: Argument list too long
Jul 14 05:10:13 scvberpat01 systemd-logind: Failed to read /run/systemd/users/0: Argument list too long
Jul 14 05:10:13 scvberpat01 systemd-logind: User enumeration failed: Argument list too long
Jul 14 05:11:58 scvberpat01 systemd-logind: Failed to stop user slice: No buffer space available
Jul 14 05:11:58 scvberpat01 systemd-logind: Failed to stop user slice: No buffer space available
Jul 14 05:11:58 scvberpat01 systemd-logind: Failed to stop user slice: No buffer space available
Jul 14 05:11:58 scvberpat01 systemd-logind: Failed to stop user slice: No buffer space available
Jul 14 05:11:58 scvberpat01 systemd-logind: Failed to start user slice user-11469.slice, ignoring: No buffer space available ((null))
Jul 14 05:11:59 scvberpat01 systemd-logind: Failed to start user slice user-0.slice, ignoring: No buffer space available ((null))
Jul 14 05:13:13 scvberpat01 systemd: systemd-logind.service watchdog timeout (limit 3min)!
Jul 14 05:13:13 scvberpat01 abrt-hook-ccpp: Process 2577 (systemd-logind) of user 0 killed by SIGABRT - dumping core
Jul 14 05:13:16 scvberpat01 ModemManager[1041]: [sleep-monitor] inhibit failed: GDBus.Error:org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message b
us)
Jul 14 05:13:16 scvberpat01 systemd: systemd-logind.service: main process exited, code=dumped, status=6/ABRT
Jul 14 05:13:16 scvberpat01 systemd: Unit systemd-logind.service entered failed state.
Jul 14 05:13:16 scvberpat01 systemd: systemd-logind.service failed.
Jul 14 05:13:21 scvberpat01 kernel: nr_pdflush_threads exported in /proc is scheduled for removal
Jul 14 05:13:40 scvberpat01 dbus[1028]: [system] Activating systemd to hand-off: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'
Jul 14 05:13:41 scvberpat01 systemd-logind: Failed to enable subscription: Connection timed out
Jul 14 05:13:41 scvberpat01 systemd-logind: Failed to fully start up daemon: Connection timed out
Jul 14 05:13:41 scvberpat01 dbus[1028]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 14 05:13:41 scvberpat01 systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
Jul 14 05:13:41 scvberpat01 systemd: Failed to start Login Service.
Jul 14 05:13:41 scvberpat01 systemd: Unit systemd-logind.service entered failed state.
Jul 14 05:13:41 scvberpat01 systemd: systemd-logind.service failed.
Jul 14 05:14:05 scvberpat01 dbus[1028]: [system] Failed to activate service 'org.freedesktop.login1': timed out
Jul 14 05:14:06 scvberpat01 systemd-logind: Failed to enable subscription: Connection timed out
Jul 14 05:14:06 scvberpat01 systemd-logind: Failed to fully start up daemon: Connection timed out
Jul 14 05:14:06 scvberpat01 dbus[1028]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 14 05:14:06 scvberpat01 systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
Jul 14 05:14:06 scvberpat01 systemd: Failed to start Login Service.
Jul 14 05:14:06 scvberpat01 systemd: Unit systemd-logind.service entered failed state.
Jul 14 05:14:06 scvberpat01 systemd: systemd-logind.service failed.
Jul 14 05:14:31 scvberpat01 systemd-logind: Failed to enable subscription: Connection timed out
Jul 14 05:14:31 scvberpat01 dbus[1028]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jul 14 05:14:31 scvberpat01 systemd-logind: Failed to fully start up daemon: Connection timed out
Jul 14 05:14:31 scvberpat01 systemd: systemd-logind.service: main process exited, code=exited, status=1/FAILURE
...
...
This went on for 1 and a half days until it impacted the Server heavily. In the end ssh, samba as well as the terminal did not respond anymore. Unfortunately it wasn't recognised beforehand so a hard reset was performed.
After the restart it seems that systemd is not able to start org.freedesktop.systemd1:
Jul 18 13:21:05 scvberpat01.pankl.local systemd-logind[1023]: Failed to start user slice user-11511.slice, ignoring: Activation of org.freedesktop.systemd1 timed out (org.freedesktop.DBus.Error.TimedOut)
Jul 18 13:21:30 scvberpat01.pankl.local systemd-logind[1023]: Failed to start session scope session-33081.scope: Connection timed out
Jul 18 13:22:05 scvberpat01.pankl.local systemd-logind[1023]: Failed to start user slice user-11511.slice, ignoring: Activation of org.freedesktop.systemd1 timed out (org.freedesktop.DBus.Error.TimedOut)
Jul 18 13:22:30 scvberpat01.pankl.local systemd-logind[1023]: Failed to start session scope session-33086.scope: Activation of org.freedesktop.systemd1 timed out
Jul 18 13:23:05 scvberpat01.pankl.local systemd-logind[1023]: Failed to start user slice user-11511.slice, ignoring: Activation of org.freedesktop.systemd1 timed out (org.freedesktop.DBus.Error.TimedOut)
Jul 18 13:23:30 scvberpat01.pankl.local systemd-logind[1023]: Failed to start session scope session-33092.scope: Activation of org.freedesktop.systemd1 timed out
Jul 18 13:24:05 scvberpat01.pankl.local systemd-logind[1023]: Failed to start user slice user-11511.slice, ignoring: Connection timed out ((null))
Jul 18 13:24:30 scvberpat01.pankl.local systemd-logind[1023]: Failed to start session scope session-33097.scope: Activation of org.freedesktop.systemd1 timed out
Jul 18 13:25:05 scvberpat01.pankl.local systemd-logind[1023]: Failed to start user slice user-11511.slice, ignoring: Connection timed out ((null))
Jul 18 13:25:30 scvberpat01.pankl.local systemd-logind[1023]: Failed to start session scope session-33102.scope: Connection timed out
"busctl --list" is showing that org.freedesktop.systemd1 is not activ
[root@scvberpat01 log]# busctl --list
NAME PID PROCESS USER CONNECTION UNIT SESSION DESCRIPTION
:1.0 1023 systemd-logind root :1.0 systemd-logind.service - -
:1.10 1076 NetworkManager root :1.10 NetworkManager.service - -
:1.2 1019 avahi-daemon avahi :1.2 avahi-daemon.service - -
:1.24 1348 cupsd root :1.24 cups.service - -
:1.25 1342 tuned root :1.25 tuned.service - -
:1.26 1583 colord colord :1.26 colord.service - -
:1.27 1348 cupsd root :1.27 cups.service - -
:1.28 2268 libvirtd root :1.28 libvirtd.service - -
:1.3 1030 rtkit-daemon root :1.3 rtkit-daemon.service - -
:1.34123 15157 abrt-dbus root :1.34123 dbus.service - -
:1.34132 9585 busctl root :1.34132 sshd.service - -
:1.384 4756 packagekitd root :1.384 packagekit.service - -
:1.4 1029 ModemManager root :1.4 ModemManager.service - -
:1.5 1020 polkitd polkitd :1.5 polkit.service - -
:1.6 1088 accounts-daemon root :1.6 accounts-daemon.service - -
:1.7 1076 NetworkManager root :1.7 NetworkManager.service - -
com.redhat.RHSM1 - - - (activatable) - -
com.redhat.RHSM1.Facts - - - (activatable) - -
com.redhat.SubscriptionManager - - - (activatable) - -
com.redhat.ifcfgrh1 1076 NetworkManager root :1.10 NetworkManager.service - -
com.redhat.problems.configuration - - - (activatable) - -
com.redhat.tuned 1342 tuned root :1.25 tuned.service - -
fi.epitest.hostap.WPASupplicant - - - (activatable) - -
fi.w1.wpa_supplicant1 - - - (activatable) - -
net.reactivated.Fprint - - - (activatable) - -
org.bluez - - - (activatable) - -
org.fedoraproject.SetroubleshootFixit - - - (activatable) - -
org.fedoraproject.Setroubleshootd - - - (activatable) - -
org.freedesktop.Accounts 1088 accounts-daemon root :1.6 accounts-daemon.service - -
org.freedesktop.Avahi 1019 avahi-daemon avahi :1.2 avahi-daemon.service - -
org.freedesktop.ColorManager 1583 colord colord :1.26 colord.service - -
org.freedesktop.DBus - - - - - - -
org.freedesktop.Flatpak.SystemHelper - - - (activatable) - -
org.freedesktop.GeoClue2 - - - (activatable) - -
org.freedesktop.ModemManager1 1029 ModemManager root :1.4 ModemManager.service - -
org.freedesktop.NetworkManager 1076 NetworkManager root :1.7 NetworkManager.service - -
org.freedesktop.PackageKit 4756 packagekitd root :1.384 packagekit.service - -
org.freedesktop.PolicyKit1 1020 polkitd polkitd :1.5 polkit.service - -
org.freedesktop.RealtimeKit1 1030 rtkit-daemon root :1.3 rtkit-daemon.service - -
org.freedesktop.UDisks2 - - - (activatable) - -
org.freedesktop.UPower - - - (activatable) - -
org.freedesktop.hostname1 - - - (activatable) - -
org.freedesktop.import1 - - - (activatable) - -
org.freedesktop.locale1 - - - (activatable) - -
org.freedesktop.login1 1023 systemd-logind root :1.0 systemd-logind.service - -
org.freedesktop.machine1 - - - (activatable) - -
org.freedesktop.nm_dispatcher - - - (activatable) - -
org.freedesktop.problems 15157 abrt-dbus root :1.34123 dbus.service - -
org.freedesktop.realmd - - - (activatable) - -
org.freedesktop.systemd1 - - - (activatable) - -
org.freedesktop.timedate1 - - - (activatable) - -
org.gnome.GConf.Defaults - - - (activatable) - -
org.opensuse.CupsPkHelper.Mechanism - - - (activatable) - -
straceing a user Change Shows exactly the timout of 30 seconds line 18:34:56. This doesnt really tell me much but maybe someone can help me here. Comparing this with our testsystem which didnt face this Major incident is showing no timeout here.
18:34:56 close(3) = 0
18:34:56 munmap(0x7fee1868b000, 4096) = 0
18:34:56 socket(AF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
18:34:56 setsockopt(3, SOL_SOCKET, SO_PASSCRED, [0], 4) = 0
18:34:56 setsockopt(3, SOL_SOCKET, SO_PASSSEC, [0], 4) = 0
18:34:56 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
18:34:56 setsockopt(3, SOL_SOCKET, SO_RCVBUFFORCE, [8388608], 4) = 0
18:34:56 getsockopt(3, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
18:34:56 setsockopt(3, SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = 0
18:34:56 connect(3, {sa_family=AF_LOCAL, sun_path="/var/run/dbus/system_bus_socket"}, 33) = 0
18:34:56 getsockopt(3, SOL_SOCKET, SO_PEERCRED, {pid=1, uid=0, gid=0}, [12]) = 0
18:34:56 getsockopt(3, SOL_SOCKET, SO_PEERSEC, 0x5645d834d4f0, 0x7ffdc6169a50) = -1 ENOPROTOOPT (Protocol not available)
18:34:56 fstat(3, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
18:34:56 getsockopt(3, SOL_SOCKET, SO_ACCEPTCONN, [0], [4]) = 0
18:34:56 getsockname(3, {sa_family=AF_LOCAL, NULL}, [2]) = 0
18:34:56 geteuid() = 0
18:34:56 sendmsg(3, {msg_name(0)=NULL, msg_iov(3)=[{"\0AUTH EXTERNAL ", 15}, {"30", 2}, {"\r\nNEGOTIATE_UNIX_FD\r\nBEGIN\r\n", 28}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 45
18:34:56 gettid() = 9342
18:34:56 getrandom("\375\326f\327\327UZ\222\347T\242\240\304\227\5}", 16, GRND_NONBLOCK) = 16
18:34:56 recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"OK 0345052a3855cdc590c849e45b4b4"..., 256}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 52
18:34:56 sendmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\1\0\1\0\0\0\0\1\0\0\0m\0\0\0\1\1o\0\25\0\0\0/org/fre"..., 128}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 128
18:34:56 recvmsg(3, 0x7ffdc6168960, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
18:34:56 ppoll([{fd=3, events=POLLIN}], 1, {24, 999810000}, NULL, 8) = 1 ([{fd=3, revents=POLLIN}], left {24, 999633311})
18:34:56 recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\2\1\1\r\0\0\0\1\0\0\0E\0\0\0\6\1s\0\10\0\0\0", 24}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 24
18:34:56 recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{":1.25140\0\0\0\0\0\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1s\0\0"..., 77}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 77
18:34:56 sendmsg(3, {msg_name(0)=NULL, msg_iov(2)=[{"l\1\0\1p\0\0\0\2\0\0\0\230\0\0\0\1\1o\0\27\0\0\0/org/fre"..., 168}, {"\342,\0\0~$\0\0\4\0\0\0su-l\0\0\0\0\3\0\0\0tty\0\4\0\0\0"..., 112}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 280
18:34:56 recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"l\4\1\1\r\0\0\0\2\0\0\0\225\0\0\0\1\1o\0\25\0\0\0", 24}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 24
18:34:56 recvmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"/org/freedesktop/DBus\0\0\0\2\1s\0\24\0\0\0"..., 157}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = 157
18:34:56 recvmsg(3, 0x7ffdc6168b10, MSG_DONTWAIT|MSG_NOSIGNAL|MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)
18:34:56 ppoll([{fd=3, events=POLLIN}], 1, {24, 999829000}, NULL, 8) = 0 (Timeout)
18:35:21 open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4
18:35:21 fstat(4, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
18:35:21 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fee1868b000
18:35:21 read(4, "# Locale name alias data base.\n#"..., 4096) = 2502
18:35:21 read(4, "", 4096) = 0
18:35:21 close(4) = 0
18:35:21 munmap(0x7fee1868b000, 4096) = 0
#
I would Need help in both the possible root cause which caused systemd to get into FAILURE state as well as a solution to org.freedesktop.systemd1 failing to start. Which in my opinion is causing a Login/user change delay of 30 seconds.
Please consider the fact that we are talking about a 24/7 production System.
Thanks a lot in advance
Mario
@EDIT: Attached Core dump
Attachments
Responses
Hello Markus Where are you getting the version information "systemd version 42.17e"? I'm running Red Hat 7.7 and it's version is "systemd 219".
thank you Markus, it just happened to me after a server reboot. 1) systemctl --version systemd 219
how did you get the version 42.17e and 52.17e version numbers from?
i'm having the same issues for the past week. Its' very sporadic. Which kernel contains systemd 52.17e package?
Hello Markus Where are you getting the version information "systemd version 42.17e"? I'm running Red Hat 7.7 and it's version is "systemd 219".
Had the same issue on one (of over 25 RHEL 7 VMS). Also had 15 second ssh login attempts do to polkit warnings. Issue was selinux preventing kernel module work, this workflow resolved my issue:
sealert -a /var/log/audit/audit.log > sealert.log grep "SELinux is preventing" sealert.log
SELinux is preventing /usr/lib/systemd/systemd-initctl from getattr access on the file /sys/kernel/kexec_loaded. SELinux is preventing /usr/lib/systemd/systemd-initctl from read access on the file kexec_loaded.
ausearch -c 'systemd-initctl' --raw | audit2allow -M my-systemdinitctl semodule -i my-systemdinitctl.pp
Hi Guys,
We had same issue, our environment as follow as lines, systemd version: 62.el7_6.6Does anybody have to resolve exact solution? e.g) yum update? or another
It will fix it, as after version 52.17e?
Kind regard,
I had the issue again in 2020. I am not sure what to do next the kernel i'm running is
systemd-219-73.el7_8.5.x86_64 Tue 05 May 2020 07:43:38 AM CDT kernel-3.10.0-1127.el7.x86_64 Tue 05 May 2020 07:45:22 AM CDT [root@server1 ~]# systemctl --version systemd 219
i'm not sure if the above is a fix. I eventually was able to login with a local account after about 5 mins of waiting.
i am not confident that this issue is solved. The fix assumes you can already log into the server. The symptoms after reboot are still happening for local accounts.
I see that there is this Bug being reported https://bugzilla.redhat.com/show_bug.cgi?id=1531486
But the environment mentioned over there is RHEL7.6, but it seen on RHEL7.8 as well. Any help or solution on this?
This problem is also coming in my RHEL 7.8 server: Jul 9 20:21:26 lexhanaobqa1 dbus[1253]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkit.service' Jul 9 20:21:26 xxx systemd: Starting Authorization Manager... Jul 9 20:21:26 xxx polkitd[9894]: Started polkitd version 0.112 Jul 9 20:21:26 xxx systemd: Started Authorization Manager. Jul 9 20:21:51 xxx dbus[1253]: [system] Failed to activate service 'org.freedesktop.PolicyKit1': timed out
Does any one got concrete solution for this issue. Regards Charanjit Singh
Right so I'm a forensic/pentester that uses fedora sec atm. What your describing is something thats only been noticed with the psuedomanuscypt and related family of true rootkits(sysrootkits and gfx rootkits, not what is in relatity a bootkit). Only 3 or 4 major virus platforms acknowledge it, and it will get passed up in platform code checks. Thanks to the way the SELinux kernel runs, it can't hook into the system level if thats the loaded kernel, but what it installs in bios files isn't able to be large enough to support its own kernel, and therefore relies on the foreground kernel of whatever unix linux or windows os is running in order to process the sys hooks I look for when I'm ripping apart packages and files. Microsoft pushed the updates released on 11JAN2022 because this family hooks in using most of the system files affected by those patches. This is why they didn't have VPNs working correctly for a while until the 17JAN2022 random update pushed on windows 10 and 11.
Now, here's the bad news: the standard linux kernel is able to integrate with this family of rootkit and loaders. Fully. Its all written in C, with minor python, jvm and other less prevalent languages used depending on the modules loaded. Thanks to the differences between the standard kernel and the SELinux kernel used by fedora and redhat, these hooks will activate but not fully integrate, resulting in errors like this. This family of viruses specifically targets whatever endpoints have the most processing power(servers, gamers with good graphics cards, security platform users, basically anything with enough power to run at minimum background crypto miners). This family of viruses also keeps a ransomwear kit in its reserve(this is the specific part I study as my specific specialty is encryption technology). Finally, the worst part of this virus: Its Chinese state made. They've only shared the control files with a few select groups. It routes signal into a cloudfare dns in NYC thats ran by SBO that uses XI research code from when they bought the company(london based with deep chinese ties) to backload the entire network it create into Shanghai and Hong Kong.
I have no full fix for this yet. If you execute that reset command, it will not fix it, just cover it up. The hooks are still present, it just can't fully integrate with the OS your running and has limited functions. If you know how to access the system itself like the bios and edit the files, then you have a chance. I did this using Kali, Parrot and then blackarch followed by using a fedora to completely rewrite the gfx and most of the sys bios. I missed a very small amount of memory in my sys bios, which is why I still get an error popup every now and again from bugreporter on my fedora boot. Thankfully, this has decimated enough of the functionality to where the most I get is a probe of this and that with no network integration. I've also been able to finally rescue my drives affected through a fedora live zeroing of 3 passes. Its a terrible bit of code, but amazingly genius if you actually find it and realize what your looking at exactly.
Route of infection is the final part I want to comment on. There are two primary ones i've documented. First is log4j or log4shell exploits(they are not all documented publicly.). The others are the classic malicious link to download malicious file. They can both be achived in multiple manners. There are other ways, but these are the two primary i've seen. It genuinly doesn't matter what your running to protect yourself or if someone remotely accessed a vm or anything like this. If someone interacts with the system in any part and gets the base to load, it starts the background processes. Depending on capabilities both security wise and power wise of your system, whatever type of system it is or architechture/instruction sets it supports, as welll as the network bandwidth available, the files can be loaded and begin to execute anywhere from 10 seconds to 10 hours. It limits background network traffic to, unless it can tap into an updater or reporter with streaming capabilites, about 100kbps.
Of the 20+ years I've been doing this, I just find and analyze then attempt to reproduce my findings independently to confirm. Redhat and Fedora are the ONLY two OS's I've experimented with on the control files that have failed to infect. Take that as you may, but seriously bravo to redhat.
Genuinely, dont overwrite the files or reset compatiabilites. access the protected system. Destroy every single file you find and forcibly rewrite the sys bios and gfx bios binaries. then either zero your drives with at least 3 passes(I personally prefer a good old 25 pass DOD grade wipe when theirs time to clean an infected drive) or replace them prior to powering on(make sure if you just zero your ensuring nothing on the HD's is accessed at bios load) and do complete reinstalls. Otherwise, just replace your system(I do realize I'm stating this to a bunch of people that typically work on $100k servers and server farms but its the truth) if you can't get that to take.
Also, genuinely don't care if you believe me or not. I realize I'm some random person who signed up just to post this message on a redhat service board, and you're probably not going to believe me. I don't care. I'm normally paid to not talk about findings like this, but tbh this one is so nasty I'll let myself look a little crazy online to strangers just to put some word out about it.
It'd be very helpful to know more about the technical mechanics involved, such as "what it installs in the bios files" as well as the ability to remain in the "foreground kernel" it relies upon to flourish. Your post here is alarming but detailed and seemingly invaluable for the nascent encryption enthusiasts. Thank you, regardless. Fascinating stuff.
"Thanks to the way the SELinux kernel runs, it can't hook into the system level if thats the loaded kernel, but what it installs in bios files isn't able to be large enough to support its own kernel, and therefore relies on the foreground kernel of whatever unix linux or windows os is running in order to process the sys hooks..."
