cockpit and libvirt silently die
This is RHEL 8.9
first cockpit. Service starts but dies in couple minutes without any error. socket remain open... I tried to stop / start many times, no change.
[root@myhost log]# systemctl stop cockpit.socket cockpit
[root@myhost log]# ss -tupln|grep 9090
[root@myhost log]#
[root@myhost log]# systemctl reset-failed cockpit.socket cockpit
[root@myhost log]#
[root@myhost log]# systemctl start cockpit
[root@myhost log]#
[root@myhost log]# journalctl -f
-- Logs begin at Mon 2024-03-11 21:33:57 UTC. --
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit Web Service Socket.
Mar 12 16:29:14 myhost systemd[1]: Starting Socket for Cockpit Web Service https instance factory.
Mar 12 16:29:14 myhost systemd[1]: Listening on Socket for Cockpit Web Service http instance.
Mar 12 16:29:14 myhost systemd[1]: Listening on Socket for Cockpit Web Service https instance factory.
Mar 12 16:29:14 myhost systemd[1]: Listening on Cockpit Web Service Socket.
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit motd updater service...
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit Web Service...
Mar 12 16:29:14 myhost systemd[1]: Started Cockpit Web Service.
Mar 12 16:29:14 myhost systemd[1]: cockpit-motd.service: Succeeded.
Mar 12 16:29:14 myhost systemd[1]: Started Cockpit motd updater service.
Mar 12 16:30:05 myhost systemd[1]: Starting system activity accounting tool...
Mar 12 16:30:05 myhost systemd[1]: sysstat-collect.service: Succeeded.
Mar 12 16:30:05 myhost systemd[1]: Started system activity accounting tool.
^C
[root@myhost log]#
[root@myhost log]# systemctl status cockpit
● cockpit.service - Cockpit Web Service
Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled)
Active: inactive (dead) since Tue 2024-03-12 16:30:44 UTC; 30s ago
Docs: man:cockpit-ws(8)
Process: 60134 ExecStart=/usr/libexec/cockpit-tls (code=exited, status=0/SUCCESS)
Process: 60127 ExecStartPre=/usr/libexec/cockpit-certificate-ensure --for-cockpit-tls (code=exited, status=0/SUCCE>
Main PID: 60134 (code=exited, status=0/SUCCESS)
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit Web Service...
Mar 12 16:29:14 myhost systemd[1]: Started Cockpit Web Service.
Mar 12 16:30:44 myhost systemd[1]: cockpit.service: Succeeded.
[root@myhost log]#
[root@myhost log]# ss -tupln|grep 9090
tcp LISTEN 0 128 *:9090 *:* users:(("cockpit-tls",pid=60134,fd=3),("systemd",pid=1,fd=28))
[root@myhost log]#
Same with libvirtd. even though it "PASS" every check for virt-host-validate
[root@myhost log]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Tue 2024-03-12 16:26:45 UTC; 17min ago
Docs: man:libvirtd(8)
https://libvirt.org
Process: 59300 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
Main PID: 59300 (code=exited, status=0/SUCCESS)
Mar 12 16:24:45 myhost systemd[1]: Starting Virtualization daemon...
Mar 12 16:24:45 myhost systemd[1]: Started Virtualization daemon.
Mar 12 16:26:45 myhost systemd[1]: libvirtd.service: Succeeded.
Here are some logs:
[root@myhost log]# tail -300 messages|grep -i libvirt
Mar 12 15:38:36 myhost systemd[1]: libvirtd.service: Found left-over process 3274 (dnsmasq) in control group while starting unit. Ignoring.
Mar 12 15:38:36 myhost systemd[1]: libvirtd.service: Found left-over process 3275 (dnsmasq) in control group while starting unit. Ignoring.
Mar 12 15:38:36 myhost dnsmasq[3274]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 12 15:38:36 myhost dnsmasq-dhcp[3274]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Mar 12 15:39:50 myhost systemd[1]: libvirtd.service: Succeeded.
Mar 12 15:39:50 myhost systemd[1]: libvirtd.service: Found left-over process 3274 (dnsmasq) in control group while starting unit. Ignoring.
Mar 12 15:39:50 myhost systemd[1]: libvirtd.service: Found left-over process 3275 (dnsmasq) in control group while starting unit. Ignoring.
Mar 12 15:39:50 myhost dnsmasq[3274]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 12 15:39:50 myhost dnsmasq-dhcp[3274]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Mar 12 15:41:54 myhost systemd[1]: libvirtd.service: Succeeded.
Mar 12 15:53:32 myhost systemd[1]: libvirtd.service: Found left-over process 3274 (dnsmasq) in control group while starting unit. Ignoring.
Mar 12 15:53:32 myhost systemd[1]: libvirtd.service: Found left-over process 3275 (dnsmasq) in control group while starting unit. Ignoring.
Mar 12 15:53:33 myhost dnsmasq[3274]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 12 15:53:33 myhost dnsmasq-dhcp[3274]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Mar 12 15:53:46 myhost systemd[1]: libvirtd.service: Succeeded.
Mar 12 15:55:46 myhost systemd[1]: libvirtd.service: Succeeded.
Mar 12 16:26:45 myhost systemd[1]: libvirtd.service: Succeeded.
[root@myhost log]#
[root@myhost log]#
[root@myhost log]# tail -300 messages|grep -i cockp
Mar 12 15:55:35 myhost systemd[1]: Starting Socket for Cockpit Web Service http instance.
Mar 12 15:55:35 myhost systemd[1]: Starting Socket for Cockpit Web Service https instance factory.
Mar 12 15:55:35 myhost systemd[1]: Listening on Socket for Cockpit Web Service http instance.
Mar 12 15:55:35 myhost systemd[1]: Listening on Socket for Cockpit Web Service https instance factory.
Mar 12 15:55:35 myhost systemd[1]: Starting Cockpit Web Service...
Mar 12 15:55:35 myhost systemd[1]: Started Cockpit Web Service.
Mar 12 15:57:05 myhost systemd[1]: cockpit.service: Succeeded.
Mar 12 15:57:05 myhost systemd[1]: cockpit-wsinstance-https-factory.socket: Succeeded.
Mar 12 15:57:05 myhost systemd[1]: Closed Socket for Cockpit Web Service https instance factory.
Mar 12 15:57:05 myhost systemd[1]: cockpit-wsinstance-http.socket: Succeeded.
Mar 12 15:57:05 myhost systemd[1]: Closed Socket for Cockpit Web Service http instance.
Mar 12 16:25:55 myhost systemd[1]: Starting Socket for Cockpit Web Service https instance factory.
Mar 12 16:25:55 myhost systemd[1]: Starting Socket for Cockpit Web Service http instance.
Mar 12 16:25:55 myhost systemd[1]: Listening on Socket for Cockpit Web Service https instance factory.
Mar 12 16:25:55 myhost systemd[1]: Listening on Socket for Cockpit Web Service http instance.
Mar 12 16:25:55 myhost systemd[1]: Starting Cockpit Web Service...
Mar 12 16:25:55 myhost systemd[1]: Started Cockpit Web Service.
Mar 12 16:27:26 myhost systemd[1]: cockpit.service: Succeeded.
Mar 12 16:27:26 myhost systemd[1]: cockpit-wsinstance-https-factory.socket: Succeeded.
Mar 12 16:27:26 myhost systemd[1]: Closed Socket for Cockpit Web Service https instance factory.
Mar 12 16:27:26 myhost systemd[1]: cockpit-wsinstance-http.socket: Succeeded.
Mar 12 16:27:26 myhost systemd[1]: Closed Socket for Cockpit Web Service http instance.
Mar 12 16:28:51 myhost systemd[1]: Stopping Cockpit Web Service Socket.
Mar 12 16:28:51 myhost systemd[1]: cockpit.socket: Succeeded.
Mar 12 16:28:51 myhost systemd[1]: Closed Cockpit Web Service Socket.
Mar 12 16:29:14 myhost systemd[1]: Starting Socket for Cockpit Web Service http instance.
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit Web Service Socket.
Mar 12 16:29:14 myhost systemd[1]: Starting Socket for Cockpit Web Service https instance factory.
Mar 12 16:29:14 myhost systemd[1]: Listening on Socket for Cockpit Web Service http instance.
Mar 12 16:29:14 myhost systemd[1]: Listening on Socket for Cockpit Web Service https instance factory.
Mar 12 16:29:14 myhost systemd[1]: Listening on Cockpit Web Service Socket.
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit motd updater service...
Mar 12 16:29:14 myhost systemd[1]: Starting Cockpit Web Service...
Mar 12 16:29:14 myhost systemd[1]: Started Cockpit Web Service.
Mar 12 16:29:14 myhost systemd[1]: cockpit-motd.service: Succeeded.
Mar 12 16:29:14 myhost systemd[1]: Started Cockpit motd updater service.
Mar 12 16:30:44 myhost systemd[1]: cockpit.service: Succeeded.
Mar 12 16:30:44 myhost systemd[1]: cockpit-wsinstance-https-factory.socket: Succeeded.
Mar 12 16:30:44 myhost systemd[1]: Closed Socket for Cockpit Web Service https instance factory.
Mar 12 16:30:44 myhost systemd[1]: cockpit-wsinstance-http.socket: Succeeded.
Mar 12 16:30:44 myhost systemd[1]: Closed Socket for Cockpit Web Service http instance.
[root@myhost log]#
The loaded packages match to a working system. Default libvirt virbr is disabled and stopped.
Where should we look for logs/details on failure?
