Restarting nova_libvirt container does not clean up pid files

Issue

• Following an overcloud deploy on an already deployed stack, nova_libvirt fails to come up. This is what is described in BZ.

• nova_libvirt refuses to start with:

# tail /var/log/containers/libvirt/libvirtd.log
2022-04-07 22:31:12.267+0000: 395157: info : libvirt version: 7.0.0, package: 14.6.module+el8.4.0+13801+378af433 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-01-10-10:42:14, )
2022-04-07 22:31:12.267+0000: 395157: info : hostname: overcloud-compute-0
2022-04-07 22:31:12.267+0000: 395157: error : virPidFileAcquirePath:369 : Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable

# systemctl status tripleo_nova_libvirt.service
● tripleo_nova_libvirt.service - nova_libvirt container
   Loaded: loaded (/etc/systemd/system/tripleo_nova_libvirt.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2022-04-08 00:31:12 CEST; 5h 48min ago
  Process: 395229 ExecStopPost=/usr/bin/podman stop -t 10 nova_libvirt (code=exited, status=0/SUCCESS)
  Process: 395073 ExecStart=/usr/libexec/tripleo-start-podman-container nova_libvirt (code=exited, status=1/FAILURE)
 Main PID: 393879 (code=exited, status=1/FAILURE)

Apr 08 00:31:12 overcloud-compute-0 systemd[1]: tripleo_nova_libvirt.service: Service RestartSec=100ms expired, scheduling restart.
Apr 08 00:31:12 overcloud-compute-0 systemd[1]: tripleo_nova_libvirt.service: Scheduled restart job, restart counter is at 695.
Apr 08 00:31:12 overcloud-compute-0 systemd[1]: Stopped nova_libvirt container.
Apr 08 00:31:12 overcloud-compute-0 systemd[1]: tripleo_nova_libvirt.service: Start request repeated too quickly.
Apr 08 00:31:12 overcloud-compute-0 systemd[1]: tripleo_nova_libvirt.service: Failed with result 'exit-code'.
Apr 08 00:31:12 overcloud-compute-0 systemd[1]: Failed to start nova_libvirt container.

• The bz was closed as not a bug. However we note that tripleo itself does not seem to respect the comment on the bz:

>  Restarting containers with podman is not supported

Apr 07 16:37:35 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG: podman stop nova_libvirt
Apr 07 16:37:35 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG STDOUT: 2baf123a2375c9fa3c2d6cc6b5bf2c2ba18c8b218192d82d01c433f019049156
Apr 07 16:37:35 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG STDERR: time="2022-04-07T16:37:35+02:00" level=warning msg="lstat /sys/fs/cgroup/devices/machine.slice/libpod-2baf123a2375c9fa3c2d6cc6b5bf2>
                                                                  time="2022-04-07T16:37:35+02:00" level=error msg="Failed to remove paths: map[cpuset:/sys/fs/cgroup/cpuset/machine.slice/libpod-2baf123a2375c9fa3c2d6cc6b5bf2c2ba1>
Apr 07 16:37:35 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG RC: 0
Apr 07 16:37:35 overcloud-compute-0 podman[977361]: 2022-04-07 16:37:35.934838379 +0200 CEST m=+0.094436575 container cleanup 2baf123a2375c9fa3c2d6cc6b5bf2c2ba18c8b218192d82d01c433f019049156 (image=undercloud.ctlplane.localdomain.>
Apr 07 16:37:35 overcloud-compute-0 podman[977361]: 2baf123a2375c9fa3c2d6cc6b5bf2c2ba18c8b218192d82d01c433f019049156
Apr 07 16:37:36 overcloud-compute-0 podman[977358]: 2022-04-07 16:37:36.004782073 +0200 CEST m=+0.180680392 container remove 2baf123a2375c9fa3c2d6cc6b5bf2c2ba18c8b218192d82d01c433f019049156 (image=undercloud.ctlplane.localdomain>
Apr 07 16:37:36 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG: podman rm -f nova_libvirt
Apr 07 16:37:36 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG STDOUT: 2baf123a2375c9fa3c2d6cc6b5bf2c2ba18c8b218192d82d01c433f019049156
Apr 07 16:37:36 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG STDERR:
Apr 07 16:37:36 overcloud-compute-0 ansible-podman_container[977083]: PODMAN-CONTAINER-DEBUG RC: 0
Apr 07 16:37:36 overcloud-compute-0 podman[977451]: Error: no container with name or ID nova_libvirt found: no such container

• As a result, nova_libvirt is not running.