libvirt.libvirtError: Unable to write to '/sys/fs/cgroup/devices/machine/qemu-1638-instance-00014415.libvirt-qemu/devices.allow

Issue

• Volumes failed to attach to instances in our overcloud. The following error message can be seen in nova-compute logs:

/var/log/containers/nova/nova-compute.log.1:2022-05-11 12:01:24.732 7 ERROR nova.compute.manager [req-77e7e500-65f7-4ae5-ba43-785a988c95e8 cb80cc8d57064caa9f9515fa0d780987 ba59061a71424382add721c1749d3918 - default default] [instance: 5037064b-9faf-4b20-a880-7d3967659c88] Failed to attach 43b7a272-a835-4ff0-b547-7e2ac92e9ea0 at /dev/vdb: libvirt.libvirtError: Unable to write to '/sys/fs/cgroup/devices/machine/qemu-1638-instance-00014415.libvirt-qemu/devices.allow': No such file or directory

• Directory /sys/fs/cgroup/devices/machine/qemu-1638-instance-00014415.libvirt-qemu does not exist on compute node:

[root@overcloud-compute-10 ~]# ls -l /sys/fs/cgroup/devices/machine/
total 0
-rw-r--r--. 1 root root 0 May 11 15:52 cgroup.clone_children
-rw-r--r--. 1 root root 0 May 11 15:52 cgroup.procs
--w-------. 1 root root 0 May 11 16:14 devices.allow
--w-------. 1 root root 0 May 11 16:14 devices.deny
-r--r--r--. 1 root root 0 May 11 15:52 devices.list
-rw-r--r--. 1 root root 0 May 11 15:52 notify_on_release
drwxr-xr-x. 2 root root 0 May  9 21:30 qemu-1685-instance-00015543.libvirt-qemu
drwxr-xr-x. 2 root root 0 May  9 21:30 qemu-1686-instance-000155dc.libvirt-qemu
drwxr-xr-x. 2 root root 0 May  9 21:30 qemu-1687-instance-000155df.libvirt-qemu
drwxr-xr-x. 2 root root 0 May 11 08:56 qemu-1688-instance-0001574a.libvirt-qemu
-rw-r--r--. 1 root root 0 May 11 15:52 tasks

• We also see the following in /var/log/containers/libvirtd/libvirtd.log:

2022-05-11 11:56:21.002+0000: 567494: error : virCgroupSetValueRaw:535 : Unable to write to '/sys/fs/cgroup/devices/machine/qemu-1180-instance-000143fd.libvirt-qemu/devices.allow': No such file or directory
2022-05-11 14:16:14.003+0000: 567495: error : virCgroupSetValueRaw:535 : Unable to write to '/sys/fs/cgroup/devices/machine/qemu-1182-instance-0001476f.libvirt-qemu/devices.deny': No such file or directory
2022-05-11 14:16:14.003+0000: 567495: warning : qemuDomainStorageSourceAccessModify:7692 : Unable to tear down cgroup access on /dev/disk/by-id/emc-vol-69b07c46249fb70f-bd9d41b00000114b