LEAPP Upgrade from RHEL 7 to 8 fails for lack of space

Latest response

I'm trying to perform an in-place upgrade of a RHEL7 system with RHEL8. At the end of the upgrade process, leapp reports:

Error Summary
-------------
Disk Requirements:
   At least 1645MB more space needed on the / filesystem.

Yet that file system appears to have plenty of free space:

[root@linux1x Temp]# df -h /
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/rhel_linux1t-root  916G   76G  840G   9% /

Any suggestions on what to look at next?

Attachments

Responses

I suspect the line

Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system

might be important, as it suggests to me that it is where leapp is actually trying to install the packages and it is failing as it is read-only. Are you expecting this to be read-only?

Not at all. After the failed attempt to upgrade, I found the path "/var/lib/leapp/el8userspace" did exist. I've created the full path to selinux and am re-running the upgrade attempt.

FWIW, that full path should live entirely in a read-write filesystem:

Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                        16G     0   16G   0% /dev
tmpfs                           16G     0   16G   0% /dev/shm
tmpfs                           16G   18M   16G   1% /run
tmpfs                           16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/rhel_linux1t-root  916G   75G  841G   9% /
/dev/sda2                      492M  257M  236M  53% /boot
/dev/sda1                      200M   11M  190M   6% /boot/efi
tmpfs                          3.2G   12K  3.2G   1% /run/user/42
tmpfs                          3.2G     0  3.2G   0% /run/user/2001
tmpfs                          3.2G     0  3.2G   0% /run/user/0

Not at all. After the failed attempt to upgrade, I found the path "/var/lib/leapp/el8userspace" did exist. I've created the full path to selinux and am re-running the upgrade attempt.

After a bit deeper dive into LEAPP, I discovered that during the upgrade process several temporary filesystems are created and mounted on existing directories in the underlying root filesystem. I believe the lack of filespace is reference to space on these temporary filesystes, not space issues with the filesystems that existed at the time LEAPP was invoked.

In my case, here are the mounts that exist before starting LEAPP, along with the "df" statistics for those filesystems:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=16306032k,nr_inodes=4076508,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/rhel_linux1t-root on / type xfs (rw,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9875)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda2 on /boot type xfs (rw,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,size=3264588k,mode=700,uid=42,gid=42)
tmpfs on /run/user/2001 type tmpfs (rw,nosuid,nodev,relatime,size=3264588k,mode=700,uid=2001,gid=502)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3264588k,mode=700)
gvfsd-fuse on /run/user/0/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0)
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                        16G     0   16G   0% /dev
tmpfs                           16G     0   16G   0% /dev/shm
tmpfs                           16G   26M   16G   1% /run
tmpfs                           16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/rhel_linux1t-root  916G   75G  841G   9% /
/dev/sda2                      492M  257M  236M  53% /boot
/dev/sda1                      200M   11M  190M   6% /boot/efi
tmpfs                          3.2G   12K  3.2G   1% /run/user/42
tmpfs                          3.2G     0  3.2G   0% /run/user/2001
tmpfs                          3.2G     0  3.2G   0% /run/user/0

When LEAPP gets to the point where RHEL8 RPMs are being downloaded and installed, these new mounts are created:

/var/lib/leapp/scratch/diskimages/root_ on /var/lib/leapp/scratch/mounts/root_ type ext4 (rw,relatime,data=ordered)
overlay2 on /var/lib/leapp/scratch/mounts/root_/system_overlay type overlay (rw,relatime,lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work)
overlay2 on /var/lib/leapp/el8userspace/installroot type overlay (rw,relatime,lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work)
/var/lib/leapp/scratch/diskimages/root_boot on /var/lib/leapp/scratch/mounts/root_boot type ext4 (rw,relatime,data=ordered)
overlay2 on /var/lib/leapp/scratch/mounts/root_boot/root_boot type overlay (rw,relatime,lowerdir=/boot,upperdir=/var/lib/leapp/scratch/mounts/root_boot/upper,workdir=/var/lib/leapp/scratch/mounts/root_boot/work)
overlay2 on /var/lib/leapp/scratch/mounts/root_/system_overlay/boot type overlay (rw,relatime,lowerdir=/boot,upperdir=/var/lib/leapp/scratch/mounts/root_boot/upper,workdir=/var/lib/leapp/scratch/mounts/root_boot/work)
overlay2 on /var/lib/leapp/el8userspace/installroot/boot type overlay (rw,relatime,lowerdir=/boot,upperdir=/var/lib/leapp/scratch/mounts/root_boot/upper,workdir=/var/lib/leapp/scratch/mounts/root_boot/work)
/dev/mapper/rhel_linux1t-root on /var/lib/leapp/scratch/mounts/root_/system_overlay/var/cache/dnf type xfs (rw,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)
/dev/mapper/rhel_linux1t-root on /var/lib/leapp/el8userspace/installroot/var/cache/dnf type xfs (rw,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota)

And the "df" specs at this point:

/dev/mapper/rhel_linux1t-root  916G   76G  840G   9% /
/dev/loop0                     2.0G  6.2M  1.8G   1% /var/lib/leapp/scratch/mounts/root_
overlay2                       2.0G  6.2M  1.8G   1% /var/lib/leapp/el8userspace/installroot
/dev/loop1                     2.0G  6.1M  1.8G   1% /var/lib/leapp/scratch/mounts/root_boot
overlay2                       2.0G  6.1M  1.8G   1% /var/lib/leapp/el8userspace/installroot/boot

As the RPMs were being downloaded, I noticed that the amount of space used in these temp filesystems never seemed to change. But I am wondering if the size of these temp/overlay filesystems is a real limiting factor on what LEAPP is trying to install?

I 'm also encountering the same issue. May I know the work around you did to solve the problem?