yum thinks I don't have enough space, is not aware of mounted partition

Latest response

I ran out of space on my disk. So I installed a new partition, mounted it to /var, and the system is running fine.

But now yum thinks I don't have enough space. I'm presuming this is because it is not aware of the new partition's free space, and is just checking the main partition.

At this point, I have two options:

  • disable disk space check, knowing that eventually I will run out of space
  • remove the old /var on the main partition

I'm not sure which is more safe

Responses

Hi Martin,

How is yum monitoring disk space? Which RHEL version is being used? At first look I presume it could be because of the cached files which were mapped to old /var. You may clear out the cached yum files by running "yum clean all" and rebuild cache and check. I don't think turning of monitoring on disk usage is a good option. If you've managed to move complete contents from old /var to new /var then you may remove the old block device for /var, but be careful enough and make necessary backups before doing this.

All the best!

Thanks for the response.

It appears the disk space check is default yum behaviour, and I can edit that with /etc/yum.conf by setting diskspacecheck=0. I don't think that yum clean all would help while the new partition is mounted, though.

I will go ahead and back up and clear the old /var to clear up space. I disabled the disk space check in the meantime, and it worked; but I don't think this is a good solution in the long term

Yes, you are right, the "diskspacecheck" is by default to set to 1 which makes yum transactions to check for sufficient space before execution.

You may run "strace" with "diskspacecheck" being enabled on any yum (yum repolist) commands which would show up list of files/libraries that it would read or access while executing, so this may give an hint..

Btw, after getting /var mounted on a new block device, was there a reboot done? I also saw your later reply to RJ with "df" output, it looks strange to see "/" and "/var" being on standard disk partitions, you would have made a lvm and used which would provide the flexibility to extend later easily.

Yeah, I put /var on a different partition months ago, following some guide I now forget

Martin, please let us know the output of df -PhT - and namely /var

I ask because we just don't know how much space you've provisioned for /var - and if you have just one partition dedicated to /var or if you have additional partitions such as /var/lib, /var/log and so forth.

If you are running out of space for /var, or some other partition, this answer will help us help you

Regards

RJ

Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/vda1      xfs        10G  9.8G  233M  98% /
devtmpfs       devtmpfs  1.9G     0  1.9G   0% /dev
tmpfs          tmpfs     1.9G  4.0K  1.9G   1% /dev/shm
tmpfs          tmpfs     1.9G   17M  1.9G   1% /run
tmpfs          tmpfs     1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb1      ext4       50G  5.6G   42G  12% /var
tmpfs          tmpfs     386M     0  386M   0% /run/user/0
tmpfs          tmpfs     386M     0  386M   0% /run/user/1000

Hi Martin,

The real issue here is not /var in this specific case, but your / file system.

2 percent remaining, and namely 233M is not enough space to take the updates, and even if it is, there's apparently a safeguard (you ran into) that will avert yum from updating when your / partition is so full.

Later today I'll post something to see if you can reduce / perhaps, but if you're not heavily invested in your system, perhaps consider rebuilding the system if you have that freedom and give yourself sufficient space in the / partition so that you're not operating with a boa constrictor snake constraining your file system (figuratively speaking)

I'll post again later

Regards

RJ

Here is the result of du:

$ du -h -d1 / 2> >(grep -v "Permission denied\|cannot access") | sort -n
0       /media
0       /mnt
0       /proc
0       /srv
0       /sys
1.9G    /usr
2.0G    /opt
2.4G    /var
4.0K    /dev
4.0K    /root
7.4G    /
17M     /run
18M     /etc
44K     /home
290M    /boot
408K    /tmp

Notice that it says I used only 7.4G in / as opposed to 9.8G from the df command :\

Martin--

I can see several reasons why you might be getting discrepancies between 'du' and 'df' output.

First, you ran 'du' as a regular user, not as 'root' (sudo du ...) - especially for the /var directory, that will cause problems since many log files are restricted to root-only access (audit, /var/log/secure, etc.)

Next, there is the issue of the old contents of the /var/ directory on the root file system - if you did not erase it before you mounted the new /var file system over it, the files there are hidden from 'du' but still take up space which is reported correctly by 'df'.

Finally, 'du' and 'df' will disagree if there are deleted files which are still being held open by some process (if you 'rm' an open file, it disappears from the directory listing immediately, and thus cannot be seen by 'du', but it is not actually removed from disk (or 'df') until the last open file handle is closed). This is common with improperly-rotated log files in /var/log, but I don't think it is the main issue here.

What I have done in the past when I make similar changes (e.g. replace a directory like /var with a mounted file system) is to rename the old directory ("mv /var /var-old") then make a new empty directory for the mount point (mkdir /var && mount /var - assuming /etc/fstab is correctly updated). That way, I can keep or clean up the old directory without have to shut down to single-user or rescue mode (again) - but I can also remove the "new" /var mount & rename the -old directory if I want to go back to the old file system layout at some point in the future.

I was actually root... ʅ(;◔౪◔)ʃ

Anyway, I will probably back up and remove the old /var

Martin,

Glad you got this sorted out.

One way to make sure when you do a du against / and only acquire results for everything under / that is not mounted (such as your case, you would:

Note: The below must be done as root.

mkdir -p /slash/notmounted
mount /dev/vda1  /slash

mountpoint /slash
echo it should say it is a mountpoint at this point
du -sk /slash/* | sort -nr
echo "the last command will show the largest things under the / file system without anything being mounted such as your old /var"
echo whatever is the largest will be sorted to the top

This procedure above will show you the largest things under the / file system and will avoid other distracting mounts, and only give you the results of what is actually filling up the / file system

Regards RJ

The bind mount option can actually be great for this. Say you want to identify only what's in /, you could do:

mount -o ro, bind / /mnt

And then do your find/du/etc. against /mnt, only what's in the / filesystem will be reported. For example:

# df -PH
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-rootVol   6.3G  4.4G  1.6G  75% /
devtmpfs                         4.1G     0  4.1G   0% /dev
tmpfs                            4.2G     0  4.2G   0% /dev/shm
tmpfs                            4.2G  590k  4.1G   1% /run
tmpfs                            4.2G     0  4.2G   0% /sys/fs/cgroup
tmpfs                            4.2G  275k  4.1G   1% /tmp
/dev/mapper/VolGroup00-varVol    2.1G  346M  1.6G  18% /var
/dev/xvda1                       475M  211M  235M  48% /boot
/dev/mapper/VolGroup00-homeVol   1.1G  2.7M  951M   1% /home
/dev/mapper/VolGroup00-logVol    2.1G   34M  1.9G   2% /var/log
/dev/mapper/VolGroup00-auditVol  8.9G   40M  8.4G   1% /var/log/audit
/dev/xvdf                         32G  1.4G   29G   5% /var/application
tmpfs                            821M     0  821M   0% /run/user/1000
# mount -o ro,bind / /mnt
# find /mnt/var
/mnt/var
#

No risk to / (due to the ro mounting). No errors associated with trying to mount a partition twice (used to be you could mount things twice - protecting with ro mount-option but seems that RHEL 7 protects against that). The bind-mounting won't include any filesystems mounted into the bind-source. Thus, your du command should report a fairly accurate number (and, minus the ro option, it used to be a way you could clean out non-empty mountpoints hidden by mounted filesystems).

I can't edit the last post...

The purpose of making a directory named /slash/notmounted is when it is not mounted, you will see a directory named "notmounted" and when it is mounted, you'll see the contents of the mounted file system. Note the command "mountpoint" and check it's man page.

It is more common than you'd realize to have a new mount point cover up the unmounted file system (in your case the old /var) which is what is actually filling up the / file system. The du procedure above examines everything and would look at what is (in your case) under the /var file system before it was mounted with the new file system you created.

Glad you got it sorted out

Regards RJ

doing procedures against /var or another directory under / might be good to do under single user mode. You might find it difficult in some cases to move certain directories that are in use until you are in single user mode. In this case you could run isolate rescue.target to get the old file system out of the way in the rescue mode.

That's a good explanation RJ :)

Thanks Sadashiva Murthy

Martin, let us know if you need any other assistance

Regards

RJ

Run the following command to know the actual utilization of / (/root) partition. cd / du -cmkx|sort -n Last one few lines show you the real picture where / is utilized. Remove some logs and other things from that place.

Thanks and Regards, Kamal Kishore

Nice tip there Tom

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.