Cannot use rhevm-logcollector for the SPM host because "No space left on device" on RHEV-M under /tmp

Solution Verified - Updated -

Environment

Red Hat Enterprise Virtualization 3.x

Issue

Cannot run rhevm-logcollector for the SPM host because the rhev-m host ran out of disk space in /tmp.
Tried using --local-tmp to /var/tmp but the compression still goes to /tmp.

Resolution

Increase the lv on which /tmp is located.

# lvextend -r -L+10G vg_rhevm/tmp

In order to check how much available space is on the disk, run vgs command and check for VFree value:

  VG       Attr   Ext   #PV #LV #SN VSize   VFree  VG UUID                               
  vg_rhevm wz--n- 4.00m   2   4   0 124.50g 34.00g AQQi8a-PRcM-N1UG-DMrh-syp1-37Rq-MSZ09n

Root Cause

Checking log collector logs: /var/log/ovirt-engine/ovirt-log-collector/ovirt-log-collector-XXXXX.log, we can see, rhev-m box is running out of space on /tmp:

2014-07-31 09:04:33::ERROR::rhevm-log-collector::750::root:: Failed to collect logs from: X.X.X.X; [Errno 28] No space left on device: '/tmp/logcollector-B_P8jl/log-collector-data/X.X.X.X'

From df output:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_rhevm-root
                      58443988   4532968  50942240   9% /
proc                         0         0         0   -  /proc
sysfs                        0         0         0   -  /sys
devpts                       0         0         0   -  /dev/pts
tmpfs                  6099368         0   6099368   0% /dev/shm
/dev/sda1               495844     52759    417485  12% /boot
/dev/mapper/vg_rhevm-tmp
                       4128448    251744   3666992   7% /tmp
/dev/mapper/vg_rhevm-var
                      24770940  11076108  12436748  48% /var
none                         0         0         0   -  /proc/sys/fs/binfmt_misc
sunrpc                       0         0         0   -  /var/lib/nfs/rpc_pipefs
nfsd                         0         0         0   -  /proc/fs/nfsd

we can see that /tmp is a separate lv, which is about 4G size.
And in this case LC from rhev-m only is of 3G size.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.