kvm guests gets shutdown with 'pthread_create failed: Resource temporarily unavailable' messages under files in /var/log/libvirt/qemu/

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6
  • kvm

Issue

  • kvm guests gets shutdown with following messages under files in /var/log/libvirt/qemu/
pthread_create failed: Resource temporarily unavailable
  • Increasing nproc is not helping.
/etc/security/limits.d/90-nproc.conf 
*          hard    nproc     <value>
  • 90-nproc.conf limits are not getting set in /proc/pidof qemu-kvm/limits file.

Resolution

  • Make sure that libvirt is 0.8.7-18.el6 or higher.

  • Edit /etc/libvirt/qemu.conf and change following..

# max_processes = 0

to

max_processes = 4096
  • Stop all guests:
# service libvirt-guests stop
  • Restart libvirtd:
# service libvirtd restart
  • Start guests:
# service libvirt-guests start

  • Check cat /proc/pidof qemu-kvm/limits to see if the limits are set.

  • If Committed_AS is reaching CommitLimit with vm.overcommit_memory set to 2, try setting it to 0.

Root Cause

  • qemu-kvm process inherits the ulimit settings from libvirtd.

  • If 'init' started libvirtd at system boot, it would get the limits from init, which will be passed on to qemu-kvm. Which would be a high value.

  • If the libvirtd is started after boot then the limits from 90-nproc.conf will be set to qemu-kvm processes started after it.

  • Reaching Committed_AS reaching CommitLimit can also result in this, when vm.overcommit_memory is set to 2 .

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments