RHEV: Repetitive "Stats function failed" and "internal error client socket is closed" errors being reported.

Solution Verified - Updated -

Issue

  • Both of our RHEL virtualization hosts are reporting hundreds of thousands of "Stats function failed" errors to syslog, such as the following;
Apr  9 09:43:00 tron vdsm vm.Vm ERROR vmId=`9ca9d5fa-5574-4b25-ab14-cff29c406fe2`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x1c04b58>#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect#012    statsFunction()#012  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__#012    retValue = self._function(*args, **kwargs)#012  File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet#012    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)#012  File "/usr/share/vdsm/libvirtvm.py", line 515, in f#012    ret = attr(*args, **kwargs)#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper#012    ret = f(*args, **kwargs)#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats#012    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)#012libvirtError: internal error client socket is closed
  • The vdsm logs contain a repetitive error sequence;
Thread-58::ERROR::2013-04-09 12:13:49,800::utils::416::vm.Vm::(collect) vmId=`68ae3c0c-9843-4f1f-8188-c125f93569af`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0xfd8b58>
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect
    statsFunction()
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__
    retValue = self._function(*args, **kwargs)
  File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet
    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
  File "/usr/share/vdsm/libvirtvm.py", line 515, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats
    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
libvirtError: internal error client socket is closed
  • Another instance of a similar error is;
Thread-952447::ERROR::2013-04-04 13:09:09,505::vm::680::vm.Vm::(_startUnderlyingVm) vmId=`4ac5f065-9a6e-4f56-8396-f4534a437119`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 642, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 1472, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error client socket is closed
  • These also get reported in /var/log/messages.

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.1
  • Red Hat Enterprise Linux (RHEL) host 6.4
  • libvirt-0.10.2-18

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content