Issue

• Both of our RHEL virtualization hosts are reporting hundreds of thousands of "Stats function failed" errors to syslog, such as the following;

Apr  9 09:43:00 tron vdsm vm.Vm ERROR vmId=`9ca9d5fa-5574-4b25-ab14-cff29c406fe2`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x1c04b58>#012Traceback (most recent call last):#012  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect#012    statsFunction()#012  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__#012    retValue = self._function(*args, **kwargs)#012  File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet#012    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)#012  File "/usr/share/vdsm/libvirtvm.py", line 515, in f#012    ret = attr(*args, **kwargs)#012  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper#012    ret = f(*args, **kwargs)#012  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats#012    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)#012libvirtError: internal error client socket is closed

• The vdsm logs contain a repetitive error sequence;

Thread-58::ERROR::2013-04-09 12:13:49,800::utils::416::vm.Vm::(collect) vmId=`68ae3c0c-9843-4f1f-8188-c125f93569af`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0xfd8b58>
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect
    statsFunction()
  File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__
    retValue = self._function(*args, **kwargs)
  File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet
    netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
  File "/usr/share/vdsm/libvirtvm.py", line 515, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats
    if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
libvirtError: internal error client socket is closed

• Another instance of a similar error is;

Thread-952447::ERROR::2013-04-04 13:09:09,505::vm::680::vm.Vm::(_startUnderlyingVm) vmId=`4ac5f065-9a6e-4f56-8396-f4534a437119`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 642, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 1472, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error client socket is closed

• These also get reported in /var/log/messages.