RHEV: Repetitive "Stats function failed" and "internal error client socket is closed" errors being reported.
Issue
- Both of our RHEL virtualization hosts are reporting hundreds of thousands of "Stats function failed" errors to syslog, such as the following;
Apr 9 09:43:00 tron vdsm vm.Vm ERROR vmId=`9ca9d5fa-5574-4b25-ab14-cff29c406fe2`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0x1c04b58>#012Traceback (most recent call last):#012 File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect#012 statsFunction()#012 File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__#012 retValue = self._function(*args, **kwargs)#012 File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet#012 netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)#012 File "/usr/share/vdsm/libvirtvm.py", line 515, in f#012 ret = attr(*args, **kwargs)#012 File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper#012 ret = f(*args, **kwargs)#012 File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats#012 if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)#012libvirtError: internal error client socket is closed
- The
vdsm
logs contain a repetitive error sequence;
Thread-58::ERROR::2013-04-09 12:13:49,800::utils::416::vm.Vm::(collect) vmId=`68ae3c0c-9843-4f1f-8188-c125f93569af`::Stats function failed: <AdvancedStatsFunction _sampleNet at 0xfd8b58>
Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 412, in collect
statsFunction()
File "/usr/lib64/python2.6/site-packages/vdsm/utils.py", line 287, in __call__
retValue = self._function(*args, **kwargs)
File "/usr/share/vdsm/libvirtvm.py", line 168, in _sampleNet
netSamples[nic.name] = self._vm._dom.interfaceStats(nic.name)
File "/usr/share/vdsm/libvirtvm.py", line 515, in f
ret = attr(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1867, in interfaceStats
if ret is None: raise libvirtError ('virDomainInterfaceStats() failed', dom=self)
libvirtError: internal error client socket is closed
- Another instance of a similar error is;
Thread-952447::ERROR::2013-04-04 13:09:09,505::vm::680::vm.Vm::(_startUnderlyingVm) vmId=`4ac5f065-9a6e-4f56-8396-f4534a437119`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/vm.py", line 642, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/libvirtvm.py", line 1472, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 83, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2645, in createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error client socket is closed
- These also get reported in
/var/log/messages
.
Environment
- Red Hat Enterprise Virtualization (RHEV) 3.1
- Red Hat Enterprise Linux (RHEL) host 6.4
- libvirt-0.10.2-18
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.