Satellite 6: "stack depth limit exceeded" when submitting 600ESXi hypervisors/6500VMs via virt-who
Issue
We are running Satellite 6.1.1+hotfix and configured virt-who (virt-who-0.14-1.el7sat.noarch) now.
The initial test was with one clusters of a VCenter which contained 300 ESXi and 3500 VMs. That task went OK in ~1.5min (virt-who logged a Timeout while talking to subscription-manager, but the data was well received by the Satellite).
Now we wanted to submit the whole VCenter, which contains ~600 ESXi and 6500 VMs. Doing so, we get the following error in virt-who:
Error in communication with subscription manager:
Runtime Error ERROR: stack depth limit exceeded
Hint: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,157
In rhsm.log we see this:
2015-11-11 15:05:27,157 [ERROR] @virtwho.py:123 - Error in communication with subscription manager:
Traceback (most recent call last):
File "/usr/share/virt-who/virtwho.py", line 107, in send
self._sendGuestAssociation(report)
File "/usr/share/virt-who/virtwho.py", line 134, in _sendGuestAssociation
result = manager.hypervisorCheckIn(report.config, report.association, report.config.type)
File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 152, in hypervisorCheckIn
return self.connection.hypervisorCheckIn(config.owner, config.env, serialized_mapping)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 781, in hypervisorCheckIn
return self.conn.request_post(url, host_guest_mapping)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 576, in request_post
return self._request("POST", method, params)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 483, in _request
response = conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 228, in read
return self._read_bio(size)
File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 213, in _read_bio
return m2.ssl_read(self.ssl, size, self._timeout)
SSLTimeoutError: timed out
This is also seen on the Satellite in foreman/production.log and the PostgreSQL log. The PostgreSQL log also contains the query that was aborted:
ERROR: stack depth limit exceeded
HINT: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.
STATEMENT: select this_.consumer_id as y0_ from cp_consumer_guests this_ inner join cp_consumer gconsumer1_ on this_.consumer_id=gconsumer1_.id inner join cp_guest_ids_checkin checkins2_ on gconsumer1_.id=checkins2_.consumer_id where gconsumer1_.owner_id=$1 and (lower(this_.guest_id)=$2 or ... or lower(this_.guest_id)=$13169) order by checkins2_.updated desc
These elipsis mean there are 13168 "OR" statements, resulting in a total of about 400KB for the single query which makes the PostgreSQL stack checker unhappy.
As the stack limit is 8MB on x86_64, we probably could just set max_stack_depth to 4 or even 6MB and be fine for now, but this will only help as long as we do not have bigger VCenters attached.
Environment
- Red Hat Satellite 6.1.1
- virt-who
- Postgresql
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.