Satellite 6: "stack depth limit exceeded" when submitting 600ESXi hypervisors/6500VMs via virt-who
Issue
We are running Satellite 6.1.1+hotfix and configured virt-who (virt-who-0.14-1.el7sat.noarch) now.
The initial test was with one clusters of a VCenter which contained 300 ESXi and 3500 VMs. That task went OK in ~1.5min (virt-who logged a Timeout while talking to subscription-manager, but the data was well received by the Satellite).
Now we wanted to submit the whole VCenter, which contains ~600 ESXi and 6500 VMs. Doing so, we get the following error in virt-who:
Error in communication with subscription manager:
Runtime Error ERROR: stack depth limit exceeded
Hint: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate. at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,157
In rhsm.log we see this:
2015-11-11 15:05:27,157 [ERROR] @virtwho.py:123 - Error in communication with subscription manager:
Traceback (most recent call last):
File "/usr/share/virt-who/virtwho.py", line 107, in send
self._sendGuestAssociation(report)
File "/usr/share/virt-who/virtwho.py", line 134, in _sendGuestAssociation
result = manager.hypervisorCheckIn(report.config, report.association, report.config.type)
File "/usr/share/virt-who/manager/subscriptionmanager/subscriptionmanager.py", line 152, in hypervisorCheckIn
return self.connection.hypervisorCheckIn(config.owner, config.env, serialized_mapping)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 781, in hypervisorCheckIn
return self.conn.request_post(url, host_guest_mapping)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 576, in request_post
return self._request("POST", method, params)
File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 483, in _request
response = conn.getresponse()
File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 228, in read
return self._read_bio(size)
File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 213, in _read_bio
return m2.ssl_read(self.ssl, size, self._timeout)
SSLTimeoutError: timed out
This is also seen on the Satellite in foreman/production.log and the PostgreSQL log. The PostgreSQL log also contains the query that was aborted:
ERROR: stack depth limit exceeded
HINT: Increase the configuration parameter "max_stack_depth" (currently 2048kB), after ensuring the platform's stack depth limit is adequate.
STATEMENT: select this_.consumer_id as y0_ from cp_consumer_guests this_ inner join cp_consumer gconsumer1_ on this_.consumer_id=gconsumer1_.id inner join cp_guest_ids_checkin checkins2_ on gconsumer1_.id=checkins2_.consumer_id where gconsumer1_.owner_id=$1 and (lower(this_.guest_id)=$2 or ... or lower(this_.guest_id)=$13169) order by checkins2_.updated desc
These elipsis mean there are 13168 "OR" statements, resulting in a total of about 400KB for the single query which makes the PostgreSQL stack checker unhappy.
As the stack limit is 8MB on x86_64, we probably could just set max_stack_depth to 4 or even 6MB and be fine for now, but this will only help as long as we do not have bigger VCenters attached.
Environment
- Red Hat Satellite 6.1.1
- virt-who
- Postgresql
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
