Web clients gets "503: Service Temporarily Unavailable" on Satellite Web service due to deadlock.

Issue

• Web clients gets "503: Service Temporarily Unavailable" on Satellite Web service due to deadlock.

• rhnServerNeededCache contained over 12 million records is the cause of the 503 errors, when attempting to navigate to the systems and errata pages.

• http error_log :

[Wed Oct 29 10:02:13 2014] [error] Exception Handler Information
[Wed Oct 29 10:02:13 2014] [error] Traceback (most recent call last):
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheRequest.py", line 123, in call_function
[Wed Oct 29 10:02:13 2014] [error]     response = apply(func, params)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/share/rhn/server/handlers/xmlrpc/registration.py", line 1178, in register_osad_jid
[Wed Oct 29 10:02:13 2014] [error]     server.register_push_client_jid(jid)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnServer/server_class.py", line 833, in register_push_client_jid
[Wed Oct 29 10:02:13 2014] [error]     ret = server_lib.update_push_client_jid(server_id, jid)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnServer/server_lib.py", line 318, in update_push_client_jid
[Wed Oct 29 10:02:13 2014] [error]     h2.execute(server_id=server_id, jid=jid)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 163, in execute
[Wed Oct 29 10:02:13 2014] [error]     return apply(self._execute_wrapper, (self._execute, ) + p, kw)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 273, in _execute_wrapper
[Wed Oct 29 10:02:13 2014] [error]     retval = apply(function, p, kw)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 217, in _execute
[Wed Oct 29 10:02:13 2014] [error]     return self._execute_(args, kwargs)
[Wed Oct 29 10:02:13 2014] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 287, in _execute_
[Wed Oct 29 10:02:13 2014] [error]     self._real_cursor.execute(self.sql, params)
[Wed Oct 29 10:02:13 2014] [error] TransactionRollbackError: deadlock detected
[Wed Oct 29 10:02:13 2014] [error] DETAIL:  Process 22581 waits for ShareLock on transaction 28818569; blocked by process 25882.
[Wed Oct 29 10:02:13 2014] [error] Process 25882 waits for ShareLock on transaction 28818570; blocked by process 22581.
[Wed Oct 29 10:02:13 2014] [error] HINT:  See server log for query details.

• taskomatic logs :

INFO   | jvm 1    | 2014/10/28 15:29:13 | SQL statement "SELECT  rhn_server.update_needed_cache( $1 )"
INFO   | jvm 1    | 2014/10/28 15:29:13 | PL/pgSQL function "update_needed_cache" line 13 at PERFORM
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.common.translation.SqlExceptionTranslator.postgreSqlException(SqlExceptionTranslator.java:54)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.common.translation.SqlExceptionTranslator.sqlException(SqlExceptionTranslator.java:44)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.common.db.datasource.CachedStatement.executeCallable(CachedStatement.java:543)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.common.db.datasource.CallableMode.execute(CallableMode.java:35)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.manager.errata.cache.ErrataCacheManager.updateErrataAndPackageCacheForChannel(ErrataCacheManager.java:392)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.manager.errata.cache.UpdateErrataCacheCommand.updateErrataCacheForChannel(UpdateErrataCacheCommand.java:180)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.taskomatic.task.errata.ErrataCacheWorker.run(ErrataCacheWorker.java:76)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2014/10/28 15:29:13 | Caused by: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
....
INFO   | jvm 1    | 2014/10/28 15:29:13 | SQL statement "SELECT  rhn_server.update_needed_cache( $1 )"
INFO   | jvm 1    | 2014/10/28 15:29:13 | PL/pgSQL function "update_needed_cache" line 13 at PERFORM
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2094)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1827)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:508)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:384)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:377)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.mchange.v2.c3p0.impl.NewProxyCallableStatement.execute(NewProxyCallableStatement.java:2706)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.common.db.NamedPreparedStatement.execute(NamedPreparedStatement.java:117)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       at com.redhat.rhn.common.db.datasource.CachedStatement.executeCallable(CachedStatement.java:528)
INFO   | jvm 1    | 2014/10/28 15:29:13 |       ... 6 more
INFO   | jvm 1    | 2014/10/28 15:29:13 | 2014-10-28 15:29:13,296 [Thread-53] ERROR com.redhat.rhn.common.db.datasource.CachedStatement - Error while processing cached statement sql: SELECT pkgcap.id, pkgcap.name, pkgcap.version, pkgp.sense
INFO   | jvm 1    | 2014/10/28 15:29:13 |     FROM rhnPackageRequires pkgp, rhnPackageCapability pkgcap
INFO   | jvm 1    | 2014/10/28 15:29:13 |    WHERE pkgp.capability_id = pkgcap.id
INFO   | jvm 1    | 2014/10/28 15:29:13 |      AND pkgp.package_id = ?
INFO   | jvm 1    | 2014/10/28 15:29:13 | com.redhat.rhn.common.db.WrappedSQLException: FATAL: terminating connection due to administrator command