500 internal server error from APIs and web GUI

Issue

• Getting 500 Internal Server Errors from the api and webui like:

$ grep ' 500 ' httpd-logs/httpd/ssl_access_log| grep 11/Jan/2016:11
xxxx - - [11/Jan/2016:11:00:01 -0700] "POST /rpc/api HTTP/1.0" 500 2846

$ grep "Mon Jan 11 11" httpd-logs/httpd/ssl_error_log 
[Mon Jan 11 11:10:01 2016] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Jan 11 11:10:01 2016] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx] mod_wsgi (pid=26241): Exception occurred processing WSGI script '/usr/share/rhn/wsgi/xmlrpc.py'.
[Mon Jan 11 11:29:51 2016] [error] [client xxxx] Traceback (most recent call last):
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/share/rhn/wsgi/xmlrpc.py", line 22, in application
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     return wsgiHandler.handle(environ, start_response, "xmlrpc", "server.xmlrpc")
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/share/rhn/wsgi/wsgiHandler.py", line 37, in handle
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     ret = parseServ(req)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheHandler.py", line 80, in headerParserHandler
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     rhnSQL.initDB()
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/__init__.py", line 142, in initDB
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     __init__DB(backend, host, port, username, password, database, sslmode, sslrootcert)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/__init__.py", line 56, in __init__DB
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     __DB.connect()
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_cx_Oracle.py", line 390, in connect
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     log_debug(1, "Connecting to database", self.dbtxt)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnLog.py", line 116, in log_debug
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     LOG.logMessage(*args)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnLog.py", line 205, in logMessage
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     self.writeMessage(msg)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnLog.py", line 213, in writeMessage
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     self.writeToLog(msg)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]   File "/usr/lib/python2.6/site-packages/spacewalk/common/rhnLog.py", line 219, in writeToLog
[Mon Jan 11 11:29:51 2016] [error] [client xxxx]     self.fd.write("%s\\n" % msg)
[Mon Jan 11 11:29:51 2016] [error] [client xxxx] IOError: [Errno 28] No space left on device
[Mon Jan 11 11:29:52 2016] [error] [client xxxx] mod_wsgi (pid=26243): Exception occurred processing WSGI script '/usr/share/rhn/wsgi/xmlrpc.py'.
[Mon Jan 11 11:29:52 2016] [error] [client xxxx] Traceback (most recent call last):
[Mon Jan 11 11:29:52 2016] [error] [client xxxx]   File "/usr/share/rhn/wsgi/xmlrpc.py", line 22, in application
[Mon Jan 11 11:29:52 2016] [error] [client xxxx]     return wsgiHandler.handle(environ, start_response, "xmlrpc", "server.xmlrpc")
[Mon Jan 11 11:29:52 2016] [error] [client xxxx]   File "/usr/share/rhn/wsgi/wsgiHandler.py", line 37, in handle
[Mon Jan 11 11:29:52 2016] [error] [client xxxx]     ret = parseServ(req)