JON Agent is failing with error OutOfMemoryError: PermGen space or Java heap space

Solution Verified - Updated -

Issue

  • The agent seems to be running out of memory.

  • The PermGen has been increased from the default to 256m and the PermGen OutOfMemoryError (OOME) still persist.

  • The following messages appear in the agent log:

        WARN  [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.mem-low}VM health check thread has detected [VM nonheap] memory has crossed the threshold [0.9] and is low: memory-usage=[init = 272629760(266240K) used = 300211520(293175K) committed = 301236224(294176K) max = 301989888(294912K)]
        WARN  [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.gc}VM health check thread is invoking the garbage collector to see if more memory can be freed
        WARN  [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.mem-low}VM health check thread has detected [VM nonheap] memory has crossed the threshold [0.9] and is low: memory-usage=[init = 272629760(266240K) used = 300211520(293175K) committed = 301236224(294176K) max = 301989888(294912K)]
        FATAL [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.mem-problem}VM health check thread sees that memory is critically low and will try to reboot the agent
        INFO  [RHQ VM Health Check Thread] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.global-concurrency-limit-disabled}Global concurrency limit has been disabled - there is no limit to the number of incoming commands allowed
        FATAL [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.startup-error}The agent encountered an error
    
  • Agent PermGen issue

  • Several of the agents in our environment have run out of Perm space.
  • The following messages appear in the agent log:

        ERROR [RHQ Server Polling Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=myjonagent.domain.com, rhq.externalizable-strategy=AGENT, rhq.security-token=1276890967324-3432556677-9876345129753124680, rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService, invocation=NameBasedInvocation[getLatestPlugins]}]]. Cause: java.lang.Exception:java.lang.OutOfMemoryError: PermGen space -> java.lang.OutOfMemoryError:PermGen space. Cause: java.lang.Exception: java.lang.OutOfMemoryError: PermGen space
        ERROR [RHQ Agent Registration Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=myjonagent.domain.com, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService, invocation=NameBasedInvocation[registerAgent]}]]. Cause: java.lang.Exception:java.lang.OutOfMemoryError: PermGen space -> java.lang.OutOfMemoryError:PermGen space. Cause: java.lang.Exception: java.lang.OutOfMemoryError: PermGen space
        WARN  [RHQ Server Polling Thread] (enterprise.communications.command.client.ServerPollingThread)- {ServerPollingThread.server-offline}The server has gone offline; client has been told to stop sending commands
        WARN  [RHQ Agent Registration Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.agent-registration-failure}Agent failed to register with the server. retry=[true], retry interval=[60,000]. Cause: java.lang.reflect.UndeclaredThrowableException:null -> java.lang.Exception:java.lang.OutOfMemoryError: PermGen space -> java.lang.OutOfMemoryError:PermGen space. Cause: java.lang.reflect.UndeclaredThrowableException
    
  • Agent out of memory issues

  • Agent is unable to communicate with server and a message similar to the following appears in the agent log:

        ERROR [InventoryManager.discovery-1] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=myagent.mydomain.com, rhq.externalizable-strategy=AGENT, rhq.security-token=1265698764567-2016581122-2331818342121239395, rhq.timeout=1800000, rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.core.clientapi.server.discovery.DiscoveryServerService, invocation=NameBasedInvocation[mergeInventoryReport]}]]. Cause: java.lang.Exception:java.lang.OutOfMemoryError: Java heap space -> java.lang.OutOfMemoryError:Java heap space. Cause: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
    
  • Auto discovery is failing and the following appears in the agent log:

        WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Exception caught while running server discovery
        java.lang.reflect.UndeclaredThrowableException
             at $Proxy4.mergeInventoryReport(Unknown Source)
             at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:873)
             at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:121)
             at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:92)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
             at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
             ...
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
             at java.lang.Thread.run(Thread.java:595)
        Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
             at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:112)
             at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:55)
             at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
             at java.util.concurrent.FutureTask.run(FutureTask.java:123)
             ... 3 more
        Caused by: java.lang.OutOfMemoryError: Java heap space
    
  • Availability is not be reported correctly by an agent and the following appears in the agent's log:

        WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Could not transmit availability report to server
        java.lang.reflect.UndeclaredThrowableException
             at $Proxy4.mergeAvailabilityReport(Unknown Source)
             at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:832)
             at org.rhq.core.pc.inventory.AvailabilityExecutor.run(AvailabilityExecutor.java:90)
             at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
             at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
             ...
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
             at java.lang.Thread.run(Thread.java:595)
        Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
             at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:112)
             at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:55)
             at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
             at java.util.concurrent.FutureTask.run(FutureTask.java:123)
             ... 3 more
        Caused by: java.lang.OutOfMemoryError: Java heap space
    
  • Discovery is failing and the agent logs:

        WARN  [ResourceDiscoveryComponent.invoker.daemon-2] (rhq.core.pluginapi.inventory.ResourceContext)- Cannot get native process for resource [/opt/jboss/eap/jboss-eap-4.3] - discovery failed
        java.lang.Exception: Discovery component invocation failed.
             at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:283)
             at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
             ...
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
             at java.lang.Thread.run(Thread.java:595)
        Caused by: java.lang.OutOfMemoryError: Java heap space
             at com.sun.org.apache.xpath.internal.VariableStack.reset(VariableStack.java:135)
             at com.sun.org.apache.xpath.internal.VariableStack.<init>(VariableStack.java:45)
             at com.sun.org.apache.xpath.internal.XPathContext.<init>(XPathContext.java:419)
             at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.eval(XPathImpl.java:201)
             at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:275)
             at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:365)
             at org.jboss.on.common.jbossas.JmxInvokerServiceConfiguration.parseDocument(JmxInvokerServiceConfiguration.java:109)
             at org.jboss.on.common.jbossas.JmxInvokerServiceConfiguration.<init>(JmxInvokerServiceConfiguration.java:55)
             at org.jboss.on.common.jbossas.JBossASDiscoveryUtils.getJmxInvokerSecurityDomain(JBossASDiscoveryUtils.java:86)
             at org.jboss.on.common.jbossas.JBossASDiscoveryUtils.getJmxInvokerUserInfo(JBossASDiscoveryUtils.java:42)
             at org.rhq.plugins.jbossas.JBossASDiscoveryComponent.processAutoDiscoveredProcesses(JBossASDiscoveryComponent.java:194)
             at org.rhq.plugins.jbossas.JBossASDiscoveryComponent.discoverResources(JBossASDiscoveryComponent.java:89)
             at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:592)
             at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:279)
             ... 5 more
        WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [RHQ Server Communications Subsystem] Resources - failed after 589 ms.
        java.lang.Exception: Discovery component invocation failed.
             at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:283)
             at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
             ...
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
             at java.lang.Thread.run(Thread.java:595)
        Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Java heap space
             at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
             at $Proxy67.queryNames(Unknown Source)
             at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:301)
             at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:326)
             at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.performDiscovery(MBeanResourceDiscoveryComponent.java:147)
             at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:96)
             at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:84)
             at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:592)
             at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:279)
             ... 5 more
        Caused by: java.lang.OutOfMemoryError: Java heap space
    
  • Metric are not being collected for one or more resources and the agent log contains:

        ERROR [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failed to run measurement collection
        java.lang.OutOfMemoryError: Java heap space
        WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=760681, type=ConnectionFactory, key=jboss.jca:name=jms/cbsconfig/queue/MAPPNOCEVENTERR,service=ConnectionFactoryBinding, name=jms/cbsconfig/queue/MAPPNOCEVENTERR Connection Factory, parent=lxdpaa01 JBossEAP 4.3.0.GA_CP03 lxdpar01 (www.example.com:1099)] - cause: java.lang.RuntimeException:Unable to load attributes on bean [jboss.jca:name=jms/myjmsconfig/queue/EVENTERR,service=ManagedConnectionPool] Connection failure Java heap space -> org.mc4j.ems.connection.EmsConnectException:Connection failure Java heap space -> java.lang.OutOfMemoryError:Java heap space
        ERROR [ResourceContainer.invoker.daemon-3] (org.rhq.plugins.jbossas.JBossASServerComponent)- Failed to obtain measurement [jboss.system:type=ServerInfo:ActiveThreadCount]
        org.mc4j.ems.connection.EmsException: Could not load attribute value Connection failure Java heap space
             at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:235)
             at org.rhq.plugins.jbossas.JBossASServerComponent.getValues(JBossASServerComponent.java:392)
             ...
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
             at java.lang.Thread.run(Thread.java:595)
        Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Java heap space
             at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
             at $Proxy67.getAttribute(Unknown Source)
             at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:199)
             ... 10 more
        Caused by: java.lang.OutOfMemoryError: Java heap space
    

Environment

  • JBoss Operations Network (JON)

    • 2.3.1
    • 2.4
  • JON Agent managing 15 or more server resources

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content