Issue

• The agent seems to be running out of memory.

• The PermGen has been increased from the default to 256m and the PermGen OutOfMemoryError (OOME) still persist.

• The following messages appear in the agent log:

      WARN  [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.mem-low}VM health check thread has detected [VM nonheap] memory has crossed the threshold [0.9] and is low: memory-usage=[init = 272629760(266240K) used = 300211520(293175K) committed = 301236224(294176K) max = 301989888(294912K)]
      WARN  [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.gc}VM health check thread is invoking the garbage collector to see if more memory can be freed
      WARN  [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.mem-low}VM health check thread has detected [VM nonheap] memory has crossed the threshold [0.9] and is low: memory-usage=[init = 272629760(266240K) used = 300211520(293175K) committed = 301236224(294176K) max = 301989888(294912K)]
      FATAL [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.VMHealthCheckThread)- {VMHealthCheckThread.mem-problem}VM health check thread sees that memory is critically low and will try to reboot the agent
      INFO  [RHQ VM Health Check Thread] (org.rhq.enterprise.communications.ServiceContainer)- {ServiceContainer.global-concurrency-limit-disabled}Global concurrency limit has been disabled - there is no limit to the number of incoming commands allowed
      FATAL [RHQ VM Health Check Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.startup-error}The agent encountered an error
  

• Agent PermGen issue

• Several of the agents in our environment have run out of Perm space.

• The following messages appear in the agent log:

      ERROR [RHQ Server Polling Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=myjonagent.domain.com, rhq.externalizable-strategy=AGENT, rhq.security-token=1276890967324-3432556677-9876345129753124680, rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService, invocation=NameBasedInvocation[getLatestPlugins]}]]. Cause: java.lang.Exception:java.lang.OutOfMemoryError: PermGen space -> java.lang.OutOfMemoryError:PermGen space. Cause: java.lang.Exception: java.lang.OutOfMemoryError: PermGen space
      ERROR [RHQ Agent Registration Thread] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=myjonagent.domain.com, rhq.externalizable-strategy=AGENT, rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.core.clientapi.server.core.CoreServerService, invocation=NameBasedInvocation[registerAgent]}]]. Cause: java.lang.Exception:java.lang.OutOfMemoryError: PermGen space -> java.lang.OutOfMemoryError:PermGen space. Cause: java.lang.Exception: java.lang.OutOfMemoryError: PermGen space
      WARN  [RHQ Server Polling Thread] (enterprise.communications.command.client.ServerPollingThread)- {ServerPollingThread.server-offline}The server has gone offline; client has been told to stop sending commands
      WARN  [RHQ Agent Registration Thread] (org.rhq.enterprise.agent.AgentMain)- {AgentMain.agent-registration-failure}Agent failed to register with the server. retry=[true], retry interval=[60,000]. Cause: java.lang.reflect.UndeclaredThrowableException:null -> java.lang.Exception:java.lang.OutOfMemoryError: PermGen space -> java.lang.OutOfMemoryError:PermGen space. Cause: java.lang.reflect.UndeclaredThrowableException
  

• Agent out of memory issues

• Agent is unable to communicate with server and a message similar to the following appears in the agent log:

      ERROR [InventoryManager.discovery-1] (enterprise.communications.command.client.ClientCommandSenderTask)- {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.agent-name=myagent.mydomain.com, rhq.externalizable-strategy=AGENT, rhq.security-token=1265698764567-2016581122-2331818342121239395, rhq.timeout=1800000, rhq.send-throttle=true}]; params=[{targetInterfaceName=org.rhq.core.clientapi.server.discovery.DiscoveryServerService, invocation=NameBasedInvocation[mergeInventoryReport]}]]. Cause: java.lang.Exception:java.lang.OutOfMemoryError: Java heap space -> java.lang.OutOfMemoryError:Java heap space. Cause: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
  

• Auto discovery is failing and the following appears in the agent log:

      WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Exception caught while running server discovery
      java.lang.reflect.UndeclaredThrowableException
           at $Proxy4.mergeInventoryReport(Unknown Source)
           at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:873)
           at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:121)
           at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.run(AutoDiscoveryExecutor.java:92)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
           at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
           ...
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
           at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
           at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:112)
           at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:55)
           at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
           at java.util.concurrent.FutureTask.run(FutureTask.java:123)
           ... 3 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
  

• Availability is not be reported correctly by an agent and the following appears in the agent's log:

      WARN  [InventoryManager.availability-1] (rhq.core.pc.inventory.InventoryManager)- Could not transmit availability report to server
      java.lang.reflect.UndeclaredThrowableException
           at $Proxy4.mergeAvailabilityReport(Unknown Source)
           at org.rhq.core.pc.inventory.InventoryManager.handleReport(InventoryManager.java:832)
           at org.rhq.core.pc.inventory.AvailabilityExecutor.run(AvailabilityExecutor.java:90)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
           at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
           ...
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
           at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
           at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:112)
           at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:55)
           at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
           at java.util.concurrent.FutureTask.run(FutureTask.java:123)
           ... 3 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
  

• Discovery is failing and the agent logs:

      WARN  [ResourceDiscoveryComponent.invoker.daemon-2] (rhq.core.pluginapi.inventory.ResourceContext)- Cannot get native process for resource [/opt/jboss/eap/jboss-eap-4.3] - discovery failed
      java.lang.Exception: Discovery component invocation failed.
           at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:283)
           at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
           ...
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
           at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.OutOfMemoryError: Java heap space
           at com.sun.org.apache.xpath.internal.VariableStack.reset(VariableStack.java:135)
           at com.sun.org.apache.xpath.internal.VariableStack.<init>(VariableStack.java:45)
           at com.sun.org.apache.xpath.internal.XPathContext.<init>(XPathContext.java:419)
           at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.eval(XPathImpl.java:201)
           at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:275)
           at com.sun.org.apache.xpath.internal.jaxp.XPathImpl.evaluate(XPathImpl.java:365)
           at org.jboss.on.common.jbossas.JmxInvokerServiceConfiguration.parseDocument(JmxInvokerServiceConfiguration.java:109)
           at org.jboss.on.common.jbossas.JmxInvokerServiceConfiguration.<init>(JmxInvokerServiceConfiguration.java:55)
           at org.jboss.on.common.jbossas.JBossASDiscoveryUtils.getJmxInvokerSecurityDomain(JBossASDiscoveryUtils.java:86)
           at org.jboss.on.common.jbossas.JBossASDiscoveryUtils.getJmxInvokerUserInfo(JBossASDiscoveryUtils.java:42)
           at org.rhq.plugins.jbossas.JBossASDiscoveryComponent.processAutoDiscoveredProcesses(JBossASDiscoveryComponent.java:194)
           at org.rhq.plugins.jbossas.JBossASDiscoveryComponent.discoverResources(JBossASDiscoveryComponent.java:89)
           at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:592)
           at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:279)
           ... 5 more
      WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Failure during discovery for [RHQ Server Communications Subsystem] Resources - failed after 589 ms.
      java.lang.Exception: Discovery component invocation failed.
           at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:283)
           at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
           ...
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
           at java.lang.Thread.run(Thread.java:595)
      Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Java heap space
           at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
           at $Proxy67.queryNames(Unknown Source)
           at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:301)
           at org.mc4j.ems.impl.jmx.connection.DConnection.queryBeans(DConnection.java:326)
           at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.performDiscovery(MBeanResourceDiscoveryComponent.java:147)
           at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:96)
           at org.rhq.plugins.jmx.MBeanResourceDiscoveryComponent.discoverResources(MBeanResourceDiscoveryComponent.java:84)
           at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:592)
           at org.rhq.core.pc.util.DiscoveryComponentProxyFactory$ComponentInvocationThread.call(DiscoveryComponentProxyFactory.java:279)
           ... 5 more
      Caused by: java.lang.OutOfMemoryError: Java heap space
  

• Metric are not being collected for one or more resources and the agent log contains:

      ERROR [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failed to run measurement collection
      java.lang.OutOfMemoryError: Java heap space
      WARN  [MeasurementManager.collector-1] (rhq.core.pc.measurement.MeasurementCollectorRunner)- Failure to collect measurement data for Resource[id=760681, type=ConnectionFactory, key=jboss.jca:name=jms/cbsconfig/queue/MAPPNOCEVENTERR,service=ConnectionFactoryBinding, name=jms/cbsconfig/queue/MAPPNOCEVENTERR Connection Factory, parent=lxdpaa01 JBossEAP 4.3.0.GA_CP03 lxdpar01 (www.example.com:1099)] - cause: java.lang.RuntimeException:Unable to load attributes on bean [jboss.jca:name=jms/myjmsconfig/queue/EVENTERR,service=ManagedConnectionPool] Connection failure Java heap space -> org.mc4j.ems.connection.EmsConnectException:Connection failure Java heap space -> java.lang.OutOfMemoryError:Java heap space
      ERROR [ResourceContainer.invoker.daemon-3] (org.rhq.plugins.jbossas.JBossASServerComponent)- Failed to obtain measurement [jboss.system:type=ServerInfo:ActiveThreadCount]
      org.mc4j.ems.connection.EmsException: Could not load attribute value Connection failure Java heap space
           at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:235)
           at org.rhq.plugins.jbossas.JBossASServerComponent.getValues(JBossASServerComponent.java:392)
           ...
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
           at java.lang.Thread.run(Thread.java:595)
      Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Java heap space
           at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
           at $Proxy67.getAttribute(Unknown Source)
           at org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute.refresh(DAttribute.java:199)
           ... 10 more
      Caused by: java.lang.OutOfMemoryError: Java heap space