High CPU and heap overhead in JMX when many rars are deployed

Solution Unverified - Updated -

Issue

  • We have many (hundreds) of rars deployed. We noticed substantial increased heap usage in this case and CPU is persistently high when we connect via JMX and mbeans are never rendered when viewed via jconsole. The heap dump shows the org.jboss.as.connector.subsystems.resourceadapters.IronJacamarResources are quite large. And thread dumps show CPU is high in calls like the following:
"RMI TCP Connection(1)-127.0.0.1" #100 daemon prio=9 os_prio=0 tid=0x00007f02240c5000 nid=0x3a43 runnable [0x00007f024d6f7000]
   java.lang.Thread.State: RUNNABLE
        at java.security.AccessController.getStackAccessControlContext(Native Method)
        at java.security.AccessController.getContext(AccessController.java:820)
        at org.jboss.as.controller.SecurityActions.getCaller(SecurityActions.java:132)
        at org.jboss.as.controller.AbstractOperationContext.getCaller(AbstractOperationContext.java:1017)
        at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1284)
        at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:646)
        at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:632)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.addParentResource(ReadResourceDescriptionHandler.java:786)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:774)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getLocalResourceAddresses(ReadResourceDescriptionHandler.java:713)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.initLocalResourceAddresses(ReadResourceDescriptionHandler.java:702)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.access$200(ReadResourceDescriptionHandler.java:689)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler.doExecuteInternal(ReadResourceDescriptionHandler.java:204)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler.doExecute(ReadResourceDescriptionHandler.java:162)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler.execute(ReadResourceDescriptionHandler.java:155)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702)
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537)
        at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188)
        at org.jboss.as.jmx.model.ResourceAccessControlUtil.getResourceAccess(ResourceAccessControlUtil.java:85)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:51)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.iterate(RootResourceIterator.java:43)

Environment

  • JBoss Enterprise Application Platform (EAP) 6.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content