JBoss ON shows JBoss EAP 4 or EAP 5 application server as unavailable due to a connection failure

Solution Verified - Updated -

Issue

  • Agent can not monitor JBoss server resource
  • JON reports the following error:
The agent reported the following error on its last attempt (8/24/09, 3:21:41 PM, CEST) to connect to this resource:
    Failed to start component for resource Resource[id=502285, type=JBossAS Server, key=/opt/jboss/jboss-eap-4.3/jboss-as/server/node1, name=DSJBOSSPL JBossEAP 4.3.0.GA_CP06 node1 (0.0.0.0:1099), parent=DSJBOSSPL, version=4.3.0.GA_CP06].
  • New EAP instance was discovered and added to inventory but it is shown as down
  • JBoss server instance shows as unavailable
  • Getting error Connection failure Failed to authenticate principal=null, securityDomain=jmx-console in Dashboard when trying to see the status of a JBoss SOA-P server resource
  • Agent log may contain:
WARN  [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas5.ApplicationServerComponent)- Failed to connect to Profile Service - cause: java.lang.SecurityException: Authentication exception, principal=admin
WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Cannot start component for Resource[id=10301, type=JBossAS Server, key=/opt/jboss-eap-5.0/jboss-as/server/default, name=jbosseap.host JBoss EAP 5.0.0.GA default (jbosseap.host:1099), parent=jbosseap.host, version=5.0.0.GA] from synchronized merge due to invalid plugin config
  • Agent log may contain:
WARN  [ResourceContainer.invoker.daemon-5] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Could not establish connection to the instance [1] times.
org.mc4j.ems.connection.EmsConnectException: Connection failure Failed to authenticate principal=null, securityDomain=jmx-console
     at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
     at $Proxy55.queryNames(Unknown Source)
     at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:137)
     at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.loadConnection(JmxConnectionHelper.java:207)
     at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.getEmsConnection(JmxConnectionHelper.java:101)
     at org.rhq.plugins.jbossas5.ApplicationServerComponent.initializeEmsConnection(ApplicationServerComponent.java:553)
     at org.rhq.plugins.jbossas5.ApplicationServerComponent.start(ApplicationServerComponent.java:175)
     at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
     ...
Caused by: java.lang.SecurityException: Failed to authenticate principal=null, securityDomain=jmx-console
     at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:88)
     at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
     at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
     at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
     at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     ...
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
     at java.lang.reflect.Method.invoke(Unknown Source)
     at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)
     ... 15 more
ERROR [ResourceContainer.invoker.daemon-5] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Component attempting to access a connection that could not be loaded
  • Viewing the resource summary page the following message is displayed:
The agent reported the following error on its last attempt (2/8/10, 4:57:10 PM, GMT) to connect to this resource:
    Failed to start component for resource Resource[id=10317, type=JBossAS Server, key=/opt/jboss-soa-p.4.3.0/jboss-as/server/default, name=jbosssoap.host JBoss SOA-P 4.3.0.GA_CP02_SOA default (0.0.0.0:1099), parent=jbosssoap.host, version=4.3.0.GA_CP02_SOA].
    For more details, see the stack trace.

    Please make sure that the managed resource is running and that its connection properties are set correctly.

    Stack trace:
org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Failed to start component for resource Resource[id=10317, type=JBossAS Server, key=/opt/jboss-soa-p.4.3.0/jboss-as/server/default, name=jbosssoap.host JBoss SOA-P 4.3.0.GA_CP02_SOA default (0.0.0.0:1099), parent=jbosssoap.host, version=4.3.0.GA_CP02_SOA].
        at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1280)
        at org.rhq.core.pc.inventory.InventoryManager.refreshResourceComponentState(InventoryManager.java:2256)
        at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2057)
        at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2063)
        .. ..
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
Caused by: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Invalid JMX credentials specified for connecting to this server.
        at org.rhq.plugins.jbossas.JBossASServerComponent.start(JBossASServerComponent.java:245)
        at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        ... 3 more
Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Failed to authenticate principal=null, securityDomain=jmx-console
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
        at $Proxy62.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:137)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1030)
        at org.rhq.plugins.jbossas.JBossASServerComponent.start(JBossASServerComponent.java:237)
        ... 9 more
Caused by: java.lang.SecurityException: Failed to authenticate principal=null, securityDomain=jmx-console
  • Server throws exception in some intervals like the following:
WorkerThread#1[192.168.56.3:38327] Worker thread initialization failure
    java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:756)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:376)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:174)
Caused by: java.net.SocketTimeoutException: Read timed out
  • Agent does not start cleanly
  • When I try to start the Agent the following errors appear:
WARN  [ResourceContainer.invoker.daemon-1] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Could not establish connection to the instance [1] times.
    org.mc4j.ems.connection.EmsConnectException: Could not connect [jnp://[0:0:0:0:0:0:0:1%1]:1099] javax.naming.CommunicationException: Could not obtain connection to any of these urls: [0:0:0:0:0:0:0:1%1]:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]]]
        at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:103)
        at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.loadConnection(JmxConnectionHelper.java:205)
        at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.getEmsConnection(JmxConnectionHelper.java:101)
        at org.rhq.plugins.jbossas5.ApplicationServerComponent.initializeEmsConnection(ApplicationServerComponent.java:553)
        at org.rhq.plugins.jbossas5.ApplicationServerComponent.start(ApplicationServerComponent.java:175)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        ...
Caused by: javax.naming.CommunicationException: Could not obtain connection to any of these urls: [0:0:0:0:0:0:0:1%1]:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]]]
        at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at org.mc4j.ems.impl.jmx.connection.support.providers.JBossConnectionProvider.doConnect(JBossConnectionProvider.java:75)
        at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:99)
        ... 14 more
Caused by: javax.naming.CommunicationException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]]
        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335)
        at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
        ... 19 more
Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]
        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305)
        ... 20 more
Caused by: java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099
        at java.net.InetAddress.getAllByName(InetAddress.java:1040)
        at java.net.InetAddress.getAllByName(InetAddress.java:1020)
        at java.net.InetAddress.getByName(InetAddress.java:970)
        at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:81)
        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301)
       ... 20 more
ERROR [ResourceContainer.invoker.daemon-1] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Component attempting to access a connection that could not be loaded
  • Agent is unable to connect to managed resource due to java.net.UnknownHostException
  • Agent's JVM /resources status down in JBoss Operations Network (JON) console
  • JBoss Application Server (AS) instance shows unavailable in JON and the following warning appears in the JON Agent log:
WARN  [ResourceContainer.invoker.daemon-2506] (org.rhq.plugins.jbossas.JBossASServerComponent)- Could not establish connection to the JBoss AS instance [11] times for resource [/opt/jboss/eap/jboss-eap-4.3_CP08/jboss-as/server/default]
    org.mc4j.ems.connection.EmsConnectException: Connection failure Opname type not recognized
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
        at $Proxy99.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:138)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1112)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getServerHome(JBossASServerComponent.java:343)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailabilityNow(JBossASServerComponent.java:316)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailability(JBossASServerComponent.java:310)
        at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
        ...
Caused by: java.lang.IllegalArgumentException: Opname type not recognized
        at org.jboss.jmx.connector.invoker.AuthorizationInterceptor.invoke(AuthorizationInterceptor.java:121)
        at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:108)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        ...
        at $Proxy98.queryNames(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        ...
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)
        ... 15 more
  • Shows instances of JBoss Application Server (AS) as down
  • Availability of JBoss AS server resource is DOWN or UNAVAILABLE due to java.net.ConnectException: Connection refused
  • The following warning appears in the agent log when attempting to monitor a JBoss AS server resource:
WARN  [ResourceContainer.invoker.daemon-79] (org.rhq.plugins.jbossas.JBossASServerComponent)- Could not establish connection to the JBoss AS instance [71] times for resource [/opt/jboss/jboss-as/server//all]
    org.mc4j.ems.connection.EmsConnectException: java.rmi.ConnectException: Connection refused to host: some.host.com; nested exception is: 
    java.net.ConnectException: Connection refused
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:137)
        at $Proxy56.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:137)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1112)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getServerHome(JBossASServerComponent.java:343)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailabilityNow(JBossASServerComponent.java:316)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailability(JBossASServerComponent.java:310)
        at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
        ...
Caused by: java.rmi.ConnectException: Connection refused to host: some.host.com; nested exception is: 
    java.net.ConnectException: Connection refused
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
        at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
        at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
        ...
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)
        ... 15 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:525)
        at java.net.Socket.connect(Socket.java:475)
        at java.net.Socket.<init>(Socket.java:372)
        at java.net.Socket.<init>(Socket.java:186)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
        ... 32 more
  • Unable to manage or monitor JBoss AS servers due to connection failure
  • Agent.log may contain:
Failed to start component for resource Resource[id=132222, type=JBossAS Server, key=/opt/jboss_wbc_ep11_04/jboss-as/server/ssq, name=jboss_wbc_ep11_04:8480 ssq, parent=G1JBOSS11P_AGENT, version=5.0.0.GA].
Caused by: java.lang.RuntimeException: Naming URL 'jnp://jboss:service=Naming
            Port:***CHANGE_ME***' is not valid: Illegal character in authority at index 6: jnp://jboss:service=Naming
            Port:***CHANGE_ME***
  • JBoss instance is not available and agent.log contains:
2013-08-19 14:20:09,948 WARN  [ResourceContainer.invoker.daemon-1] (jboss.on.common.jbossas.JmxConnectionHelper)- Could not establish connection to the instance [1] times.
org.mc4j.ems.connection.EmsConnectException: Could not connect [jnp://127.0.0.1:1099] javax.naming.NameNotFoundException: jmx not bound
    at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:106)
    at org.jboss.on.common.jbossas.JmxConnectionHelper.loadConnection(JmxConnectionHelper.java:211)
    at org.jboss.on.common.jbossas.JmxConnectionHelper.getEmsConnection(JmxConnectionHelper.java:101)
    at org.rhq.plugins.jbossas5.ApplicationServerComponent.initializeEmsConnection(ApplicationServerComponent.java:505)
    at org.rhq.plugins.jbossas5.ApplicationServerComponent.start(ApplicationServerComponent.java:181)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: javax.naming.NameNotFoundException: jmx not bound
    at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
    at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
    at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
    at org.jnp.server.NamingServer.lookup(NamingServer.java:396)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
    at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:726)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
    at javax.naming.InitialContext.lookup(InitialContext.java:392)
    at org.mc4j.ems.impl.jmx.connection.support.providers.JBossConnectionProvider.doConnect(JBossConnectionProvider.java:77)
    at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:101)
    ... 14 more
  • We are not seeing any JVM metric data for our EAP server

Environment

  • Red Hat JBoss Operations Network (ON) 2.2, 2.3, 2.4, 2.4.1, 2.4.2, 3.0, 3.0.1, 3.1, 3.1.1, 3.1.2, 3.2.0
  • Red Hat JBoss Application Server (AS) or JBoss AS 5 plug-in
  • JBoss application server has been discovered and imported into inventory

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.