JBoss ON shows JBoss EAP 4 or EAP 5 application server as unavailable due to a connection failure

Issue

• Agent can not monitor JBoss server resource
• JON reports the following error:

The agent reported the following error on its last attempt (8/24/09, 3:21:41 PM, CEST) to connect to this resource:
    Failed to start component for resource Resource[id=502285, type=JBossAS Server, key=/opt/jboss/jboss-eap-4.3/jboss-as/server/node1, name=DSJBOSSPL JBossEAP 4.3.0.GA_CP06 node1 (0.0.0.0:1099), parent=DSJBOSSPL, version=4.3.0.GA_CP06].

• New EAP instance was discovered and added to inventory but it is shown as down
• JBoss server instance shows as unavailable
• Getting error Connection failure Failed to authenticate principal=null, securityDomain=jmx-console in Dashboard when trying to see the status of a JBoss SOA-P server resource
• Agent log may contain:

WARN  [ResourceContainer.invoker.daemon-1] (org.rhq.plugins.jbossas5.ApplicationServerComponent)- Failed to connect to Profile Service - cause: java.lang.SecurityException: Authentication exception, principal=admin
WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Cannot start component for Resource[id=10301, type=JBossAS Server, key=/opt/jboss-eap-5.0/jboss-as/server/default, name=jbosseap.host JBoss EAP 5.0.0.GA default (jbosseap.host:1099), parent=jbosseap.host, version=5.0.0.GA] from synchronized merge due to invalid plugin config

• Agent log may contain:

WARN  [ResourceContainer.invoker.daemon-5] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Could not establish connection to the instance [1] times.
org.mc4j.ems.connection.EmsConnectException: Connection failure Failed to authenticate principal=null, securityDomain=jmx-console
     at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
     at $Proxy55.queryNames(Unknown Source)
     at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:137)
     at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.loadConnection(JmxConnectionHelper.java:207)
     at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.getEmsConnection(JmxConnectionHelper.java:101)
     at org.rhq.plugins.jbossas5.ApplicationServerComponent.initializeEmsConnection(ApplicationServerComponent.java:553)
     at org.rhq.plugins.jbossas5.ApplicationServerComponent.start(ApplicationServerComponent.java:175)
     at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
     ...
Caused by: java.lang.SecurityException: Failed to authenticate principal=null, securityDomain=jmx-console
     at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:88)
     at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
     at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
     at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
     at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:180)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     ...
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
     at java.lang.reflect.Method.invoke(Unknown Source)
     at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)
     ... 15 more
ERROR [ResourceContainer.invoker.daemon-5] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Component attempting to access a connection that could not be loaded

• Viewing the resource summary page the following message is displayed:

The agent reported the following error on its last attempt (2/8/10, 4:57:10 PM, GMT) to connect to this resource:
    Failed to start component for resource Resource[id=10317, type=JBossAS Server, key=/opt/jboss-soa-p.4.3.0/jboss-as/server/default, name=jbosssoap.host JBoss SOA-P 4.3.0.GA_CP02_SOA default (0.0.0.0:1099), parent=jbosssoap.host, version=4.3.0.GA_CP02_SOA].
    For more details, see the stack trace.

    Please make sure that the managed resource is running and that its connection properties are set correctly.

    Stack trace:
org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Failed to start component for resource Resource[id=10317, type=JBossAS Server, key=/opt/jboss-soa-p.4.3.0/jboss-as/server/default, name=jbosssoap.host JBoss SOA-P 4.3.0.GA_CP02_SOA default (0.0.0.0:1099), parent=jbosssoap.host, version=4.3.0.GA_CP02_SOA].
        at org.rhq.core.pc.inventory.InventoryManager.activateResource(InventoryManager.java:1280)
        at org.rhq.core.pc.inventory.InventoryManager.refreshResourceComponentState(InventoryManager.java:2256)
        at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2057)
        at org.rhq.core.pc.inventory.InventoryManager.processSyncInfo(InventoryManager.java:2063)
        .. ..
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
Caused by: org.rhq.core.pluginapi.inventory.InvalidPluginConfigurationException: Invalid JMX credentials specified for connecting to this server.
        at org.rhq.plugins.jbossas.JBossASServerComponent.start(JBossASServerComponent.java:245)
        at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:525)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        ... 3 more
Caused by: org.mc4j.ems.connection.EmsConnectException: Connection failure Failed to authenticate principal=null, securityDomain=jmx-console
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
        at $Proxy62.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:137)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1030)
        at org.rhq.plugins.jbossas.JBossASServerComponent.start(JBossASServerComponent.java:237)
        ... 9 more
Caused by: java.lang.SecurityException: Failed to authenticate principal=null, securityDomain=jmx-console

• Server throws exception in some intervals like the following:

WorkerThread#1[192.168.56.3:38327] Worker thread initialization failure
    java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:756)
        at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:376)
        at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:174)
Caused by: java.net.SocketTimeoutException: Read timed out

• Agent does not start cleanly
• When I try to start the Agent the following errors appear:

WARN  [ResourceContainer.invoker.daemon-1] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Could not establish connection to the instance [1] times.
    org.mc4j.ems.connection.EmsConnectException: Could not connect [jnp://[0:0:0:0:0:0:0:1%1]:1099] javax.naming.CommunicationException: Could not obtain connection to any of these urls: [0:0:0:0:0:0:0:1%1]:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]]]
        at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:103)
        at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.loadConnection(JmxConnectionHelper.java:205)
        at org.rhq.plugins.jbossas5.helper.JmxConnectionHelper.getEmsConnection(JmxConnectionHelper.java:101)
        at org.rhq.plugins.jbossas5.ApplicationServerComponent.initializeEmsConnection(ApplicationServerComponent.java:553)
        at org.rhq.plugins.jbossas5.ApplicationServerComponent.start(ApplicationServerComponent.java:175)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        ...
Caused by: javax.naming.CommunicationException: Could not obtain connection to any of these urls: [0:0:0:0:0:0:0:1%1]:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]]]
        at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at org.mc4j.ems.impl.jmx.connection.support.providers.JBossConnectionProvider.doConnect(JBossConnectionProvider.java:75)
        at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:99)
        ... 14 more
Caused by: javax.naming.CommunicationException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]]
        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335)
        at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
        ... 19 more
Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server [0:0:0:0:0:0:0:1%1]:1099:1099 [Root exception is java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099]
        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305)
        ... 20 more
Caused by: java.net.UnknownHostException: [0:0:0:0:0:0:0:1%1]:1099
        at java.net.InetAddress.getAllByName(InetAddress.java:1040)
        at java.net.InetAddress.getAllByName(InetAddress.java:1020)
        at java.net.InetAddress.getByName(InetAddress.java:970)
        at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:81)
        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301)
       ... 20 more
ERROR [ResourceContainer.invoker.daemon-1] (rhq.plugins.jbossas5.helper.JmxConnectionHelper)- Component attempting to access a connection that could not be loaded

• Agent is unable to connect to managed resource due to java.net.UnknownHostException
• Agent's JVM /resources status down in JBoss Operations Network (JON) console
• JBoss Application Server (AS) instance shows unavailable in JON and the following warning appears in the JON Agent log:

WARN  [ResourceContainer.invoker.daemon-2506] (org.rhq.plugins.jbossas.JBossASServerComponent)- Could not establish connection to the JBoss AS instance [11] times for resource [/opt/jboss/eap/jboss-eap-4.3_CP08/jboss-as/server/default]
    org.mc4j.ems.connection.EmsConnectException: Connection failure Opname type not recognized
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:160)
        at $Proxy99.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:138)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1112)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getServerHome(JBossASServerComponent.java:343)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailabilityNow(JBossASServerComponent.java:316)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailability(JBossASServerComponent.java:310)
        at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
        ...
Caused by: java.lang.IllegalArgumentException: Opname type not recognized
        at org.jboss.jmx.connector.invoker.AuthorizationInterceptor.invoke(AuthorizationInterceptor.java:121)
        at org.jboss.jmx.connector.invoker.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:108)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        ...
        at $Proxy98.queryNames(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        ...
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)
        ... 15 more

• Shows instances of JBoss Application Server (AS) as down
• Availability of JBoss AS server resource is DOWN or UNAVAILABLE due to java.net.ConnectException: Connection refused
• The following warning appears in the agent log when attempting to monitor a JBoss AS server resource:

WARN  [ResourceContainer.invoker.daemon-79] (org.rhq.plugins.jbossas.JBossASServerComponent)- Could not establish connection to the JBoss AS instance [71] times for resource [/opt/jboss/jboss-as/server//all]
    org.mc4j.ems.connection.EmsConnectException: java.rmi.ConnectException: Connection refused to host: some.host.com; nested exception is: 
    java.net.ConnectException: Connection refused
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:137)
        at $Proxy56.queryNames(Unknown Source)
        at org.mc4j.ems.impl.jmx.connection.DConnection.loadSynchronous(DConnection.java:137)
        at org.rhq.plugins.jbossas.JBossASServerComponent.loadConnection(JBossASServerComponent.java:1112)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getServerHome(JBossASServerComponent.java:343)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailabilityNow(JBossASServerComponent.java:316)
        at org.rhq.plugins.jbossas.JBossASServerComponent.getAvailability(JBossASServerComponent.java:310)
        at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
        ...
Caused by: java.rmi.ConnectException: Connection refused to host: some.host.com; nested exception is: 
    java.net.ConnectException: Connection refused
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
        at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
        at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
        ...
        at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.GenericMBeanServerProxy.invoke(GenericMBeanServerProxy.java:123)
        ... 15 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:525)
        at java.net.Socket.connect(Socket.java:475)
        at java.net.Socket.<init>(Socket.java:372)
        at java.net.Socket.<init>(Socket.java:186)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
        ... 32 more

• Unable to manage or monitor JBoss AS servers due to connection failure
• Agent.log may contain:

Failed to start component for resource Resource[id=132222, type=JBossAS Server, key=/opt/jboss_wbc_ep11_04/jboss-as/server/ssq, name=jboss_wbc_ep11_04:8480 ssq, parent=G1JBOSS11P_AGENT, version=5.0.0.GA].
Caused by: java.lang.RuntimeException: Naming URL 'jnp://jboss:service=Naming
            Port:***CHANGE_ME***' is not valid: Illegal character in authority at index 6: jnp://jboss:service=Naming
            Port:***CHANGE_ME***

• JBoss instance is not available and agent.log contains:

2013-08-19 14:20:09,948 WARN  [ResourceContainer.invoker.daemon-1] (jboss.on.common.jbossas.JmxConnectionHelper)- Could not establish connection to the instance [1] times.
org.mc4j.ems.connection.EmsConnectException: Could not connect [jnp://127.0.0.1:1099] javax.naming.NameNotFoundException: jmx not bound
    at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:106)
    at org.jboss.on.common.jbossas.JmxConnectionHelper.loadConnection(JmxConnectionHelper.java:211)
    at org.jboss.on.common.jbossas.JmxConnectionHelper.getEmsConnection(JmxConnectionHelper.java:101)
    at org.rhq.plugins.jbossas5.ApplicationServerComponent.initializeEmsConnection(ApplicationServerComponent.java:505)
    at org.rhq.plugins.jbossas5.ApplicationServerComponent.start(ApplicationServerComponent.java:181)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocationThread.call(ResourceContainer.java:634)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: javax.naming.NameNotFoundException: jmx not bound
    at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
    at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
    at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
    at org.jnp.server.NamingServer.lookup(NamingServer.java:396)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
    at sun.rmi.transport.Transport$1.run(Transport.java:159)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
    at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:726)
    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
    at javax.naming.InitialContext.lookup(InitialContext.java:392)
    at org.mc4j.ems.impl.jmx.connection.support.providers.JBossConnectionProvider.doConnect(JBossConnectionProvider.java:77)
    at org.mc4j.ems.impl.jmx.connection.support.providers.AbstractConnectionProvider.connect(AbstractConnectionProvider.java:101)
    ... 14 more

• We are not seeing any JVM metric data for our EAP server