JConsole unable to connect to remote JMX Fuse process

Issue

• I am trying to connect Fuse process using JConsole with admin account using SSH tunneling and SSH X11 forwarding... both are failing with below error:

[jboss-fuse@oags ~]$ jconsole -debug
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.NameNotFoundException: karaf-root
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
        at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:350)
        at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:296)
        at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:280)
Caused by: javax.naming.NameNotFoundException: karaf-root
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:99)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:255)
        ... 4 more

• I have done some investigation before raising this ticket, I saw a similar ticket https://access.redhat.com/solutions/1607203, but i verified serviceUrl it looks fine to me

JBossFuse:dartadmin@root> config:list | grep serviceUrl
   serviceUrl = service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root

JBossFuse:dartadmin@root> config:list
   ---  
   jmxRole = admin
   rmiRegistryHost = 0.0.0.0
   serviceUrl = service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root
   rmiRegistryPort = 1099
   felix.fileinstall.filename = file:/apps/jboss-fuse-full/etc/org.apache.karaf.management.cfg
   service.pid = org.apache.karaf.management
   threaded = true
   rmiServerPort = 44444
   objectName = connector:name=rmi
   jmxRealm = karaf
   daemon = true


[jboss-fuse@oags ~]$ ps -ef | grep fuse
498      19273 19271  2 Sep24 ?        00:31:55 java -Dkaraf.home=/apps/jboss-fuse-full -Dkaraf.base=/apps/jboss-fuse-full -Dkaraf.data=/apps/jboss-fuse-full/data -Dcom.sun.management.jmxremote -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -Djava.endorsed.dirs=/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/jre/lib/endorsed:/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/lib/endorsed:/apps/jboss-fuse-full/lib/endorsed -Djava.ext.dirs=/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/jre/lib/ext:/usr/lib/jvm/java-1.6.0-sun-1.6.0.30.x86_64/lib/ext:/apps/jboss-fuse-full/lib/ext -XX:PermSize=512M -XX:MaxPermSize=512M -Xms4096M -Xmx4096M -agentpath:/usr/local/jprofiler8/bin/linux-x64/libjprofilerti.so=port=7291,nowait -Djava.io.tmpdir=/apps/jboss-fuse-full/data/tmp -XX:+StartAttachListener -Xverify:none -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/apps/jboss-fuse-full/data/log/gc.log -XX:+DisableExplicitGC -XX:+UseCompressedOops -Dcom.sun.jndi.ldap.connect.pool.timeout=900000 -Dcom.sun.jndi.ldap.connect.timeout=10000 -Dcom.sun.jndi.ldap.connect.pool.prefsize=50 -Dcom.sun.jndi.ldap.connect.pool.initsize=1 -Dcom.sun.jndi.ldap.connect.pool.maxsize=200 -XX:NewRatio=2 -XX:SurvivorRatio=10 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:+UseLargePages -XX:+AlwaysPreTouch -XX:LargePageSizeInBytes=2m -XX:+AggressiveOpts -XX:CMSFullGCsBeforeCompaction=1 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost -Djava.library.path=/apps/jboss-fuse-full/lib/ -classpath /apps/jboss-fuse-full/lib/karaf-wrapper.jar:/apps/jboss-fuse-full/lib/karaf.jar:/apps/jboss-fuse-full/lib/karaf-jaas-boot.jar:/apps/jboss-fuse-full/lib/karaf-wrapper-main.jar -Dwrapper.key=hWDPqNPz5UoPrWzl -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=19271 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.apache.karaf.shell.wrapper.Main

[jboss-fuse@oags201f01 ~]$ netstat -lp | grep 19273
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 localhost:33204             *:*                         LISTEN      19273/java
tcp        0      0 *:8181                      *:*                         LISTEN      19273/java
tcp        0      0 *:33881                     *:*                         LISTEN      19273/java
tcp        0      0 *:7291                      *:*                         LISTEN      19273/java
tcp        0      0 localhost:32000             *:*                         LISTEN      19273/java
tcp        0      0 *:ldoms-migr                *:*                         LISTEN      19273/java
tcp        0      0 *:rmiregistry               *:*                         LISTEN      19273/java
tcp        0      0 *:61616                     *:*                         LISTEN      19273/java
unix  2      [ ACC ]     STREAM     LISTENING     588839524 19273/java          /tmp/.java_pid19273.tmp

• But if you look at above it is not binding 44444. Is that the reason I am not able to connect?
• Once followed steps as in described in Diagnostic Steps, received following exception message

Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:49999/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException: Could not start JMX connector server
        at org.apache.karaf.management.ConnectorServerFactory$1.run(ConnectorServerFactory.java:252)
Caused by: java.io.IOException: Cannot bind to URL [rmi://0.0.0.0:1099/karaf-root]: javax.naming.NoPermissionException [Root exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
        java.rmi.AccessException: Cannot modify this registry]
        at javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:804)
        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:417)
        at org.apache.karaf.management.ConnectorServerFactory$1.run(ConnectorServerFactory.java:239)
Caused by: javax.naming.NoPermissionException [Root exception is java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
        java.rmi.AccessException: Cannot modify this registry]
        at com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:126)
        at com.sun.jndi.toolkit.url.GenericURLContext.bind(GenericURLContext.java:208)
        at org.apache.aries.jndi.DelegateContext.bind(DelegateContext.java:82)
        at javax.naming.InitialContext.bind(InitialContext.java:400)
        at javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:625)
        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:412)
        ... 1 more
Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
        java.rmi.AccessException: Cannot modify this registry
        at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:400)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:248)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:359)
        at sun.rmi.registry.RegistryImpl_Stub.bind(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.bind(RegistryContext.java:120)
        ... 6 more
Caused by: java.rmi.AccessException: Cannot modify this registry
        at sun.management.jmxremote.SingleEntryRegistry.bind(SingleEntryRegistry.java:61)
        at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)
        at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:390)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:248)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)