Hawtio keeps linking to the wrong containers

Solution In Progress - Updated -

Environment

  • Red Hat JBoss Fuse 6.1.0

Issue

I have four containers running, but only three HTTP ports are listening: 8181, 8182, & 8183. The fourth container used to listen on 8184, but based on the logs I can see it is trying to listen on port 8181::

2014-07-30 15:56:51,433 | WARN  | pool-9-thread-1  | NIOSocketConnectorWrapper        | ternal.NIOSocketConnectorWrapper   40 | 101 - org.ops4j.pax.web.pax-web-jetty - 3.0.6 | Connection on port 8181 cannot be open. Exception:java.net.BindException: Address already in use Reason: Address already in use

The Jolokia URL for the fourth container is shown as being on port 8181, which is the same as one of the other containers. Clicking on the "open" link in Hawtio takes me to the first container.

How do we fix this?

  • We configured jetty.xml ignored to hawtio console for karaf server. We used VIrtual IP for hawtio host, and 8181 for port. It works well. But every time we start karaf server, we see this warning in the karaf server log file:
2015-10-21 13:15:52,277 | WARN  | pool-6-thread-1  | NIOSocketConnectorWrapper        | 111 - org.ops4j.pax.web.pax-web-jetty - 3.2.3 | Connection on port 8181 cannot be open. Exception:java.net.BindException: Address already in use Reason: Address already in use

Resolution

You may update the ports on a specific container by running the following command:

config:proplist -p org.ops4j.pax.web

This will produce output similar to the following:

JBossFuse:karaf@root> config:proplist -p org.ops4j.pax.web
   service.pid = org.ops4j.pax.web
   javax.servlet.context.tempdir = /Users/jsherman/Development/tools/jboss-fuse/jboss-fuse-6.1.0.redhat-379/data/pax-web-jsp
   org.ops4j.pax.web.config.checksum = 3355609679
   org.osgi.service.http.port = 8181
   org.ops4j.pax.web.config.url = profile:jetty.xml
   fabric.zookeeper.pid = org.ops4j.pax.web

Once you confirm the port is indeed incorrect, then you can use the config:edit, config:propset and config:update commands to change it. For example:

JBossFuse:karaf@root> config:edit org.ops4j.pax.web
JBossFuse:karaf@root> config:propset org.osgi.service.http.port
JBossFuse:karaf@root> config:propset org.osgi.service.http.port 8184
JBossFuse:karaf@root> config:update

Root Cause

The root cause as to why the the ports were in conflict is still be investigated.

Diagnostic Steps

  • Capture all logs from affected containers
  • Note JVM process running in the background, was the container started multiple times?
  • Note steps that may be related to reproduce the issue

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments