Common Configuration Issues

JkShmFile on a NFS share

Placing the JkShmFile on a NFS share can cause unexplained pauses in mod_jk and odd behavior. It is strongly recommended that the JkShmFile always be placed on local storage to avoid problems.

A firewall between Apache HTTP Server and JBoss Enterprise Application Platform

If there is a firewall between Apache HTTP Server and JBoss Enterprise Application Platform and no socket_keepalive parameter is set, the firewall can close connections unexpectedly.

MaxClients higher than maxThreads

Setting the MaxClients parameter in Apache HTTP Server higher than the maxThreads setting in JBoss (with a high load on the server) will result in Apache HTTP Server overwhelming the JBoss instance with threads which will cause hung and/or dropped connections.

No connectionTimeout parameter set

The connectionTimeout parameter set in JBoss is required for proper maintenance of old connections.

No CPing/CPong set

The CPing/CPong property in mod_jk is the most important worker property setting, allowing mod_jk to test and detect faulty connections. Not setting this parameter can lead to bad connections not being detected as quickly which can lead to web requests behaving as if 'hung'.

Running an old version of mod_jk.

There are known issues with sticky sessions in versions prior to mod_jk 1.2.27.

Running an older version of EAP.

There is a bug in EAP 4.2 base and EAP 4.2 CP01 which causes sockets to be left in the CLOSE_WAIT state, thus causing the appearance of hung requests again. This issue has been reported and fixed https://jira.jboss.org/jira/browse/JBPAPP-366

Unresponsive back end server

java.lang.OutOfMemoryError errors or high pause times can cause the back end server to become unresponsive.

Common Errors

"CPing/CPong" Errors

Presents with errors like the following:

[info] ajp_handle_cping_cpong::jk_ajp_common.c (865): timeout in reply cpong
...
[error] ajp_connect_to_endpoint::jk_ajp_common.c (957): (nodeA) cping/cpong after connecting to the backend server failed (errno=110)
[error] ajp_send_request::jk_ajp_common.c (1507): (nodeA) connecting to backend failed. Tomcat is probably not started or is listening on the  wrong port (errno=110)

These CPing/CPong messages do not indicate a problem with mod_jk at all, they indicate that JBoss did not respond in the defined CPing/CPong time.

This is seen many times when there is high load on the JVM JBoss is running on causing high garbage collection or potentially thread contention. It could also be that the JBoss instance is overloaded, or even that a firewall is blocking the connection or there are network issues.

The following workflow may assist to correct these type of issues:

Procedure 4.1. Resolving "CPing/CPong" Errors

1. Optimize your Apache HTTP Server and JBoss Enterprise Application Platform configuration. You can contact Red Hat's Global Support Services for assistance with this.
   If this does not resolve the issue, proceed to Step 2
2. Confirm that there is no firewall blocking or dropping the AJP connections.

"Tomcat is down" Errors

Presents with errors like the following:

1. [error] ajp_get_reply::jk_ajp_common.c (2020): (node1) Timeout with waiting reply from tomcat. Tomcat is down, stopped or network problems (errno=110)

   The above error means that JBoss did not respond in the configured reply_timeout time. The solution can be one (or both) of the following:
   1. Increase the reply_timeout.
   2. Verify there are no garbage collection issues/long pause times in JBoss that may prevent the request from responding thus causing that error.

2. [Fri May 25 11:53:37 2012][11159:3086420192] [debug] init_ws_service::mod_jk.c (977): Service protocol=HTTP/1.1 method=POST ssl=false host=(null) addr=127.0.0.1 name=localhost port=80 auth=(null) user=(null) laddr=127.0.0.1 raddr=127.0.0.1 uri=/foo/bar
   ...
   [Fri May 25 11:58:39 2012][11159:3086420192] [debug] jk_shutdown_socket::jk_connect.c (681): About to shutdown socket 17
   [Fri May 25 11:58:39 2012][11159:3086420192] [debug] jk_shutdown_socket::jk_connect.c (689): Failed sending SHUT_WR for socket 17
   [Fri May 25 11:58:39 2012][11159:3086420192] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1150): (node1) can not receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=104)
   [Fri May 25 11:58:39 2012][11159:3086420192] [error] ajp_get_reply::jk_ajp_common.c (1962): (node1) Tomcat is down or refused connection. No response has been sent to the client (yet)

   The above error likely means that JBoss Enterprise Application Platform did not respond within the configured core Apache HTTP Server timeout period.
   Note that with these messages the [11159:3086420192] portion of the message serves as an identifier for the connection/request in question. Therefore tracing back from the point of the error in logs can help clarify the activity around the connection/request that lead to the error.
   In this case, that helps clarify that the error was experienced five minutes after the response was sent to JBoss, which likely points to a five minute timeout (this is Apache HTTP Server's Timeout directive default if not specified). If the Timeout is interrupting mod_jk requests, then it should be increased from the current value to allow for the maximum acceptable response time.

   Procedure 4.2. Resolving "Tomcat is down" Errors

   1. Optimize your Apache HTTP Server and JBoss Enterprise Application Platform configuration. You can contact Red Hat's Global Support Services for assistance with this.
      If this does not resolve the issue, proceed to Step 2
   2. Confirm that there is no firewall blocking or dropping the AJP connections.

General Performance Issues

Presents with errors like the following:

ERROR [org.apache.coyote.ajp.AjpMessage] (ajp-192.168.0.101-8001-13) Invalid message received with signature 12336

The above exception when using mod_jk in JBoss Web typically indicates a non AJP request sent to the AJP connector.

Workflows that may assist in resolving these kinds of issues is below:

Procedure 4.3. General Performance Problems

1. Optimize your Apache HTTP Server and JBoss Enterprise Application Platform configuration. You can contact Red Hat's Global Support Services for assistance with this.
   If this does not resolve the issue, proceed to Step 2
2. Gather garbage collection logs for analysis.
   If the logs show long garbage collection pause times then you should optimize the Java Virtual Machine to reduce the garbage collection pauses and gather/recheck updated logs. Refer to https://access.redhat.com/knowledge/solutions/19932 (Red Hat account required) for more information.
   If this is not the case, or did not resolve the issue, try Step 3, Step 4 and/or Step 5 until your issue is resolved.
3. Determine how long the longest request should take. Factor in transaction times. You may need to increase the reply_timeout to resolve the problem.
   If this does not resolve the issue, continue to Step 4.
4. Determine if your current environment can handle the given load. If not, you may need to upgrade or add more machines.
   If this does not resolve the issue, continue to Step 5.
5. Confirm that there is no firewall blocking or dropping the AJP connections.

Procedure 4.4. 503 Errors

1. Optimize your Apache HTTP Server and JBoss Enterprise Application Platform configuration. You can contact Red Hat's Global Support Services for assistance with this.
   If this does not resolve the issue, proceed to Step 2
2. Gather garbage collection logs for analysis.
   If the logs show long garbage collection pause times then you should optimize the Java Virtual Machine to reduce the garbage collection pauses and gather/recheck updated logs. Refer to https://access.redhat.com/knowledge/solutions/19932 (Red Hat account required) for more information.
   If this is not the case, or does not resolve the issue, continue to Step 3
3. Determine how long the longest request should take. Factor in transaction times. You may need to increase the reply_timeout to resolve the issue.
   If this does not resolve the issue, move on to Step 4.
4. Determine if your current environment can handle the given load. If not, you may need to upgrade or add more machines.

JBoss/JVM-related Issues

May present with errors like:

[error] service::jk_lb_worker.c (1473): All tomcat instances failed, no more workers left

If Apache HTTP Server and JBoss Enterprise Application Platform are optimized and you still receive "no more workers left" this typically indicates an issue on the JBoss/JVM side. A number of JVM-related problems could lead to mod_jk not being able to get a connection to JBoss in the configured timeouts, thus causing the worker to go into the error state and producing this message.

Procedure 4.5. JBoss/JVM Problems

1. Enable garbage collection logging.
   1. For UNIX based systems, the options should be placed in run.conf, not run.sh. The run.conf in the server configuration directory (e.g. <JBOSS_HOME>/server/<PROFILE>/run.conf) takes precedence over the run.conf in the <JBOSS_HOME>/bin directory (except in JBoss EAP 5.0.0 due to a regression fixed in version 5.0.1).
   2. For Windows, the options need to be added to run.bat, as it does not read run.conf.
   3. Check boot.log to see the value of the user.dir environment variable (e.g. <JBOSS_HOME>/bin), the default location for garbage collection logging when no path is provided. If you are running multiple instances of JBoss against the same directory like so:

      ./run.sh -c node1 -b 127.0.0.1 -Djboss.messaging.ServerPeerID=1
      ./run.sh -c node2 -b 127.0.0.1 -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-01

   4. Then for the gc.log files to be properly separated you will need to make sure each <PROFILE> has a unique run.conf with the JVM_OPTS specific to that <PROFILE>.
      For example node1 will contain a <JBOSS_HOME>/server/node1/run.conf with contents:

      JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:gc_node1.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

   5. And <JBOSS_HOME>/server/node2/run.conf with contents:

      JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:gc_node2.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

      Important

      gc.log is recreated every time JBoss starts.

      Be sure to back up gc.log if you are restarting the server. Alternatively you may be able to add a timestamp to the file name depending on the OS and/or shell. For example, with OpenJDK or Oracle/Sun JDK on Linux: -Xloggc:gc.log.`date +%Y%m%d%H%M%S`.
   6. On Windows, you can use

      for /f "tokens=2-4 delims=/ " %%a in ('date /t') do (set mydate=%%c-%%a-%%b)
      for /f "tokens=1-2 delims=/:" %%a in ("%TIME%") do (set mytime=%%a%%b)
      set "JAVA_OPTS=%JAVA_OPTS% -Xloggc:C:/log/gv.log.%mydate%_%mytime%

2. For the time period when there are slowdowns, hangs, or errors, gather the following data:
   • Garbage collection logs. Follow Procedure 4.5, “JBoss/JVM Problems”.
   • High CPU data coupled with thread dumps (depending upon platform):
     The links below can assist in gathering Java thread data. A Red Hat subscription is required.
     • CPU utilization by Java threads on Linux/Solaris: https://access.redhat.com/knowledge/node/46596.
     • CPU utilization by Java threads on Windows: https://access.redhat.com/knowledge/node/46598.
   • For cases where the Java application is an application server, gather log files:
     In JBoss:
     • <JBOSS_HOME>/server/<PROFILE>/log/server.log
     • <JBOSS_HOME>/server/<PROFILE>/log/boot.log
     In Tomcat:
     • catalina.out
3. Determine if the CPU utilization is caused by the JVM (Java application). Here, you want to validate that a Java process is indeed using an unexpected amount of CPU.
   The Java thread data gathered in the first step should help identify this.
4. Assuming a Java process is identified as the cause of high CPU, the most common cause is java Garbage collection. Determine if the high CPU is caused by Java garbage collection by analyzing the garbage collection for long pause times and/or low throughput overall at the time of the issue.
   To find the garbage collection logging related to the issue, it is necessary to determine the number of seconds after JVM startup that the issue happens (that is the typical format of garbage collection logging timestamps). To determine the time elapsed, you can use the first timestamp in the high CPU data gathered and the first timestamp in the console log, boot.log (JBoss), server.log (JBoss), or catalina.out (Tomcat.)
   If you see long pause times and/or low throughput overall, refer to the following Knowledge Base article (Red Hat subscription required) https://access.redhat.com/knowledge/node/19932.
5. If Garbage collection is not responsible for the high CPU, use the thread dump information gathered when validating CPU information to identify the threads.

Apache HTTP Server-based

The JBoss HTTP Connector mod-cluster uses Apache HTTP Server as the proxy server.

Real-time load-balancing calculation

The JBoss HTTP Connector mod_cluster creates a feedback network between the worker nodes and the proxy server. The mod_cluster service is deployed on each of the worker nodes. This service feeds real-time load information to the proxy server. The proxy server then makes intelligent decisions about where to allocate work, based on the current load on each worker node. This real-time adaptive load distribution results in increased optimization of resources.

The information that is reported by the worker nodes and the load-balancing policy used by the proxy are both customizable.

Routing based on real-time application life-cycle

The JBoss HTTP Connector mod_cluster service deployed on the worker nodes relays application life-cycle events to the proxy server. This allows the server to dynamically update its routing table. When an application is undeployed on a node, the proxy server no longer routes traffic for that application to that node.

Automatic Proxy Discovery

The proxy server can be configured to announce its presence via UDP multicast. New worker nodes discover the proxy server and add themselves to the load-balancing cluster automatically. This greatly reduces the configuration and maintenance needed. When UDP multicast is not available or is undesirable, worker nodes are configured with a static list of proxies.

Multiple Protocol Support

The JBoss HTTP Connector mod_cluster can use HTTP, HTTPS, or Apache JServ Protocol (AJP) for communication between the proxy and the worker nodes.

Proxy Server

Shared Memory Manager module: mod_slotmem.so

The Shared Memory Manager module, mod_slotmem, makes the real-time worker node information available to multiple Apache HTTP Server server processes.

Cluster Manager module: mod_manager.so

The Cluster Manager module, mod_manager, receives and acknowledges messages from nodes, including worker node registrations, worker node load data, and worker node application life-cycle events.

Proxy Balancer module: mod_proxy_cluster.so

The Proxy Balancer module, mod_proxy_cluster, handles the routing of requests to cluster nodes. The Proxy Balancer selects the appropriate node to forward the request to, based on application location in the cluster, current state of each of the cluster nodes, and the Session ID (if a request is part of an established session).

Proxy Advertisement module: mod_advertise.so

The Proxy Advertisement module, mod_advertise.so, broadcasts the existence of the proxy server via UDP multicast messages. The server advertisement messages contain the IP address and port number on which the proxy is listening for responses from nodes that wish to join the load-balancing cluster.

Worker Node Components

Worker node service: mod-cluster.sar

This service provides the proxy with real-time information on the worker node's state and sends notification of application life-cycle events; as well as allowing the node to discover and register itself with any proxies running on the same network.

JBoss HTTP Connector non-clustered operation

In non-clustered mode each worker node communicates directly with the proxy.

JBoss HTTP Connector clustered operation

In clustered mode multiple worker nodes form a JBoss HA (High Availability) cluster domain. A single worker node communicates with the proxy on behalf of the other nodes in the cluster domain.

Terms

Active Sessions

A session is considered active if a client thread will ever send a request associated with the session. When client threads stop using a session, they can either send a request to invalidate it, or abandon it by no longer including its session cookie in requests.

Once a session has been abandoned, it is no longer reflected in the Session Balancing chart, but will continue to exist on the worker node until it is removed based on session timeout values.

Total Clients

The number of client threads created since the last time the Start button was clicked.

Live Clients

The number of client threads currently running.

Failed Clients

The number of clients threads that terminated abnormally, for example, a request that resulted in something other than a HTTP 200 response.

Terms

ISAPI and CGI Restrictions

ISAPI and CGI restrictions are request handlers that allow dynamic content to execute on a server. These restrictions are either CGI files (.exe) or ISAPI extensions (.dll). You can add custom ISAPI or CGI restrictions if the IIS configuration system allows this. [ Configuring ISAPI and CGI Restrictions in IIS 7 ] .

<replication-trigger>

element determines when the container should consider that session data must be replicated across the cluster. The rationale for this setting is that after a mutable object stored as a session attribute is accessed from the session, in the absence of a setAttribute call, the container has no clear way to know if the object (and hence the session state) has been modified and needs to be replicated. This element has 3 valid values:

SET_AND_GET is conservative but not optimal (performance-wise): it will always replicate session data even if its content has only been accessed and not modified. This setting made (a little) sense in JBoss Enterprise Application Platform 4 since using it was a way to ensure that every request triggered replication of the session's timestamp. Since setting max_unreplicated_interval to 0 accomplishes the same thing at much lower cost, using SET_AND_GET makes no sense with JBoss Enterprise Application Platform 5 or JBoss Enterprise Web Platform 5.

SET_AND_NON_PRIMITIVE_GET is conservative but will only replicate if an object of a non-primitive type has been accessed (in effect, the object is not of a well-known immutable JDK type such as Integer, Long, String, etc.) This is the default value.

SET assumes that the developer will explicitly call setAttribute on the session if the data needs to be replicated. This setting prevents unnecessary replication and can have a major beneficial impact on performance, but requires very good coding practices to ensure setAttribute is always called whenever a mutable object stored in the session is modified.

In all cases, calling setAttribute marks the session as needing replication.

<cacheName>

Specifies the name of the JBoss Cache configuration that should be used for storing distributable sessions and replicating them around the cluster. This element lets web applications that require different caching characteristics specify the use of separate, differently configured, JBoss Cache instances. In JBoss Enterprise Application Platform 4 the cache to use was a server-wide configuration that could not be changed per web application. The default value is standard-session-cache. See Section 17.3, “Configure the JBoss Cache instance used for session state replication” for more details on JBoss Cache configuration for web-tier clustering.

<replication-field-batch-mode>

Specifies whether all replication messages associated with a request will be batched into one message. This is applicable only if replication-granularity is FIELD. If replication-field-batch-mode is set to true, fine-grained changes made to objects stored in the session attribute map will replicate only when the HTTP request is finished; otherwise they replicate as they occur. Setting this to false is not advised because it increases the number of replication requests and the chance of session state being out of sync. Default is true.

Important

The FIELD granularity option is now deprecated as JBoss Cache, which provides this feature, is to be subsituted by Infinispan (Infinispan does not support this granularity).

<useJK>

Specifies whether the container should assume that a JK-based software load balancer (for example,. mod_jk, mod_proxy, mod_cluster) is being used for load balancing for this web application. If set to true, the container will examine the session ID associated with every request and replace the jvmRoute portion of the session ID if it detects a failover.

You need only set this to false for web applications whose URL cannot be handled by the JK load balancer.

<max-unreplicated-interval>

Specifies the maximum interval between requests, in seconds, after which a request will trigger replication of the session's timestamp regardless of whether the request has otherwise made the session dirty. Such replication ensures that other nodes in the cluster are aware of the most recent value for the session's timestamp and will not incorrectly expire an unreplicated session upon failover. It also results in correct values for HttpSession.getLastAccessedTime() calls following failover.

The default value is null (in effect, unspecified). In this case the session manager will use the presence or absence of a jvmRoute configuration on its enclosing JBoss Web Engine (see Section 3.2, “Configuring JBoss to work with mod_jk”) to determine whether JK is used.

A value of 0 means the timestamp will be replicated whenever the session is accessed. A value of -1 means the timestamp will be replicated only if some other activity during the request (for example,. modifying an attribute) has resulted in other replication work involving the session. A positive value greater than the HttpSession.getMaxInactiveInterval() value will be treated as probable misconfiguration and converted to 0; (in effect, replicate the metadata on every request). Default value is 60.

<snapshot-mode>

Specifies when sessions are replicated to the other nodes. Possible values are INSTANT (the default) and INTERVAL.

The typical value, INSTANT, replicates changes to the other nodes at the end of requests, using the request processing thread to perform the replication. In this case, the snapshot-interval property is ignored.

With INTERVAL mode, a background task is created that runs every snapshot-interval milliseconds, checking for modified sessions and replicating them.

Note that this property has no effect if replication-granularity is set to FIELD. If it is FIELD, INSTANT mode will be used.

<snapshot-interval>

Specifies how often (in milliseconds) the background task that replicates modified sessions should be started for this web application. Only meaningful if snapshot-mode is set to INTERVAL.

<session-notification-policy>

Specifies the fully qualified class name of the implementation of the ClusteredSessionNotificationPolicy interface that should be used to govern whether servlet specification notifications should be emitted to any registered HttpSessionListener, HttpSessionAttributeListener and/or HttpSessionBindingListener.

Important

Event notifications that may be appropriate in non-clustered environment may not necessarily be appropriate in a clustered environment; see https://jira.jboss.org/jira/browse/JBAS-5778 for an example of why a notification may not be desired. Configuring an appropriate ClusteredSessionNotificationPolicy gives the application author fine-grained control over what notifications are issued.

Compulsory Properties

className

fully-qualified class name of the org.apache.catalina.Manager implementation (that is, org.jboss.web.tomcat.service.session.persistent.DataSourcePersistentManager)

dataSourceJndiName

JNDI name of the data source, which defines the database connection to the httpsessions

Properties Defining the Database Connection Properties

connectionName

value of the username parameter to pass to the DataSource.getConnection method (if null, the getConnection with no arguments is called)

connectionPassword

password to pass to DataSource.getConnection

sessionTable

name of the database session table in which sessions are stored (by default httpsessions)

sessionAppCol

name of the column with the web application name associated with the session (by default app)

sessionIdCol

name of the column with the core, immutable part of a session ID (by default id; this and the sessionAppCol columns form the unique index for the table)

sessionFullIdCol

name of the column with the full session ID (including any mutable element added to the core session ID, for example a jvmRoute; by default fullid)

sessionCreationTimeCol

name of the column with the time when the session was created (of the long datatype, by default creationtime)

sessionMaxInactiveCol

name of the column with the maximum number of milliseconds the session can remain unaccessed before expiring (by default maxinactive)

sessionVersionCol

name of the column which stores the session's "version" (session version is incremented each time the session is persisted; by default version)

sessionLastAccessedCol

name of the column with the timestamp of the last session access (take the long data type value; by default lastaccess)

sessionNewCol

Name of the column with the flag indicating whether the session is new (session is considered new if it was not yet joined by the client; by default isnew

sessionValidCol

name of the column with the flag indicating whether the session is valid (by default isvalid

sessionMetadataCol

name of the column which can store serialized metadata about the session (currently unused; by default metadata

sessionAttributeCol

name of the column with the serialized session attribute map (by default attributes)

Properties Defining the Manager's Behavior

cleanupInterval

minimum period of time in seconds that must lapse from the last cleaning of old "abandoned" sessions from the database before the next cleaning (by default 14400 seconds, that is, 4 hours)

A session is abandoned if the only server that was handling the session requests was shut down before the session expired, no further requests for the session were received so that the session did not fail over to another server. Such session do not expire on the normal session expiration checks. Therefore a special process runs periodically to clean the session from the database.

replicationTriggerString

activity executed during a request that makes the session database persistent (the activity with the replication-trigger property SET)

maxUnreplicatedInterval

maximum interval between requests, in seconds, after which the session's timestamp is persisted regardless of whether the request caused the session to become dirty in any other way

useJK

flag determining whether the container assumes a JK-based software load balancer is used for load balancing (if set to true, the container examines the session ID associated with every request and replace the jvmRoute portion of the session ID if it detects a failover)

maxActiveAllowed

maximum number of active sessions

useSessionPassivation

flag for enabling/disabling session passivation (session is removed from memory but remains always in the persistent store)

passivationMinIdleTime

minimum time, in seconds, a session must be inactive before passivated

passivationMaxIdleTime

maximum time, in seconds, a session can be inactive before passivated

processExpiresFrequency

frequency at which the background process thread calls the session manager to perform background processes (for example, expire or passivate sessions; (by default, every 10 seconds)

This configuration is defined as value N with the background cleanup process called 1 in N callings to the session manager. Default is 1, that is, the cleanup process is performed every time the manager is called by the background process, that is cleanup is performed every 10 seconds. For example, if set to 6, the manager performs the cleanup once a minute (1/6, that is once in 60 seconds).

sessionNotificationPolicy

fully qualified class name of the implementation of the ClusteredSessionNotificationPolicy interface that is used to govern whether servlet specification notifications is emitted to any registered HttpSessionListener, HttpSessionAttributeListener or HttpSessionBindingListener.