Session Replication Failing With The Exception java.io.NotSerializableException
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 5.x
- 6.x
- 7.x
Issue
- The following error appears in the logs. What is the implication for session replication and how can it be resolved?
ERROR [JBossCacheService] externalizeSession(): exception occurred
externalizing session SessionBasedClusteredSession[id:
nFTh1yoC83B5EHj4+8qWDQ**.node1 lastAccessedTime: 1268256303847 version:
4 lastOutdated: 0]
java.io.NotSerializableException: <name of class>
- I discovered that even though our own code only puts Serializable objects in the session, Spring puts non-serializable objects in the session when we use the Spring option sessionForm=true. That obviously causes problems when the cluster tries to synchronize the session between the nodes.
When working in our application (web part/ejb part), we are getting the following exception:
ERROR [org.jboss.cache.marshall.CacheMarshaller300] (http-127.0.0.1-8080-2) Error while marshalling object: ReplicateCommand{cmds=PutDataMapCommand{fqn=/JSESSION/sg_localhost/wumZ8GxmbL5ojIJkIS9eBg__, dataVersion=null, data={0=1, 1=1322745674120, 2=org.jboss.web.tomcat.service.session.distributedcache.spi.DistributableSessionMetadata@7de3f9f1, 3=org.jboss.ha.framework.server.SimpleCachableMarshalledValue{raw=java.util.HashMap@1284729517serialized=false}}, globalTransaction=null, erase=false}}
java.io.NotSerializableException: java.lang.Object
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1156)
- Which object is not serializable? Is there an easy way to identify the objects?
- I have setup 2 JBoss EAP 5.1.0 into cluster then in our web.xml added
<distributable/>tag and in logs files I see the below error :
ERROR [CommandAwareRpcDispatcher] java.io.NotSerializableException: java.util.concurrent.FutureTask
In EAP 6, serialization fails with the following ERROR:
ERROR [org.infinispan.transaction.TransactionCoordinator] (http-/192.168.82.19:8332-8) ISPN000097: Error while processing a prepare in a single-phase transaction: org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:153)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.broadcastRemoteCommands(CommandAwareRpcDispatcher.java:197)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:498)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247)
...
Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:332)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:146)
... 80 more
Caused by: org.infinispan.marshall.NotSerializableException: this.is.my.object
Caused by: an exception which occurred:
in field target
in field targetSource
in field advised
in field ccSolicitudesService
in object java.util.HashMap@55f09b27
in object org.jboss.as.clustering.SimpleMarshalledValue@55f09b27
in object org.infinispan.atomic.PutOperation@c94cb0d4
in object java.util.LinkedList@7d469a95
in object org.infinispan.atomic.AtomicHashMapDelta@1f4646d7
in object org.infinispan.commands.write.PutKeyValueCommand@4739152a
in object org.infinispan.commands.tx.PrepareCommand@92d8fdd3
ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-/10.38.73.150:8009-2) ISPN000136: Execution error: org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:173)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:194)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:251)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:238)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:233)
...
Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
... 80 more
Caused by: org.infinispan.marshall.NotSerializableException: this.is.my.object
Caused by: an exception which occurred:
in field convFactors
in field orderItemList
in field materialBean
in object java.util.HashMap@3ceddee4
in object org.jboss.as.clustering.SimpleMarshalledValue@3ceddee4
in object org.infinispan.atomic.PutOperation@20530a83
in object java.util.LinkedList@f2784c57
in object org.infinispan.atomic.AtomicHashMapDelta@13452861
in object org.infinispan.commands.write.PutKeyValueCommand@e593a539
in object org.infinispan.commands.tx.PrepareCommand@7acfe1a3
-
Errors
ISPN000136,ISPN000097,ISPN000111 -
We are running three nodes. It would be possible to use farming instead of clustering them. Would this solve the NotSerializableException problem? Are there any parameters for Infinispan that would solve the problem? Is it possible to deactivate Infinispan?
2019-05-28 08:21:37,807 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (default task-2) ISPN000073: Unexpected error while replicating: org.infinispan.commons.marshall.NotSerializableException: org.apache.commons.configuration.XMLConfiguration$XMLFileConfigurationDelegate
Caused by: an exception which occurred:
in field org.apache.commons.configuration.AbstractHierarchicalFileConfiguration.delegate
in object org.apache.commons.configuration.XMLConfiguration@a9d253e
in field org.jboss.weld.context.SerializableContextualInstanceImpl.instance
in object org.jboss.weld.context.SerializableContextualInstanceImpl@374c2140
in object org.jboss.weld.context.SerializableContextualInstanceImpl@374c2140
in object java.util.HashMap@58f48a74
in object org.wildfly.clustering.marshalling.jboss.SimpleMarshalledValue@58f48a74
in object org.infinispan.commands.write.PutKeyValueCommand@841a26d6
in object org.infinispan.commands.tx.PrepareCommand@2fd31e47
Resolution
-
Make sure all classes put in the session are Serializable.
-
All members of a Serializable class must also be Serializable or marked as transient.
-
Disable clustering to avoid session replication.
Log4j
If an object in the session has a log4j Logger in it, but the Log4J Loggers are not serializable you can do one of two things
Mark the logger as transient ( if you are using a separate logger for each instance)
Mark the logger as static
Root Cause
The NotSerializableException will occur whenever a non-serializable class resides in the session and the session is replicated.
JBoss EAP has no way to serialize a class that does not implement the Serializable interface. It will, therefore, throw an exception. Note that clustering will not work properly if this issue is not resolved, this is because the marshalling process, i.e passing convert information from POJO to the wire is a vital process on the cluster. The marshalling process is based in JBoss Marshalling Framework, instead of Java framework for improving the performance.
The purpose of clustering is to achieve high availability, which is achieved through replication. The NotSerializableException error hinders replication and interferes in this process.
Diagnostic Steps
To isolate what object is causing the issue:
For Java 6 and above (EAP 5,EAP 6and EAP 7)
- add the system parameter -Dsun.io.serialization.extendedDebugInfo=true to log additional information about the object.
For Java 5 (only EAP 5)
- use the attached ServletFilter to determine which session attribute is causing the issue.
- To use it add it to your WAR's code and your web.xml.
- It attempts to serialize everything in the session after every request, using JBoss Serialization's check-for-Serializable mode so you should get better messages about the actual problem
<filter>
<filter-name>SerializableSessionFilter</filter-name>
<filter-class>com.redhat.example.SerializableSessionFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>SerializableSessionFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
JBoss EAP subsystem Infinispan is responsible for the process, so enabling trace logging in Infinispan can also help since it adds more information to debug the case and the specific object can't be serialized.
N.B Make sure to compile against $JBOSS_HOME/jboss-as/client/jboss-serialization.jar as the filter has dependencies on JBoss serialization
N.B Where possible, this filter should be tested in a development or test environment as it does add additional overhead to every request
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.