Serialization issues with Struts portlet in JPP 6.1.0 cluster

Issue

We have set up a 3 node cluster of JBoss Portal 6.1.0 instances.

The issue is when any of the node is stopped and restarted, it fails to start up properly with the following error:

15:21:32,330 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 70) MSC000001: Failed to start service jboss.infinispan.web.default-host/simplest-hello-world-portlet: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/simplest-hello-world-portlet: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
    at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
    at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
    at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
    at org.infinispan.CacheImpl.start(CacheImpl.java:582)
    at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
    at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
    at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
    at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
    at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
    at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)
    at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
    ... 4 more
Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/simplest-hello-world-portlet on node2/web
    at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
    at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
    ... 18 more

On the master node, we see the following exception:

15:20:33,853 ERROR [org.infinispan.marshall.VersionAwareMarshaller] (transport-thread-11) ISPN000065: Exception while marshalling object: java.io.NotActiveException: Fields were never written
    at org.jboss.marshalling.river.RiverObjectOutputStream.finish(RiverObjectOutputStream.java:175)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1009)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:680)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85)
    at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175)
    at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47)
    at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36)
    at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:459)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.marshall.MarshallUtil.marshallMap(MarshallUtil.java:59)
    at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:63)
    at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:47)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:250)
    at org.infinispan.atomic.AtomicHashMap$Externalizer.writeObject(AtomicHashMap.java:247)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154)
    at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48)
    at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50)
    at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:103)
    at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:88)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48)
    at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50)
    at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87)
    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128)
    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112)
    at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73)
    at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:410)
    at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
    at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
    at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
    at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96)
    at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)
    at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64)
    at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109)
    at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
    at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257)
    at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
Caused by: an exception which occurred:
    in field parameters
    in field navigationalState
    in object java.util.HashMap@7b2c68fe
    in object org.jboss.as.clustering.SimpleMarshalledValue@7b2c68fe
        -> toString = {prc.lastPortalName=classic, psm.web/BannerPortlet/43e242e6-c7bc-4f1b-b35d-8814e2f47f02=org.exoplatform.portal.application.replication.ApplicationState@354ce320, psm.web/NavigationPortlet/564f53b7-19ec-4d05-92d7-ada481e05c0b=org.exoplatform.portal.application.replication.ApplicationState@4ad36536, psm.web/BreadcumbsPortlet/eada742d-a505-494b-9f16-97efffb97f46=org.exoplatform.portal.application.replication.ApplicationState@7c99e29a, org.gatein.LAST_LOCALE=en, portal.principal={cached_markup.d2eda1dc-f89f-4075-a87d-e1ea226bd4df=org.gatein.pc.portlet.aspects.ConsumerCacheInterceptor$CacheEntry@2f945c9f}, psm.PortalApplication=org.exoplatform.portal.application.replication.ApplicationState@5aec2c5e, psm.web/FooterPortlet/ff3a7cde-b6bf-4b63-a905-4c47c2c269af=org.exoplatform.portal.application.replication.ApplicationState@4754fdf4, psm.web/HomePagePortlet/06d54ec6-4724-49aa-b14f-45f3a75bff7c=org.exoplatform.portal.application.replication.ApplicationState@1cecc2c2}
    in object org.infinispan.util.FastCopyHashMap@74aa4c9
        -> toString = {0=2, 1=1390486814258, 2=org.jboss.as.clustering.web.DistributableSessionMetadata@3178c6e7, 3={prc.lastPortalName=classic, psm.web/BannerPortlet/43e242e6-c7bc-4f1b-b35d-8814e2f47f02=org.exoplatform.portal.application.replication.ApplicationState@354ce320, psm.web/NavigationPortlet/564f53b7-19ec-4d05-92d7-ada481e05c0b=org.exoplatform.portal.application.replication.ApplicationState@4ad36536, psm.web/BreadcumbsPortlet/eada742d-a505-494b-9f16-97efffb97f46=org.exoplatform.portal.application.replication.ApplicationState@7c99e29a, org.gatein.LAST_LOCALE=en, portal.principal={cached_markup.d2eda1dc-f89f-4075-a87d-e1ea226bd4df=org.gatein.pc.portlet.aspects.ConsumerCacheInterceptor$CacheEntry@2f945c9f}, psm.PortalApplication=org.exoplatform.portal.application.replication.ApplicationState@5aec2c5e, psm.web/FooterPortlet/ff3a7cde-b6bf-4b63-a905-4c47c2c269af=org.exoplatform.portal.application.replication.ApplicationState@4754fdf4, psm.web/HomePagePortlet/06d54ec6-4724-49aa-b14f-45f3a75bff7c=org.exoplatform.portal.application.replication.ApplicationState@1cecc2c2}}
    in object org.infinispan.atomic.AtomicHashMap@2b86709a
        -> toString = AtomicHashMap
    in object org.infinispan.container.entries.ImmortalCacheEntry@bd216bc5
        -> toString = ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}
    in object java.util.ArrayList@bd216be4
        -> toString = [ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}]
    in object org.infinispan.statetransfer.StateChunk@53cb05ee
        -> toString = StateChunk{segmentId=0, cacheEntries=[ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}], isLastChunk=true}
    in object java.util.ArrayList@53cb060d
        -> toString = [StateChunk{segmentId=0, cacheEntries=[ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}], isLastChunk=true}]
    in object org.infinispan.statetransfer.StateResponseCommand@9e18334
        -> toString = StateResponseCommand{cache=default-host/portal, origin=node1/web, topologyId=4}

Note: simplest-hello-world-portlet is a Struts based portlet similar to our real portlets.