Issue

We have set up a 3 node cluster of JBoss Portal 6.1.0 instances.

The issue is when any of the node is stopped and restarted, it fails to start up properly with the following error:

15:21:32,330 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 70) MSC000001: Failed to start service jboss.infinispan.web.default-host/simplest-hello-world-portlet: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/simplest-hello-world-portlet: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
    at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
    at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
    at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
    at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
    at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
    at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
    at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
    at org.infinispan.CacheImpl.start(CacheImpl.java:582)
    at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
    at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
    at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
    at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
    at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109)
    at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100)
    at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
    at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)
    ... 4 more
Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/simplest-hello-world-portlet on node2/web
    at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]
    at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
    ... 18 more

On the master node, we see the following exception:

15:20:32,364 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-16) Failed to send entries to node node2/web : java.lang.RuntimeException: Failure to marshal argument(s): org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
    at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
    at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
    at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
    at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257)
    at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300)
    at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
    ... 12 more
Caused by: org.infinispan.marshall.NotSerializableException: org.apache.portals.bridges.util.ServletPortletSessionProxy
Caused by: an exception which occurred:
    in object java.util.HashMap@1df8fa77
    in object org.jboss.as.clustering.SimpleMarshalledValue@1df8fa77
        -> toString = {javax.portlet.p.d2eda1dc-f89f-4075-a87d-e1ea226bd4df?org.apache.portals.bridges.util.portlet_window_id=d2eda1dc-f89f-4075-a87d-e1ea226bd4df, javax.portlet.p.d2eda1dc-f89f-4075-a87d-e1ea226bd4df?org.apache.portals.bridges.util.servlet_portlet_session_proxy=org.apache.catalina.session.StandardSessionFacade@3ade2072, javax.portlet.p.d2eda1dc-f89f-4075-a87d-e1ea226bd4dforg.apache.struts.action.LOCALE=en_US}
    in object org.infinispan.util.FastCopyHashMap@1a009df9
        -> toString = {0=2, 1=1390486814391, 2=org.jboss.as.clustering.web.DistributableSessionMetadata@442a1df, 3={javax.portlet.p.d2eda1dc-f89f-4075-a87d-e1ea226bd4df?org.apache.portals.bridges.util.portlet_window_id=d2eda1dc-f89f-4075-a87d-e1ea226bd4df, javax.portlet.p.d2eda1dc-f89f-4075-a87d-e1ea226bd4df?org.apache.portals.bridges.util.servlet_portlet_session_proxy=org.apache.catalina.session.StandardSessionFacade@3ade2072, javax.portlet.p.d2eda1dc-f89f-4075-a87d-e1ea226bd4dforg.apache.struts.action.LOCALE=en_US}}
    in object org.infinispan.atomic.AtomicHashMap@6e3fbacb
        -> toString = AtomicHashMap
    in object org.infinispan.container.entries.ImmortalCacheEntry@ffdab5f6
        -> toString = ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}
    in object java.util.ArrayList@ffdab615
        -> toString = [ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}]
    in object org.infinispan.statetransfer.StateChunk@329a3a3a
        -> toString = StateChunk{segmentId=0, cacheEntries=[ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}], isLastChunk=true}
    in object java.util.ArrayList@329a3a59
        -> toString = [StateChunk{segmentId=0, cacheEntries=[ImmortalCacheEntry{key=xxOi7c7PwmgkmaDGX8NzYUro, value=ImmortalCacheValue {value=AtomicHashMap}}], isLastChunk=true}]
    in object org.infinispan.statetransfer.StateResponseCommand@5e985470
        -> toString = StateResponseCommand{cache=default-host/simplest-hello-world-portlet, origin=node1/web, topologyId=4}

Note: simplest-hello-world-portlet is a Struts based portlet similar to our real portlets.