ArrayIndexOutOfBoundsException after cache segments changed

Solution Verified - Updated -

Issue

  • I changed the segments setting from default 256 to 512 as below:
<distributed-cache name="TestCache" owners="2" mode="SYNC" segments="512" remote-timeout="30000" start="EAGER" statistics="true">
        <locking isolation="READ_COMMITTED" striping="true" acquire-timeout="30000" concurrency-level="1000"/>
         <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
        <memory>
            <object size="524000000" strategy="LRU"/>
        </memory>
        <state-transfer await-initial-transfer="false" enabled="true" timeout="240000" chunk-size="1024"/>
 </distributed-cache>

Then I restarted my cluster members one by one. After that, simply putting & getting cache entries will cause ArrayIndexOutOfBoundsExceptionlike below:

[root@jdg733-sh1 bin]# ./cli.sh -c --controller=127.0.0.1 --commands="container clustered,cache TestCache,put testKey1 testValue1"
java.lang.ArrayIndexOutOfBoundsException: 404
2020-02-20 14:29:53,898 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (management-handler-thread - 3) ISPN000136: Error executing command PutKeyValueCommand on Cache 'TestCache', writing keys [WrappedByteArray{bytes=[B0x033E08746573744B..[11], hashCode=0}]: java.lang.ArrayIndexOutOfBoundsException: 404
        at org.infinispan.distribution.LocalizedCacheTopology.getDistribution(LocalizedCacheTopology.java:213)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.isLockOwner(AbstractLockingInterceptor.java:266)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.needRemoteLocks(PessimisticLockingInterceptor.java:228)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:147)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:82)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150)
        at org.infinispan.interceptors.impl.TxInterceptor.handleWriteCommand(TxInterceptor.java:402)
        at org.infinispan.interceptors.impl.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:235)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
        at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:260)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:250)
        at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:96)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150)
        at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:217)
        at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:179)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123)
        at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56)
        at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:60)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
        at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1939)
        at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1917)
        at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1433)
        at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:2043)
        at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:351)
        at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:427)
        at org.infinispan.cli.interpreter.statement.PutStatement.execute(PutStatement.java:76)
        at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:144)
        at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.lambda$executeInterpreter$7(SecurityActions.java:274)
        at org.infinispan.security.Security.doPrivileged(Security.java:59)
        at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.doPrivileged(SecurityActions.java:83)
        at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.executeInterpreter(SecurityActions.java:275)
        at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:70)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1411)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
        at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:265)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:231)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:240)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$400(ModelControllerClientOperationHandler.java:138)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:162)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:158)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:246)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:254)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:225)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:158)
        at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
        at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:485)

What I don't understand is, I already resarted all the servers one by one, but the issue still persists.

  • Also, on some cluster member logs, below warning:
2020-02-20 15:09:44,762 WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p2-t2) ISPN000071: Caught exception when handling command StateRequestCommand{cache=TestCache, origin=node3sh, type=START_STATE_TRANSFER, topologyId=6, segments={0 3-8 12-13 16 19 21 27-31 35 39-40 44 50 52-53 57-64 69 73-77 82-83 90 94 97 108-109 112 116-117 126-128 138 144-148 155-158 163-164 167-168 172-173 184 188 194 205 207-209 212-213 221 229 235 241-242 248-249 251-253 259-260 263 273 275 282 284 286 288-293 295-297 299 303 309 313-314 318-319 321 326-328 330-335 337-338 348 352 357 369-370 379-380 383-384 386-387 393 401 407-408 414 417-418 420 424-425 427 436 439-440 442-443 446-448 450 453 455-458 460-461 464 466-467 482-483 489-490 492 497 503-507 511}}: java.lang.IllegalArgumentException: Provided integer was larger than originally initialized size
        at org.infinispan.commons.util.ConcurrentSmallIntSet.checkBounds(ConcurrentSmallIntSet.java:64)
        at org.infinispan.commons.util.ConcurrentSmallIntSet.add(ConcurrentSmallIntSet.java:100)
        at org.infinispan.commons.util.ConcurrentSmallIntSet.set(ConcurrentSmallIntSet.java:117)
        at org.infinispan.commons.util.SmallIntSet.forEach(SmallIntSet.java:400)
        at org.infinispan.commons.util.IntSets.concurrentCopyFrom(IntSets.java:183)
        at org.infinispan.statetransfer.OutboundTransferTask.<init>(OutboundTransferTask.java:122)
        at org.infinispan.statetransfer.StateProviderImpl.startOutboundTransfer(StateProviderImpl.java:276)
        at org.infinispan.statetransfer.StateRequestCommand.invokeAsync(StateRequestCommand.java:100)
        at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:117)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Environment

  • Red Hat Data Grid
    • 7.3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content