ArrayIndexOutOfBoundsException after cache segments changed

Issue

• I changed the segments setting from default 256 to 512 as below:

<distributed-cache name="TestCache" owners="2" mode="SYNC" segments="512" remote-timeout="30000" start="EAGER" statistics="true">
        <locking isolation="READ_COMMITTED" striping="true" acquire-timeout="30000" concurrency-level="1000"/>
         <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
        <memory>
            <object size="524000000" strategy="LRU"/>
        </memory>
        <state-transfer await-initial-transfer="false" enabled="true" timeout="240000" chunk-size="1024"/>
 </distributed-cache>

Then I restarted my cluster members one by one. After that, simply putting & getting cache entries will cause ArrayIndexOutOfBoundsExceptionlike below:

[root@jdg733-sh1 bin]# ./cli.sh -c --controller=127.0.0.1 --commands="container clustered,cache TestCache,put testKey1 testValue1"
java.lang.ArrayIndexOutOfBoundsException: 404

2020-02-20 14:29:53,898 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (management-handler-thread - 3) ISPN000136: Error executing command PutKeyValueCommand on Cache 'TestCache', writing keys [WrappedByteArray{bytes=[B0x033E08746573744B..[11], hashCode=0}]: java.lang.ArrayIndexOutOfBoundsException: 404
        at org.infinispan.distribution.LocalizedCacheTopology.getDistribution(LocalizedCacheTopology.java:213)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.isLockOwner(AbstractLockingInterceptor.java:266)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.needRemoteLocks(PessimisticLockingInterceptor.java:228)
        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataWriteCommand(PessimisticLockingInterceptor.java:147)
        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:82)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150)
        at org.infinispan.interceptors.impl.TxInterceptor.handleWriteCommand(TxInterceptor.java:402)
        at org.infinispan.interceptors.impl.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:235)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
        at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:185)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:260)
        at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:250)
        at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:96)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:150)
        at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:217)
        at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:179)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123)
        at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
        at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:56)
        at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:60)
        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:73)
        at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
        at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
        at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1939)
        at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1917)
        at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1433)
        at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:2043)
        at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:351)
        at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:427)
        at org.infinispan.cli.interpreter.statement.PutStatement.execute(PutStatement.java:76)
        at org.infinispan.cli.interpreter.Interpreter.execute(Interpreter.java:144)
        at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.lambda$executeInterpreter$7(SecurityActions.java:274)
        at org.infinispan.security.Security.doPrivileged(Security.java:59)
        at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.doPrivileged(SecurityActions.java:83)
        at org.jboss.as.clustering.infinispan.subsystem.SecurityActions.executeInterpreter(SecurityActions.java:275)
        at org.jboss.as.clustering.infinispan.subsystem.CliInterpreterHandler.execute(CliInterpreterHandler.java:70)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1411)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
        at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:265)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:231)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:240)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$400(ModelControllerClientOperationHandler.java:138)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:162)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:158)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:246)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:254)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:225)
        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:158)
        at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
        at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:485)

What I don't understand is, I already resarted all the servers one by one, but the issue still persists.

• Also, on some cluster member logs, below warning:

2020-02-20 15:09:44,762 WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p2-t2) ISPN000071: Caught exception when handling command StateRequestCommand{cache=TestCache, origin=node3sh, type=START_STATE_TRANSFER, topologyId=6, segments={0 3-8 12-13 16 19 21 27-31 35 39-40 44 50 52-53 57-64 69 73-77 82-83 90 94 97 108-109 112 116-117 126-128 138 144-148 155-158 163-164 167-168 172-173 184 188 194 205 207-209 212-213 221 229 235 241-242 248-249 251-253 259-260 263 273 275 282 284 286 288-293 295-297 299 303 309 313-314 318-319 321 326-328 330-335 337-338 348 352 357 369-370 379-380 383-384 386-387 393 401 407-408 414 417-418 420 424-425 427 436 439-440 442-443 446-448 450 453 455-458 460-461 464 466-467 482-483 489-490 492 497 503-507 511}}: java.lang.IllegalArgumentException: Provided integer was larger than originally initialized size
        at org.infinispan.commons.util.ConcurrentSmallIntSet.checkBounds(ConcurrentSmallIntSet.java:64)
        at org.infinispan.commons.util.ConcurrentSmallIntSet.add(ConcurrentSmallIntSet.java:100)
        at org.infinispan.commons.util.ConcurrentSmallIntSet.set(ConcurrentSmallIntSet.java:117)
        at org.infinispan.commons.util.SmallIntSet.forEach(SmallIntSet.java:400)
        at org.infinispan.commons.util.IntSets.concurrentCopyFrom(IntSets.java:183)
        at org.infinispan.statetransfer.OutboundTransferTask.<init>(OutboundTransferTask.java:122)
        at org.infinispan.statetransfer.StateProviderImpl.startOutboundTransfer(StateProviderImpl.java:276)
        at org.infinispan.statetransfer.StateRequestCommand.invokeAsync(StateRequestCommand.java:100)
        at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:117)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
        at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)