DG 8.3.1 query deadlock in index writer threads under heavy load

Solution Verified - Updated -

Issue

Under heavy load DG 8.3.1 has might incur in a deadlock in index writer threads:

"Hibernate Search - default backend - Worker thread - 0" #98 prio=0 tid=0x62 nid=NA waiting
   java.lang.Thread.State: WAITING
    at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
    - parking to wait for <0x603e22ea> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
    at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
    at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
    at org.hibernate.search.engine.backend.orchestration.spi.BatchingExecutor.submit(BatchingExecutor.java:116)
    at org.hibernate.search.backend.lucene.orchestration.impl.LuceneSerialWorkOrchestratorImpl.doSubmit(LuceneSerialWorkOrchestratorImpl.java:94)
    at org.hibernate.search.backend.lucene.orchestration.impl.LuceneSerialWorkOrchestratorImpl.doSubmit(LuceneSerialWorkOrchestratorImpl.java:21)
    at org.hibernate.search.engine.backend.orchestration.spi.AbstractWorkOrchestrator.submit(AbstractWorkOrchestrator.java:135)
    at org.hibernate.search.backend.lucene.orchestration.impl.LuceneSerialWorkOrchestratorImpl.submit(LuceneSerialWorkOrchestratorImpl.java:21)
    at org.hibernate.search.backend.lucene.orchestration.impl.LuceneSerialWorkOrchestrator.submit(LuceneSerialWorkOrchestrator.java:24)
    at org.hibernate.search.backend.lucene.work.execution.impl.LuceneIndexIndexer.submit(LuceneIndexIndexer.java:125)
    at org.hibernate.search.backend.lucene.work.execution.impl.LuceneIndexIndexer.addOrUpdate(LuceneIndexIndexer.java:70)
    at org.hibernate.search.mapper.pojo.work.impl.PojoTypeIndexer.addOrUpdate(PojoTypeIndexer.java:82)
    at org.hibernate.search.mapper.pojo.work.impl.PojoIndexerImpl.addOrUpdate(PojoIndexerImpl.java:48)
    at org.infinispan.search.mapper.work.impl.SearchIndexerImpl.addOrUpdate(SearchIndexerImpl.java:48)
    at org.infinispan.query.backend.QueryInterceptor.updateIndexes(QueryInterceptor.java:378)
    at org.infinispan.query.backend.QueryInterceptor.lambda$processChange$7(QueryInterceptor.java:428)
    at org.infinispan.query.backend.QueryInterceptor$$Lambda$1430/0x0000000840cc9c40.apply(Unknown Source)
    at java.base@11.0.16/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
    at java.base@11.0.16/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
    at java.base@11.0.16/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
    at org.hibernate.search.backend.lucene.orchestration.impl.LuceneBatchedWork.submitTo(LuceneBatchedWork.java:27)
    at org.hibernate.search.backend.lucene.orchestration.impl.LuceneBatchedWork.submitTo(LuceneBatchedWork.java:14)
    at org.hibernate.search.engine.backend.orchestration.spi.BatchingExecutor$BatchWorker.work(BatchingExecutor.java:173)
    at org.hibernate.search.engine.backend.orchestration.spi.SingletonTask$RunnableWrapper.run(SingletonTask.java:191)
    at java.base@11.0.16/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base@11.0.16/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   Locked ownable synchronizers:
    - <0x249111b9> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Environment

  • Red Hat Data Grid (RHDG)
    • 8.x
    • Hibernate Search

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content