Logging Pod Consuming CPU and Memory in OpenShift Enterprise 3
Issue
[2016-06-07 09:25:58,843][WARN ][cluster.action.shard ] [Contessa] [.operations.2016.06.03][2] received shard failed for [.operations.2016.06.03][2], node[eXOvc4_jQyyp5dFCJE8ZBw], [P], s[INITIALIZING], indexUUID [K3dawwT3TPWrKx43aXdsmg], reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[.operations.2016.06.03][2] failed to recover shard]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x348392f5, got: 0x0]; ]]
[2016-06-07 09:25:58,846][WARN ][index.engine ] [Contessa] [.operations.2016.06.03][1] failed to sync translog
[2016-06-07 09:25:59,189][WARN ][indices.cluster ] [Contessa] [[.operations.2016.06.03][1]] marking and sending shard failed due to [failed recovery]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [.operations.2016.06.03][1] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:290)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog stream is corrupted, expected: 0x7831da06, got: 0x0
at org.elasticsearch.index.translog.ChecksummedTranslogStream.verifyChecksum(ChecksummedTranslogStream.java:51)
at org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:74)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:260)
Environment
- Red Hat OpenShift Enterprise
- 3.1.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.