Cassandra pod failing with corruption errors in OCP 3
Issue
- OpenShift metrics isn't working.
- Cassandra and hawkular pods are throwing errors.
-
Cassandra has errors like the following:
Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/cassandra_data/data/hawkular_metrics/data_compressed-[HASH]/mc-113-big-Data.db): corruption detected, chunk at 1708577 of length 21043.
-
Cassandra fails with
CorruptSSTableException
:INFO 03:47:25 Opening /cassandra_data/data/hawkular_metrics/data-RANDOM-STRING/FILENAME (841306 bytes) ERROR 03:47:25 Exiting forcefully due to file system exception on startup, disk failure policy "stop" org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.UTFDataFormatException: malformed input around byte 3 at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:125) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:701) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:662) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:456) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:361) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:499) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] Caused by: java.io.UTFDataFormatException: malformed input around byte 3 at java.io.DataInputStream.readUTF(DataInputStream.java:634) ~[na:1.8.0_121] at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_121] at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:101) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2] ... 15 common frames omitted
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.