Cassandra pod failing with corruption errors in OCP 3

Solution Unverified - Updated -

Issue

  • OpenShift metrics isn't working.
  • Cassandra and hawkular pods are throwing errors.
  • Cassandra has errors like the following:

    Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/cassandra_data/data/hawkular_metrics/data_compressed-[HASH]/mc-113-big-Data.db): corruption detected, chunk at 1708577 of length 21043.
    
  • Cassandra fails with CorruptSSTableException:

    INFO  03:47:25 Opening /cassandra_data/data/hawkular_metrics/data-RANDOM-STRING/FILENAME (841306 bytes)
    ERROR 03:47:25 Exiting forcefully due to file system exception on startup, disk failure policy "stop"
    org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.UTFDataFormatException: malformed input around byte 3
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:125) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:701) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:662) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:456) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:361) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:499) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
    Caused by: java.io.UTFDataFormatException: malformed input around byte 3
    at java.io.DataInputStream.readUTF(DataInputStream.java:634) ~[na:1.8.0_121]
    at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_121]
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:101) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    ... 15 common frames omitted
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content