Hawkular metrics cassandra CorruptSSTableException in OpenShift

Solution In Progress - Updated -

Issue

  • Hawkular metrics cassandra fails with CorruptSSTableException

    INFO  03:47:25 Opening /cassandra_data/data/hawkular_metrics/data-RANDOM-STRING/FILENAME (841306 bytes)
    ERROR 03:47:25 Exiting forcefully due to file system exception on startup, disk failure policy "stop"
    org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.UTFDataFormatException: malformed input around byte 3
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:125) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:86) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:142) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:101) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:186) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:178) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:701) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:662) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:456) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:361) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:499) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
    Caused by: java.io.UTFDataFormatException: malformed input around byte 3
    at java.io.DataInputStream.readUTF(DataInputStream.java:634) ~[na:1.8.0_121]
    at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.8.0_121]
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:101) ~[apache-cassandra-2.2.1.redhat-2.jar:2.2.1.redhat-2]
    ... 15 common frames omitted
    

Environment

  • OpenShift Container Platform 3.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.