CrashLoopBackOff repeats with "java.io.IOException: No space left on device" in Kafka pod

Issue

• CrashLoopBackOff[1] repeats with "java.io.IOException: No space left on device" in Kafka pod[2]

[1] CrashLoopBackOff status in Kafka pod

apiVersion: v1
kind: Pod
metadata:
...
  name: my-cluster-kafka-0
...
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-11-11T09:51:51Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-11-11T09:59:02Z"
    message: 'containers with unready status: [kafka]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-11-11T09:59:02Z"
    message: 'containers with unready status: [kafka]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-11-11T09:51:51Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: cri-o://9472baad3cfceca2b2f8d7e228b7fff3fda7b396e60e6299c19a0072530a7caf
    image: registry.redhat.io/amq7/amq-streams-kafka-31-rhel8:2.1.0
    imageID: registry.redhat.io/amq7/amq-streams-kafka-31-rhel8@sha256:1572ced1a3ad802e778f73d6babba532b1e87328183e2e817c2c0980b81ddd26
    lastState:
      terminated:
        containerID: cri-o://9472baad3cfceca2b2f8d7e228b7fff3fda7b396e60e6299c19a0072530a7caf
        exitCode: 1
        finishedAt: "2022-11-11T10:00:31Z"
        reason: Error
        startedAt: "2022-11-11T10:00:22Z"
    name: kafka
    ready: false
    restartCount: 5
    started: false
    state:
      waiting:
        message: back-off 2m40s restarting failed container=kafka pod=my-cluster-kafka-0_p03357335(a3ce4b91-ef8e-4b48-ac99-ffc72ff5ea36)
        reason: CrashLoopBackOff

[2] "java.io.IOException: No space left on device"

2022-11-11 10:08:46,355 ERROR Error while appending records to my-topic-0 in dir /var/lib/kafka/data-0/kafka-log0 (kafka.server.LogDirFailureChannel) [ReplicaFetcherThread-0-1]
java.io.IOException: No space left on device
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
        at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280)
        at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:92)
        at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:188)
        at kafka.log.LogSegment.append(LogSegment.scala:158)
        at kafka.log.LocalLog.append(LocalLog.scala:394)
        at kafka.log.UnifiedLog.append(UnifiedLog.scala:907)
        at kafka.log.UnifiedLog.appendAsFollower(UnifiedLog.scala:736)
        at kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica(Partition.scala:1011)
        at kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica(Partition.scala:1018)
        at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:180)
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:349)
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:338)
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:337)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry(JavaCollectionWrappers.scala:359)
        at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry$(JavaCollectionWrappers.scala:355)
        at scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.foreachEntry(JavaCollectionWrappers.scala:309)
        at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:337)
        at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:137)
        at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:136)
        at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:119)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2022-11-11 10:08:46,376 ERROR [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error while processing data for partition my-topic-0 at offset 1003 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
org.apache.kafka.common.errors.KafkaStorageException: Error while appending records to my-topic-0 in dir /var/lib/kafka/data-0/kafka-log0
Caused by: java.io.IOException: No space left on device
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
        at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280)
        at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:92)
        at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:188)
        at kafka.log.LogSegment.append(LogSegment.scala:158)
        at kafka.log.LocalLog.append(LocalLog.scala:394)
        at kafka.log.UnifiedLog.append(UnifiedLog.scala:907)
        at kafka.log.UnifiedLog.appendAsFollower(UnifiedLog.scala:736)
        at kafka.cluster.Partition.doAppendRecordsToFollowerOrFutureReplica(Partition.scala:1011)
        at kafka.cluster.Partition.appendRecordsToFollowerOrFutureReplica(Partition.scala:1018)
        at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:180)
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:349)
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:338)
        at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:337)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry(JavaCollectionWrappers.scala:359)
        at scala.collection.convert.JavaCollectionWrappers$JMapWrapperLike.foreachEntry$(JavaCollectionWrappers.scala:355)
        at scala.collection.convert.JavaCollectionWrappers$AbstractJMapWrapper.foreachEntry(JavaCollectionWrappers.scala:309)
        at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:337)
        at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:137)
        at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:136)
        at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:119)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
2022-11-11 10:08:46,376 WARN [ReplicaManager broker=0] Stopping serving replicas in dir /var/lib/kafka/data-0/kafka-log0 (kafka.server.ReplicaManager) [LogDirFailureHandler]
2022-11-11 10:08:46,384 WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Partition my-topic-0 marked as failed (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2022-11-11 10:08:46,385 ERROR [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error while processing data for partition __strimzi_store_topic-0 at offset 4 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
org.apache.kafka.common.errors.KafkaStorageException: The log dir /var/lib/kafka/data-0/kafka-log0 is already offline due to a previous IO exception.
2022-11-11 10:08:46,385 WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Partition __strimzi_store_topic-0 marked as failed (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
2022-11-11 10:08:46,385 ERROR [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error while processing data for partition __strimzi-topic-operator-kstreams-topic-store-changelog-0 at offset 4 (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
org.apache.kafka.common.errors.KafkaStorageException: The log dir /var/lib/kafka/data-0/kafka-log0 is already offline due to a previous IO exception.
2022-11-11 10:08:46,385 WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Partition __strimzi-topic-operator-kstreams-topic-store-changelog-0 marked as failed (kafka.server.ReplicaFetcherThread) [ReplicaFetcherThread-0-1]
...
2022-11-11 10:08:46,441 WARN Stopping serving logs in dir /var/lib/kafka/data-0/kafka-log0 (kafka.log.LogManager) [LogDirFailureHandler]
2022-11-11 10:08:46,448 ERROR Shutdown broker because all log dirs in /var/lib/kafka/data-0/kafka-log0 have failed (kafka.log.LogManager) [LogDirFailureHandler]