[AMQ Streams] NotEnoughReplicasException for partition 47 of the topic __consumer-offsets-47

Issue

• the Kafka cluster is currently experiencing a "NotEnoughReplicasException" error, leading to the "Not Ready" status. The error is specifically related to the "__consumer_offsets-47" partition. The error message indicates that the size of the In-Sync Replica (ISR) set is insufficient to satisfy the required minimum ISR count of 2. Here's the relevant error log:

  2023-08-10 09:49:40,935 ERROR [ReplicaManager broker=0] Error processing append operation on partition __consumer_offsets-47 (kafka.server.ReplicaManager) [data-plane-kafka-request-handler-0] org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(0) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-47. 2023-08-10 09:49:40,935 INFO [GroupCoordinator 0]: Preparing to rebalance group app-pai-instruction-manager in state PreparingRebalance with old generation 257713 (__consumer_offsets-47) (reason: Error COORDINATOR_NOT_AVAILABLE when storing group assignment during SyncGroup (member: rdkafka-e2bff1cb-054b-417e-9c00-9fd370bd0578)).