Kafka Partitions and Consumer Group Behavior

Q: Explain Kafka partitions and how consumers interact with them.

A: Key points to cover:

• A Kafka topic is split into partitions, each an ordered, append-only log; partitioning enables parallelism since different partitions can be produced to and consumed from independently.
• Within a consumer group, each partition is consumed by exactly one consumer at a time, so the maximum useful parallelism for a group is bounded by the number of partitions - more consumers than partitions leaves some consumers idle.
• Ordering is only guaranteed within a single partition, not across the whole topic; messages needing relative ordering (e.g., events for the same user/order) should share a partition key.
• Consumer offsets are tracked per partition per consumer group, allowing independent consumer groups to read the same topic at their own pace, and allowing rebalancing when consumers join/leave a group (partition reassignment).
• Rebalancing triggers a stop-the-world pause for the group while partitions are reassigned; frequent rebalances (e.g., due to slow consumers hitting session timeouts) hurt throughput.