How Kafka Works Internally

Q: Explain how Kafka works internally. A: Kafka is a distributed publish-subscribe log. Producers write records to topics, which are split into partitions for parallelism and ordering guarantees (order is only guaranteed within a partition). Each partition is an append-only log replicated across brokers (leader + followers) for fault tolerance. Consumers, organized into consumer groups, read partitions at their own pace tracked via offsets (historically in ZooKeeper, now in an internal Kafka topic __consumer_offsets). Kafka relies on the OS page cache and sequential disk I/O for high throughput, uses a leader-based replication protocol (ISR - in-sync replicas) for durability, and batches/compresses messages to reduce network overhead. Interviewers commonly probe partitioning strategy, consumer group rebalancing, and delivery semantics (at-most-once, at-least-once, exactly-once).