Design a Reliable Event Ingestion Pipeline (Out-of-Order)

Problem

Design a reliable event ingestion pipeline that handles out-of-order records at scale.

Requirements

Functional:

• Ingest events from many producers
• Handle late/out-of-order events
• Exactly-once-ish downstream delivery
• Replay/backfill support

Non-functional:

• Millions of events/sec
• Durable, ordered per key
• Fault tolerant

Discussion points

1. Kafka topics, partitioning by key, retention
2. Watermarks/windowing for out-of-order events (Flink)
3. Idempotency and exactly-once sinks
4. Consumer groups, offset management, lag
5. Replay and backfill