Design a Product Analytics Ingestion & Query System
Problem
Design Amplitude's core: ingest product analytics events and answer funnel/retention/segmentation queries.
Requirements
Functional:
- Ingest billions of events/day from SDKs
- Compute funnels, retention, and segmentation
- Interactive query latency for dashboards
- Real-time + historical
Non-functional:
- Billions of events/day, high cardinality
- Sub-second interactive queries
- Cost-efficient storage
Discussion points
- Ingestion pipeline (Kafka) and dedup
- Columnar storage for analytical queries
- Pre-aggregation vs on-the-fly funnel computation
- User/session modeling
- Sharding and query parallelism
added …