Design a Metrics Monitoring & Alerting System
Problem
Design Datadog's core: a metrics ingestion, storage, and alerting platform.
Requirements
Functional:
- Ingest billions of metric points/sec from agents
- Store time-series for querying/dashboards
- Evaluate alert rules in near-real-time
- Aggregations (sum/p95) over windows
Non-functional:
- Massive write volume, high cardinality
- Low-latency queries and alerts
- Cost-efficient long retention
Discussion points
- Ingestion pipeline and agent protocol
- Time-series storage (rollups, downsampling, cardinality control)
- Query engine for dashboards
- Alert evaluation engine
- Sharding and back-pressure
added …