Design a Log Search & Analytics Platform
Problem
Design Splunk's core: a log ingestion, indexing, and search/analytics platform.
Requirements
Functional:
- Ingest high-volume logs from many sources
- Index for fast full-text and field search
- Run aggregation queries over time ranges
- Alerting on search results
Non-functional:
- TBs/day ingestion, multi-tenant
- Fast search over huge volumes
- Long retention
Discussion points
- Ingestion pipeline and parsing
- Time-partitioned inverted index
- Distributed search (scatter-gather)
- Aggregation/query engine
- Retention tiers and cost control
added …