Design a Distributed Full-Text Search Engine

Problem

Design Elasticsearch's core: a distributed full-text search engine over sharded, replicated documents.

Requirements

Functional:

• Index documents; full-text and field queries
• Shard and replicate for scale + redundancy
• Ranked results (BM25)
• Near-real-time indexing

Non-functional:

• Billions of documents
• Low-latency search, high availability
• Survive node failures

Discussion points

1. Inverted index + sharding strategy
2. Replication and node roles (data/master/ingest)
3. Scatter-gather query + ranking
4. Near-real-time refresh and segment merges
5. Cluster state and failover