Design a Scalable Document Search Service
Problem
Design a full-text document search service over a large enterprise corpus.
Requirements
Functional:
- Index and search millions of documents
- Ranked results with snippets
- Access control per document
- Incremental indexing on updates
Non-functional:
- 10s of millions of docs
- Sub-second search
- Freshness within minutes
Discussion points
- Inverted index and sharding
- Ranking (TF-IDF/BM25)
- Incremental indexing pipeline
- ACL filtering at query time
- Caching hot queries
added …