Why Sub-word Tokenization (BPE) Over Word-level Tokenization

Question: Why do we use sub-word tokenization (like Byte-Pair Encoding) instead of word-level tokenization? Key points: Word-level tokenization produces huge vocabularies and cannot represent out-of-vocabulary/rare words at all. Sub-word schemes like BPE build a vocabulary of frequent character sequences, letting rare or unseen words be represented as combinations of known sub-word units. This keeps vocabulary size manageable, handles morphological variation (prefixes/suffixes) gracefully, and avoids the OOV problem entirely.