Word Embeddings and Transformer Architecture Deep Dive

Prompt Explain word embeddings and why they beat sparse representations, then walk through the transformer architecture end to end.

Be ready to discuss

• Sparse (one-hot, TF-IDF) vs dense embeddings: dimensionality, no notion of similarity in one-hot space, embeddings capturing semantic/syntactic structure (word2vec CBOW/skip-gram, GloVe; contextual embeddings from BERT-style models vs static vectors).
• Self-attention mechanics: queries/keys/values, scaled dot-product (why the √d_k scaling), attention weights as a soft lookup over the sequence.
• Multi-head attention: parallel subspace projections and what different heads learn.
• Positional encoding: why attention alone is permutation-invariant; sinusoidal vs learned positions.
• Encoder vs decoder blocks: masked self-attention in the decoder, cross-attention, residual connections + layer norm, feed-forward sublayers.
• Why transformers displaced RNNs: parallel training, long-range dependencies, compute trade-off (O(n^2) attention).
• Follow-up areas raised in this round: topic modeling (LDA vs embedding-cluster approaches) and aspect extraction in reviews (rule/seed lexicons vs sequence labeling).