Build a Recommendation Algorithm Using PySpark

Problem Implement a recommendation algorithm using PySpark over large-scale user-item interaction data.

Functional requirements

• Ingest and preprocess raw interaction data (views, clicks, orders) into a user-item matrix.
• Train a collaborative-filtering model producing top-N recommendations per user.
• Fall back to content/feature-based scoring where collaborative signal is too sparse.
• Write recommendations out to a store the serving layer can read.

Non-functional requirements

• Tens of millions of users and millions of items; billions of interaction rows.
• Full retrain runs as a nightly batch job inside a few-hours window.
• Recommendations refreshed daily; serving reads them back in single-digit milliseconds.
• The job must survive skew — a handful of viral items appear in a disproportionate share of rows.

Key components

• Loading and preprocessing with Spark DataFrames: dedup, sessionization, implicit-feedback weighting, and dropping users/items below a minimum interaction count.
• Indexing string user/item IDs to integer indices (StringIndexer) for the factorization.
• Model: Spark MLlib's ALS (Alternating Least Squares), factorizing the user-item matrix into latent factors — chosen because alternating between fixed user factors and fixed item factors parallelizes cleanly across the cluster.
• Content/feature-based scoring path covering cold-start users and items.
• Output stage: recommendForAllUsers, then persisting top-N to a key-value store for serving.
• Orchestration and monitoring of job duration, user coverage, and offline metrics.

Deep dives / trade-offs

• Explicit vs. implicit feedback: ratings are scarce and self-selected, while clicks and orders are abundant but have no true negatives — ALS's implicit mode with confidence weighting handles the latter.
• Partitioning and skew: how to partition interaction data, why popular items create hot partitions and stragglers, and salting/repartitioning to fix it.
• Tuning ALS: rank (number of latent dimensions), regularization, and iteration count, against the shuffle and memory cost of the factor matrices.
• Cold-start: ALS emits NaN for unseen users/items — how coldStartStrategy, popularity fallbacks, and content features cover the gap.
• Batch vs. real-time: nightly factors go stale within a day for a fast-moving catalogue, and where a real-time layer would sit on top.