Random Forest vs XGBoost: Mechanics and Evaluation Metrics

Problem Compare Random Forest and XGBoost — how does each model work, and how would you evaluate them?

Be ready to discuss

• Random Forest is bagging: many deep trees trained independently and in parallel on bootstrap samples with random feature subsets, then averaged or voted. Averaging decorrelated trees is what reduces variance.
• XGBoost is boosting: trees built sequentially, each fit to the gradient of the loss left over by the ensemble so far. It drives down bias, and needs regularization (learning rate/shrinkage, max depth, subsampling, L1/L2 on leaf weights) to avoid overfitting.
• Consequences of the difference: RF parallelizes, is hard to overfit, and is forgiving of defaults; XGBoost usually wins on accuracy but is tuning-sensitive and inherently sequential.
• What XGBoost adds over a plain GBM: second-order (Newton) approximation of the loss, built-in regularization, sparsity-aware split finding, and histogram/approximate split algorithms for speed.
• Evaluation: start from the confusion matrix, then precision, recall, and F1 — especially on imbalanced classification, where accuracy hides everything. PR-AUC to compare across thresholds rather than one arbitrary cutoff.
• Methodology: proper cross-validation, early stopping on a validation fold for XGBoost, and out-of-bag error as Random Forest's free generalization estimate.