Regression, Overfitting, and Out-of-Sample Truth

Machine learning in finance is less about exotic models and more about a single discipline: never believe in-sample results. This lesson builds that discipline from the bias-variance trade-off up.

Why in-sample performance lies

Fit a model flexible enough and it will "explain" anything — including the noise. Add predictors to a regression and in-sample $R^2$ can only go up, even if the predictors are random numbers. The fitted model has partly memorized the sample it saw, and that memorized component contributes nothing — usually less than nothing — out of sample.

The formal decomposition of expected prediction error:

$$E[(y - \hat{f})^2] = \underbrace{\text{Bias}^2}_{\text{model too simple}} + \underbrace{\text{Variance}}_{\text{model too sensitive to its sample}} + \underbrace{\sigma^2_{\text{noise}}}_{\text{irreducible}}$$

Simple models are biased but stable; flexible models are unbiased but high-variance — they'd change wildly on a different draw of the data. The optimum is in between, and it moves toward simpler as data gets noisier. Financial data is the noisiest data commonly studied (signal-to-noise near zero, non-stationary), which is why finance ML skews so heavily toward simple, regularized models — a point interviewers reward you for making unprompted.

Regularization: buying stability with bias

Instead of choosing model size discretely, penalize coefficient magnitude:

• Ridge (L2): minimize $\sum (y_i - x_i^\top\beta)^2 + \lambda \|\beta\|_2^2$ — shrinks all coefficients smoothly toward zero; equivalent to adding $\lambda I$ to $X^\top X$, which also fixes the multicollinearity blow-up from the linear-algebra lesson. Two problems, one hyperparameter.
• Lasso (L1): penalize $\lambda\|\beta\|_1$ — the corners of the L1 ball drive some coefficients exactly to zero, so it selects features. Less stable than ridge when predictors are correlated (it arbitrarily picks one of a correlated group).

The Bayesian reading is worth one sentence in an interview: ridge is a Gaussian prior on coefficients, lasso a Laplace prior — regularization is a formalized prior belief that most effects are small.

Validation: the only honest scoreboard

• Train / validation / test: fit on train, tune $\lambda$ on validation, touch the test set once. Every time you peek at test performance and change something, the test set quietly becomes a validation set, and its estimate inflates.
• Cross-validation: rotate the held-out fold to use limited data efficiently. But with time series, random folds leak the future into the past (autocorrelation, overlapping observations). Use walk-forward validation: train on the past, test strictly on the subsequent period, roll onward. Saying "k-fold is wrong for returns data, I'd walk forward" is one of the highest-value sentences in a quant-research interview.
• Multiple testing: try 100 signals at the 5% significance level and ~5 look great by luck. Every backtest ever shown to you survived a selection process; the honest question is "how many things were tried?" Corrections exist (Bonferroni, deflated Sharpe ratios), but the cultural fix is registering hypotheses before testing.

The interview version

"Your intern's model has $R^2 = 0.4$ predicting daily returns in-sample. Reaction?" — Daily-return predictability of that size is off by an order of magnitude from anything real; assume leakage (future data in features, target contamination, survivorship) until proven otherwise. Then the checklist: what's the out-of-sample $R^2$, was validation walk-forward, how many model variants were tried, and does the signal survive costs? A candidate who reflexively distrusts good results reads as someone who has lost money to a beautiful backtest — which is the persona these interviews are screening for.