Backtesting: The Seven Deadly Sins

A backtest is a simulation of a strategy on history — and the most dangerous artifact in quant finance, because it looks like evidence while every incentive pushes it toward flattery. Interviewers at every firm ask some version of "why is this backtest wrong?" This is the checklist that answers it.

The seven sins

1. Lookahead bias. Using information before it existed: closing prices to trade that same close, earnings "as of" quarter-end that were announced weeks later, restated financials. The fix is point-in-time data and a hard rule: every feature timestamped, every trade strictly after its information. This is the most common fatal bug, and the first thing to hunt.

2. Survivorship bias. Backtesting on today's index members means testing on companies that survived — the delisted, bankrupt, and acquired vanish from the sample. Long-only equity backtests on survivor-only data overstate returns by percentage points per year. Ask of any dataset: where are the dead?

3. Overfitting / selection bias. The multiple-testing sin from the validation lesson at industrial scale: thousands of parameter sets and signal variants tried, the best shown. The best of $n$ noise draws looks like skill (the order-statistics lesson, weaponized). Defenses: held-out periods touched once, walk-forward evaluation, counting everything tried (deflated Sharpe), and suspicion of any strategy with more parameters than trades... or whose story arrived after its numbers.

4. Ignored costs. Spreads, fees, borrow costs for shorts, and market impact — which scales with size and against you. A strategy trading daily with 0.1% per-trade edge dies at 0.1% round-trip costs; high-turnover backtests without a cost model aren't optimistic, they're fiction. Impact modeling matters more the better the gross numbers look.

5. Capacity confusion. Great returns on \1M often can't absorb \100M — the edge is in small/illiquid names or fleeting microstructure. The backtest question isn't "does it work?" but "at what size does it stop working?"

6. Regime blindness. Ten years of data = one macro regime, maybe two. A carry strategy tested 2010–2020 never met 2022's rate shock. Stress the strategy on the eras that would have hurt it most — and if the sample doesn't contain such an era, say so out loud rather than extrapolating calm.

7. Execution fantasy. Assuming fills at mid, at the close print, at sizes the book never showed; ignoring shorting constraints and locate availability. The sim must trade the way you actually could — the microstructure course is the reason why.

The discipline that survives the checklist

Hypothesis before mining, point-in-time everything, walk-forward always, one untouched final period, cost and impact models scaled to intended size, a count of experiments run, and pre-registered kill criteria for live monitoring (the ML-in-markets lesson's ending). None of it is glamorous; all of it is the job. A useful mental reframe from practitioner culture: a backtest is not evidence that a strategy works — it's a check that your hypothesis isn't obviously false. The ordering of those two sentences is the entire epistemology of quant research.

The interview version

You'll be shown a beautiful equity curve and asked for reactions. Run the list in priority order: lookahead? survivorship? how many variants were tried? costs and turnover? capacity at real size? which regimes does the sample include? realistic fills? Seven questions, thirty seconds, and — the meta-signal — visible disappointment at a Sharpe of 3, because you know base rates. The candidate who wants the backtest to be wrong, and only then believes it, is the one who gets hired to run real money.