Correlated observations: how a backtest lies to the person who ran it

Every backtest reports a sample size. Almost none of them report the *effective* sample size, and the gap between those two numbers is where most published edges quietly die.

What we found

Our NBA work ran across 1,070 backtest games. That is the number you would put in a headline. It is not the number that governs the error bars, because backtest observations are not independent draws — they share pitchers, parks, weather, market-maker priors and, most of all, each other.

Today's 8-game MLB board is a clean illustration. Across those 8 games, 8 of 8 flagged total edges point over, led by SEA @ TOR where the model prices a fair 10.05 against a line of 7.5 — a +2.55-run gap. If our run-environment assumption is even slightly hot, all 8 of those are wrong together. They are not 8 observations. On the dimension that matters, they are closer to one, repeated 8 times.

The same clustering shows on the moneyline side. The model leans underdog in 6 spots today — COL ML +208, LAA ML +174, CIN ML +127 among them — with the top read at COL @ ATL, where the model says 37.9% against a +208 price for EV +16.9%. Six correlated dog leans on one slate is one bet on "the market overprices favourites tonight," sliced six ways. A backtest that counts each as a separate row will show a beautifully tight confidence interval around a number it has no right to be confident about.

Breakeven at standard juice is 52.38%. The distance between a signal that clears it and a signal that only appeared to is usually not the point estimate. It is the variance you understated by treating correlated rows as independent ones.

The trends

The market gives you an independent read on the same question, and today it is loud.

Concentrated money is landing pregame: the top campaign is $57,404 on 340LADDET LAD ML at 97% one-sided. Behind it, $36,549 on MIL ML at 90% one-sided and $27,155 on BOS ML at 98% one-sided. Total tracked whale money on today's games is $239,330 across 12 campaigns — while the biggest single ticket on the board is only $6,200. That shape matters: a campaign is many tickets moving one direction, which is itself a correlated-observation problem on the money side.

Lines are drifting toward that money. 340LADDET-LAD went 55.5c -> 62.5c (+7.0) over the last 20 hours, 340KCCLE-CLE 60.5 -> 63.5c (+3.0), 335BOSNYY-BOS 51.0 -> 53.0c (+2.0). The other side moved with it: 335BOSNYY-NYY drifted away, 49.0c -> 46.5c (-2.5). 335COLATL-ATL crept 66.5c -> 67.5c (+1.0) — against the model's COL lean.

Pitching trends cut both ways. Jack Perkins (ATH) is trending the wrong way — 8.5 ERA over his last 5, 11.68 over his last 3, against a season mark of 6.68. Cal Quantrill (TEX) is the board's hottest arm at 0.93 ERA over his last 5 with 8.7 K/9. Both are small windows, and both are exactly the kind of input that makes a slate's edges correlate.

What it means

The fix is not to distrust backtests. It is to price the clustering. We block observations by slate, by starter and by run-environment regime, and we require a signal to survive out-of-sample grading before it earns size. That is what the proven-signal book is for: VELO_DROP is holding out-of-sample at 16.7% ROI over 138 graded bets, TRAILING_LATE at 16.3% over 206, THIRD_TIME_THROUGH at 6.3% over 55. Those are graded results, not backtest rows — and the n on the third one is small enough that we say so.

Across 428 indexed data points and 9 agents, the discipline is the same: the headline sample size is a claim, and the effective sample size is the finding.

What would change our mind: if slate-blocked resampling reproduced the same confidence intervals as naive row-counting, the correlation penalty would be overstated and we would loosen it. 21+ where legal.