Backtest validation checklist: twelve checks
Short answer
Before you trust a backtest, check twelve things: the overall summary, drawdown, consistency by period, the trade distribution, worse costs, resampled history, neighbouring settings, walk-forward, overfitting, lookahead, the recent trades and the written specification.
1. Scorecard
The summary: the score, the verdict, net profit, drawdown and the key checks in one place.
S = min(sum over k of w_k s_k, c): a weighted score, capped by c when a key check fails.
2. Equity and drawdown
Net profit after costs over time, and the deepest drawdowns with how long each took to recover.
DD_t = max over s <= t of E_s, minus E_t: the drop of equity E below its running peak.
3. Period consistency
Profit by month, week and year, so one lucky stretch cannot hide a weak idea.
p+ = #{m : pi_m > 0} / M: the share of months with a profit.
4. Trade distribution
How often the idea trades, how big the winners and losers are, whether the win rate beats the break-even for that win-loss ratio, and how trades end: at the target, at the stop or on time.
kappa = m4 / m2^2 - 3: the excess kurtosis of trade results (fat tails show up here).
5. Cost stress
The same idea with worse fees and slippage. Thin edges disappear here.
Pi(c) = sum over trades of (g_i - 2cv - f): gross result minus c ticks of slippage per side at tick value v, minus the fee.
6. Monte Carlo
History is resampled many times, in blocks of trading days, to show how bad luck could have looked.
p5 = Q_0.05(E_T^(b)): the 5th percentile of final net E_T over the resampled paths b.
7. Parameter stability
Neighbouring settings are tested too. An idea that works at one exact setting only is fragile.
r = |N+| / |N|: the share of neighbouring settings that stay profitable, also under stress.
8. Walk-forward
The idea is tuned on one slice of history and judged on the next one, again and again.
theta_i = argmax over theta of Pi(theta; B_<i): chosen on the blocks before block i, then judged on block i.
9. Overfitting (PBO)
The probability of backtest overfitting: how likely it is that the best setting from testing was just the luckiest one.
PBO = P(lambda < 0) with lambda = ln(w / (1 - w)), w the out-of-sample rank of the in-sample winner.
10. Lookahead
Proof that every decision used only data that existed at that moment.
d_t(x~) = d_t(x) for every t <= tau, where x~ changes only the data after tau.
11. Recent trades
The latest trades with entry and exit times, profit in dollars and the move in ticks from entry.
pi_i = q_i v (Delta_i - s_i) - f_i: contracts times tick value times the move in ticks net of slippage, minus fees.
12. Specification
The idea in plain words, its rules, how it was chosen and a record of the run.
theta* = argmax over the grid Theta of the net on the selection window: how the setting was chosen.
How to use the list
Start with the checks that can kill an idea outright: lookahead, costs and overfitting. There is no point studying the equity curve of a strategy that reads the future. Then read the trades yourself, because a clean report can still describe an idea you would never trade.
The formulas above are textbook definitions. They show what each check measures, not where any pass mark sits.
How BenchAlgo handles it
These twelve are the screens of the report every run ends with. Each scored screen ends in a Passed, Weak, Failed or N/A badge, and the run ends with a score from 0 to 10 and a verdict: Lead, Validated or Rejected.