How to tell if a backtest is overfit
Short answer
Judge the strategy on data it was not tuned on, test the settings next to the chosen one, and count every variant you tried. An overfit backtest looks best at one exact setting on one stretch of history and falls apart everywhere else.
Why good backtests are often overfit
Try enough variants on the same history and one of them will look great by luck alone. The more variants you try, the better the best one looks, even when none of them has an edge. Cheap compute and AI-written code make this easier to do by accident.
Walk-forward testing
Tune the settings on one slice of history, then judge them on the next slice, which the tuning never saw. Move forward and repeat until the history runs out. Only the results on the unseen slices count.
theta_i = argmax over theta of Pi(theta; B_<i): chosen on the blocks before block i, then judged on block i.
Neighbouring settings
Run the settings around the chosen one. A real effect usually fades gradually as you move away from the best setting. If the profit exists at exactly one value and its neighbours lose money, you found noise.
r = |N+| / |N|: the share of neighbouring settings that stay profitable, also under stress.
The probability of backtest overfitting
Split the history into equal blocks and form many ways of choosing half of them for tuning and the other half for testing. In each split, take the setting that did best on the tuning half and see where it ranks on the testing half. The probability of backtest overfitting (PBO) is the share of splits where that winner lands in the bottom half of the ranking.
PBO = P(lambda < 0) with lambda = ln(w / (1 - w)), w the out-of-sample rank of the in-sample winner.
A high PBO means the process that picked your best setting tends to pick a loser. The method comes from Bailey, Borwein, López de Prado and Zhu, linked below.
Count every trial
Every attempt counts, including runs that crashed or were thrown away. The same best result from ten variants and from two hundred variants is very different evidence. Keep the count per research campaign and never reset it.
Warning signs
- Nearly all the profit comes from a handful of trades or from one year.
- The best setting sits at the edge of the range you tested.
- Adding one more filter always improves the curve.
- Results on the unseen slices are far weaker than on the tuning slices.
How BenchAlgo handles it
- Walk-forward, Parameter stability and Overfitting (PBO) are screens in every report.
- A strategy declares a grid of at most 512 settings, and the platform runs it. The neighbours are tested, not assumed.
- Every run counts as a trial, per campaign, and the count cannot be reset.
- A failed key check caps the score, however good the curve looks.