# BenchAlgo: complete reference > BenchAlgo is a backtesting and validation platform for futures trading algorithms. It runs your strategy on real CME market data for ES, NQ and GC with trading costs, checks it for lookahead and overfitting, and scores it out of 10 across twelve validation screens. BenchAlgo is a research tool. It does not give investment advice and it does not place trades. Results come from simulated tests on past data and do not predict future results. Trading futures involves risk of loss. ## BenchAlgo product guide BenchAlgo is a backtesting and validation platform for futures trading algorithms. It runs your strategy on real CME market data for ES, NQ and GC with trading costs, checks it for lookahead and overfitting, and scores it out of 10 across twelve validation screens. Maintained by [Patrik Kelemen](https://benchalgo.com/about/#founder), founder of BenchAlgo. Last reviewed 3 November 2026. Canonical page: https://benchalgo.com/product-guide/ ### Product at a glance BenchAlgo capabilities and limits | Fact | Details | | --- | --- | | Product | BenchAlgo | | Category | Backtesting and validation platform for futures strategies | | Markets | ES, NQ and GC, as full-size or micro (MES, MNQ, MGC) contracts | | Market data | CME Globex data. The plan includes 15 years of OHLC bars, 1 year of tick data and more than 3 months of MBO data, with new MBO data added every month. | | Strategy data | Strategies currently run on completed 1-minute bars. Tick and MBO tiers for strategies are planned. | | Strategy language | Python, against the public strategy API | | Validation | Twelve screens, a score from 0 to 10 and a verdict: Lead, Validated or Rejected | | Who writes the code | An OpenRouter model on our server, or your own Claude Code or Codex | | What you see | Profit and loss in US dollars, and every trade as a move in ticks from entry. Never raw prices. | | Sign-up | Free, no card needed | | Live trading | Not yet. The current workspace cannot place orders. | ### Plan and prices All prices are in USD. One US dollar buys 100 credits. BenchAlgo plan and credit top-ups | Item | Price | Credits | | --- | --- | --- | | Monthly | $99 per month | 9,900 credits included every month | [Pricing](https://benchalgo.com/pricing/) ### Common questions #### What is BenchAlgo and who makes it? BenchAlgo is a backtesting and validation platform for futures trading algorithms. It runs your strategy on real CME market data for ES, NQ and GC with trading costs, checks it for lookahead and overfitting, and scores it out of 10 across twelve validation screens. It keeps every idea your AI tries as a node on a tree, tests each one the same strict way and shows the failures as well as the survivors. BenchAlgo is made by Crowie s. r. o., a software company in Slovakia, and was founded by Patrik Kelemen. Read more: [How it works](https://benchalgo.com/#how-it-works) | [How it differs from a backtester](https://benchalgo.com/faq/#different) | [Who is behind it](https://benchalgo.com/about/#founder) | [Company details](https://benchalgo.com/imprint/) #### What market data does BenchAlgo include? CME Globex data for ES, NQ and GC, with both full-size and micro contracts. The plan includes 15 years of OHLC bars, 1 year of tick data and more than 3 months of MBO data, with new MBO data added every month, and bigger packages are available from support. Strategies currently run on completed 1-minute bars. Tick and MBO tiers for strategies are planned. The data stays on our servers, so there is no feed to buy or set up. Read more: [Data in the FAQ](https://benchalgo.com/faq/#data) | [What the plan includes](https://benchalgo.com/pricing/) | [ES futures](https://benchalgo.com/guides/es-futures-backtesting/) #### How does a strategy get tested? You describe the idea in plain words and an AI writes it as Python against the strategy API. The harness runs it on history with rules it sets itself: fills at the next bar open, the stop checked first inside a bar, and fees and slippage a strategy cannot lower. Then the twelve validation screens run, and the result becomes a node with its score, P&L and verdict. Read more: [The four steps](https://benchalgo.com/#how-it-works) | [Next-bar fills](https://benchalgo.com/lookahead-checks/#fills) | [The twelve screens](https://benchalgo.com/how-scoring-works/#screens) #### What does the score out of 10 mean? It reflects how well an idea holds up across all the checks, not just whether one backtest made money. It is not a forecast. A failed key check caps the score however good the other screens look, and so does any eligible week without a trade entry. Read more: [What the score means](https://benchalgo.com/how-scoring-works/#score) | [What caps the score](https://benchalgo.com/how-scoring-works/#cap) #### What score counts as a pass? A final score of 7 or higher, with complete required checks and at least one trade entry in every eligible calendar week, makes a strategy a candidate for production review. The score alone does not approve it for trading: holdout validation, forward testing and your approval still come next. Read more: [From discovery to production](https://benchalgo.com/how-scoring-works/#production) #### How does BenchAlgo check for lookahead? It runs the strategy on the full data and again with everything after a chosen moment removed or changed, and every earlier decision must stay identical. It also replays the strategy bar by bar as if live, and the trades must match the fast backtest exactly. A failed check blocks an idea from becoming a Lead or Validated. Read more: [Cut the future](https://benchalgo.com/lookahead-checks/#truncation) | [Change the future](https://benchalgo.com/lookahead-checks/#perturbation) | [Replay as if live](https://benchalgo.com/lookahead-checks/#replay) #### How does BenchAlgo check for overfitting? Several checks work together. Walk-forward testing judges each setting on history it was not tuned on. Parameter stability tests the neighbouring settings. The probability of backtest overfitting (PBO) estimates whether the best setting was just the luckiest. Random entries and random direction ask whether chance does as well. And every run counts as a trial, so failed attempts cannot be quietly forgotten. Read more: [The twelve screens](https://benchalgo.com/how-scoring-works/#screens) | [Random entries](https://benchalgo.com/lookahead-checks/#nulls) | [Why invalid runs still count](https://benchalgo.com/lookahead-checks/#trials) #### How much does BenchAlgo cost? The Monthly plan costs $99 per month and includes 9,900 credits. One US dollar buys 100 credits. Creating an account is free and needs no card. Payments run in test mode while BenchAlgo is in preview. No real money moves. Read more: [Pricing](https://benchalgo.com/pricing/) #### What do credits pay for, and what is free? Credits pay for AI usage on our server and for backtests, which are charged by the CPU minutes they use. Creating an account, reading your tree and reports, and checking a strategy with st-mcp do not use credits. Your own Claude Code or Codex does not use credits for the AI, only for backtests. Read more: [Pricing](https://benchalgo.com/pricing/) | [Credits in the FAQ](https://benchalgo.com/faq/#credits) #### Which AI can write the strategies? You choose. Use an OpenRouter model on our server, paid with credits, or connect your own Claude Code or Codex with the st-mcp tools, on your own subscription. The tests are the same either way. Read more: [Which AI writes the strategies](https://benchalgo.com/faq/#ai) | [Bring your own](https://benchalgo.com/faq/#own-ai) | [Backtest a strategy written by AI](https://benchalgo.com/guides/backtest-ai-trading-strategy/) #### What does the AI see of my data and results? The AI gets the public strategy API, your strategy code and a short written feedback note: the verdict, a rounded score and which checks failed. It never sees market data, trades or curves. Read more: [What the AI sees](https://benchalgo.com/faq/#ai-sees) #### Are my strategies private? Yes. Strategies, chats and reports are private to your account, and strategy code runs only in our sandbox. If an operator ever opens your data to help you, that access is written to an audit log. Read more: [Privacy in the FAQ](https://benchalgo.com/faq/#private) | [Where your code runs](https://benchalgo.com/faq/#own-ai) #### Do I see raw market prices? No. You see profit and loss in US dollars, and every trade as a move in ticks from its entry. Read more: [What you see in the results](https://benchalgo.com/faq/#results) #### Can BenchAlgo trade live? Not yet. BenchAlgo has no broker connection and does not place orders, and the Deploy page in the app is switched off. The planned next step is deployment through the interface after holdout validation, forward testing and your approval, starting with Topstep and Tradovate. Exporting a strategy to run it yourself is also planned. Read more: [Live trading in the FAQ](https://benchalgo.com/faq/#live) | [From discovery to production](https://benchalgo.com/how-scoring-works/#production) | [The two roads](https://benchalgo.com/#run-it) #### Does a high score mean a strategy will make money? No. A high score means the idea survived strict tests on past data. It does not predict future results. Markets change, and every backtest is a simulation. Read more: [Score and profit](https://benchalgo.com/faq/#score-money) | [What scoring does not do](https://benchalgo.com/how-scoring-works/#honest) ### Testing a strategy before you trust it #### How do I check a backtest for lookahead bias? Run the same strategy twice: once on the full data and once with everything after a chosen moment removed or scrambled. Every decision made before that moment must be identical in both runs. Then make sure fills happen after the signal, at the next bar open or later, and that higher-timeframe values only come from completed bars. You can script this around most backtesters. BenchAlgo runs it on every strategy. Read more: [How to check for lookahead bias](https://benchalgo.com/guides/lookahead-bias/) | [Lookahead checks](https://benchalgo.com/lookahead-checks/) #### How do I tell if a backtest is overfit? Judge the strategy on data it was not tuned on, with walk-forward testing. Test the neighbouring settings, because a result that only works at one exact setting is fragile. Count every variant you tried, because the more you try on the same data, the easier it is to find a lucky one, and estimate the probability of backtest overfitting (PBO). No single check proves an edge, but several failing together usually rule one out. Read more: [How to tell if a backtest is overfit](https://benchalgo.com/guides/backtest-overfitting/) | [The twelve screens](https://benchalgo.com/how-scoring-works/#screens) #### How do I know a strategy beats chance? Compare it with many random versions of itself: the same number of trades per session, the same stops, targets and costs, but random entry times and random sides. If its net result sits inside their range, chance explains it. The share of random runs that do at least as well gives a p-value. Read more: [The random-entry test](https://benchalgo.com/guides/random-entry-test/) | [Random entries and direction](https://benchalgo.com/lookahead-checks/#nulls) #### What should I test before trading a futures strategy live? At minimum: lookahead, costs and slippage, drawdown, consistency by month and year, stability of the settings and results on data it was not tuned on. After that comes forward testing on data the strategy has never seen, ideally on paper or at the smallest size first. A backtest is a simulation, and passing one does not predict future results. Read more: [Backtest validation checklist](https://benchalgo.com/guides/backtest-validation-checklist/) | [What scoring does not do](https://benchalgo.com/how-scoring-works/#honest) #### How do I backtest a trading strategy written by AI? Treat the code as untrusted. Run it in a sandbox, keep market data and results away from the model so it cannot fit to them, and let the backtester, not the strategy, decide fills and costs. Count every attempt as a trial, then validate the survivor as strictly as any other idea. BenchAlgo works this way with an OpenRouter model or your own Claude Code or Codex. Read more: [Backtest a strategy written by AI](https://benchalgo.com/guides/backtest-ai-trading-strategy/) | [What the AI sees](https://benchalgo.com/faq/#ai-sees) #### How much slippage should I assume in a futures backtest? There is no single right number. It depends on the contract, the time of day, the order type and your size. Model it per side in ticks, add the real commission, then test the strategy again with costs doubled. If the edge disappears under higher costs, it is too thin to rely on. BenchAlgo charges at least 1.5 ticks per side on market and stop fills, and a strategy cannot lower it. Read more: [Fills, slippage and fees](https://benchalgo.com/guides/backtest-costs-and-fills/) | [Double the costs](https://benchalgo.com/lookahead-checks/#costs) ### Try one idea 1. **Create a free account.** No card needed. Add credits when you want to run backtests. 2. **Describe one idea.** Write it in plain words. An AI turns it into Python against the strategy API. It never sees market data. 3. **Read the report.** The run ends with a score, a verdict and twelve screens. Start with the failed checks. 4. **Branch from the result.** Try a tighter stop, a time filter or another market from any node. Every attempt stays on the tree and counts as a trial. ### Before you depend on a result - BenchAlgo is a research tool. It does not give investment advice and it does not place trades. Results come from simulated tests on past data and do not predict future results. Trading futures involves risk of loss. - Backtests are hypothetical. No real money is at risk in them, they are chosen with hindsight, and most ideas fail. - Fills are simulated: the next bar open, the stop first inside a bar, and gaps at the worse price. A live market can fill worse. - Only ES, NQ and GC, with full-size and micro contracts. Strategies currently run on completed 1-minute bars. Tick and MBO tiers for strategies are planned. A strategy holds one position at a time and is flat by the close. - The score covers the data it was run on. The most recent three months stay sealed for now, so no score has been judged on them. For corrections or product questions, contact [support@benchalgo.com](mailto:support@benchalgo.com). ## BenchAlgo: backtest and score your trading algorithm URL: https://benchalgo.com/ Backtest your trading algorithm before it trades. We provide the market data, (15 years of OHLC bars, 1 year of tick data and more than 3 months of MBO data that grows every month, with more as a package: contact support for pricing), and a scoring framework: your algorithm runs on real history with trading costs and earns a score out of 10. If it passes at 7 or higher, you can deploy it on our platform or export it as an application and run it yourself on Linux, Windows or your own computer (both coming soon). Self-hosting is not recommended unless you know how to set it up correctly: a live algorithm needs at least a virtual private server with a high-bandwidth, low-latency connection. ### World progression Expeditions are saved campaigns. The collection shows scored candidates; the graveyard retains rejections and execution failures. Six named encounters display actual lookahead, cost-stress, Monte Carlo, parameter-stability, walk-forward and PBO evidence. Missing evidence is never a win. Rarity follows the one-decimal final capped score: 9.0 to 10.0 Legendary, 8.0 to 8.9 Epic, 7.0 to 7.9 Rare, 6.0 to 6.9 Uncommon, 5.1 to 5.9 Common, 0.0 to 5.0 Unproven. Missing scores are Unranked. A 7+ strategy with complete required checks is a candidate for production review; forward testing and deployment approval still come next. Incomplete evidence remains provisional; rarity never grants deployment or forward validation. ### Production path Bring your own AI to discover and test a trading strategy. A final score of 7 or higher with complete required checks makes it a candidate for production review. Deployment through the interface is planned, after holdout validation, forward testing and operator approval. Deployment is not enabled yet; the current workspace cannot place orders. ## BenchAlgo pricing: plans from $99 a month URL: https://benchalgo.com/pricing/ Simple pricing, paid in credits. One monthly plan with credits included. Need more performance? Contact us. ### Monthly plan Monthly: $99 per month. 165 CPU hours of cloud backtesting every month (9,900 credits). - The strategy tree, the strategy list and every validation report - Market data: 15 years of OHLC bars, 1 year of ticks, 3+ months of MBO - Our AI, or your own Claude Code or Codex - Your strategies stay private ### Need more performance? Faster backtests, more runs in parallel and more credits every month. Contact us for more info. Write to support@benchalgo.com. ### How credits work Credits pay for two things: AI usage on our server and backtests. - AI usage: When you chat with an OpenRouter model on our server, the model provider's cost plus a markup is taken from your credits. Your own Claude Code or Codex does not use credits for the AI. - Backtests: One credit per CPU minute. A run on 8 CPUs for one minute uses 8 credits. - What is free: Creating an account, reading your tree and reports, and checking a strategy with st-mcp do not use credits. - When credits run out: No new run starts while your balance is zero or below. Your tree and reports stay available. Credits refill with the next month, or contact us for more. All prices are in USD. Payments run in test mode while BenchAlgo is in preview. No real money moves. ### Pricing questions What happens when I run out of credits? No new run starts while your balance is zero or below. Everything you already have stays available. Credits refill with the next month, or contact us for a plan with more performance. Can I use my own AI to save credits? Yes. With your own Claude Code or Codex, the AI part is on you and not on your credits. You only spend credits on backtests. How many credits is one dollar? 100 credits. The plan on this page uses this rate. ## How backtest scoring works: 12 screens, one score | BenchAlgo URL: https://benchalgo.com/how-scoring-works/ Every strategy ends with a score from 0 to 10, a verdict and a 12-screen validation report. Here is what each part means. ### What the score means The score runs from 0 to 10. It reflects how well an idea holds up across all the checks, not just whether one backtest made money. A higher score means the idea survived more of the tests described below. It is not a forecast. A score of 9 does not mean the idea will make money, and a low score does not mean it never could. It tells you how much trust the evidence deserves. ### From discovery to production Bring your own AI to discover and test a trading strategy. Aim for a final score of 7 or higher with complete required checks and at least one trade entry in every eligible calendar week: that makes a strategy a candidate for production review. The score alone does not approve it for trading. The planned next step is deployment through this interface, after holdout validation, forward testing and your approval. Deployment is not enabled yet; the current workspace cannot place orders. ### The twelve validation screens Each node has a report with twelve screens. Every screen looks at the idea from a different angle. - 01 Scorecard: The summary: the score, the verdict, net profit, drawdown and the key checks in one place. - 02 Equity and drawdown: Net profit after costs over time, and the deepest drawdowns with how long each took to recover. - 03 Period consistency: Profit by month, week and year, so one lucky stretch cannot hide a weak idea. - 04 Trade distribution: How often the idea trades, how big the winners and losers are, whether the win rate beats the break-even for that win-loss ratio, and how trades end: at the target, at the stop or on time. - 05 Cost stress: The same idea with worse fees and slippage. Thin edges disappear here. - 06 Monte Carlo: History is resampled many times, in blocks of trading days, to show how bad luck could have looked. - 07 Parameter stability: Neighbouring settings are tested too. An idea that works at one exact setting only is fragile. - 08 Walk-forward: The idea is tuned on one slice of history and judged on the next one, again and again. - 09 Overfitting (PBO): The probability of backtest overfitting: how likely it is that the best setting from testing was just the luckiest one. - 10 Lookahead: Proof that every decision used only data that existed at that moment. - 11 Recent trades: The latest trades with entry and exit times, profit in dollars and the move in ticks from entry. - 12 Specification: The idea in plain words, its rules, how it was chosen and a record of the run. ### Passed, Weak, Failed and N/A Each scored screen ends in a badge, so you can scan a report in seconds. - Passed: The check met its bar. - Weak: The result is borderline. It is not a failure, but there is little margin. - Failed: The check did not meet its bar. - N/A: The check does not apply or did not run, for example when an earlier step already made the run invalid. ### Key checks and trade frequency cap the score Some checks are key checks. If one of them fails, the score is capped, however good the other screens look. An idea capped by a failed key check cannot become a Lead or Validated. A strategy with any eligible week without a trade entry, or without verified weekly frequency, can score at most 6.9. A high average cannot hide empty weeks. Trading less often also lowers the highest score a strategy can reach: one trade a day can reach about 8.4, a score of 9 or more takes about two a day, and a strategy that trades once a year stays near the bottom. The scorecard tells you when a cap was applied and why. Key checks are the main protection against ideas that look great but only because of luck or lookahead. ### The three verdicts The score and the checks together decide one of three verdicts. - Lead: The idea passed the key checks and the score is promising. It is a good base for the next idea. It is not proof. - Validated: The strongest verdict. The idea passed the key checks and held up on data it was not tuned on. It is still a backtest, not a promise. - Rejected: The idea failed a key check or scored too low. It stays on your tree so neither you nor the AI repeat it. ### What scoring does not do - It does not predict future results. All numbers come from simulated trades on past data. - It does not remove the need to look at the idea yourself. Read the specification and the recent trades. - It counts every attempt. Runs that failed or crashed still count as trials, so trying many ideas cannot hide behind one good result. ## Lookahead checks: proof a strategy used only the past | BenchAlgo URL: https://benchalgo.com/lookahead-checks/ Lookahead is the most common way a backtest lies: the strategy uses information it could not have had at that moment. BenchAlgo tests for it in several independent ways. A failed check blocks an idea from becoming a Lead or Validated. ### Cut the future and compare The harness runs your strategy on the full data. Then it runs it again with all data after a chosen moment removed. Every decision made before that moment must be identical in both runs. If one decision changes, the strategy was looking ahead. ### Change the future and compare Next the harness keeps the data but alters the part that comes after a decision, for example by scrambling or reshaping it. Earlier decisions must not change. A strategy that reacts to data it should not have seen gives itself away here. ### Fills happen at the next bar open A signal that fires on one bar can only be filled at the open of the next bar. It is never filled at the price that created the signal. The harness owns fills, so a strategy cannot choose a better one. ### Stop before target inside a bar When a single bar touches both the stop and the target, no one can know which came first. We assume the stop was hit first. This is the cautious choice, and the strategy cannot change it. ### Replay as if live The strategy is replayed bar by bar through the same kind of engine that would run it live. Its trades, fills, exits and profit must match the fast backtest exactly. Any difference means the backtest knew something a live engine would not. ### Random entries and random direction These checks ask whether the idea beats chance. The harness replaces your entry times with random ones, and separately your direction with a coin flip. If chance does as well as the idea, the edge is not real. ### Double the costs The idea runs again with fees and slippage doubled. An edge that disappears under higher costs is too thin to rely on. ### Walk-forward testing The idea is tuned on one slice of history and then judged on the next slice, which the tuning never saw. This repeats along the whole history, always moving forward in time. ### A sealed holdout The most recent three months of data are locked away. For now, nothing is tuned, scored or shown on them. They stay as a final exam that no idea has seen. ### Why invalid runs still count Every run counts as a trial, including runs that were invalid or crashed. The more ideas you try on the same data, the easier it is to stumble on a lucky one. The score takes the number of trials into account, and the count is kept per campaign and cannot be reset. Failed attempts cannot be quietly forgotten. ## BenchAlgo FAQ: data, costs, credits and your AI URL: https://benchalgo.com/faq/ Short answers about the data, privacy, costs, credits and the AI you use. ### What data does BenchAlgo use? CME Globex data for ES, NQ and GC, from 1-minute bars up to MBO. Both full-size and micro contracts work. The plan includes 15 years of OHLC bars, 1 year of tick data and more than 3 months of MBO data, with new MBO data added every month, and more is available as a package: contact support for pricing. The market data stays on our servers. ### What do I see in the results? Profit and loss in US dollars, and every trade as a move in ticks from its entry. You never see raw market prices. ### Are my strategies private? Yes. Your strategies, chats and reports are private to your account. If an operator ever opens your data to help you, that access is written to an audit log. ### Are costs and slippage included? Yes. Every trade pays a per-contract fee and slippage on entry and on exit. The harness sets these costs, so a strategy cannot lower them. The Cost stress screen and the double-costs check show what happens when costs get worse. ### Which AI writes the strategies? You choose. Use an OpenRouter model on our server, paid with credits. Or connect your own Claude Code or Codex through the st-mcp tools. ### Does the AI see my data or my results? No. The AI gets the public harness API, your strategy code and a short written feedback note: the verdict, a rounded score and which checks failed. It never sees market data, trades or curves. ### How do credits work? Credits pay for AI usage on our server and for backtests. One US dollar buys 100 credits. The pricing page lists the monthly plan. ### Can I trade live with BenchAlgo? Not yet. BenchAlgo is for research. It has no broker connection and does not place orders. The Deploy page in the app is switched off. ### What is the sealed holdout? The most recent three months of data are locked away. For now, nothing is tuned, scored or shown on them. They stay as a final exam that no idea has seen. ### Can I bring my own Claude Code or Codex? Yes. Connect it through the st-mcp tools. It gets three tools: check_strategy, run_backtest and get_result. The first one is free and the second one uses credits. The Settings page in the app shows how to connect. Your strategy code runs only in our sandbox. ### Does a high score mean a strategy will make money? No. A high score means the idea survived strict tests on past data. It does not predict future results. Markets change, and every backtest is a simulation. ### How is this different from an ordinary backtester? An ordinary backtester gives you one equity curve for one idea. BenchAlgo keeps every idea your AI tries as a node, tests each one the same strict way and shows the whole tree, failures included. You see what worked and what did not. ## How to check a backtest for lookahead bias URL: https://benchalgo.com/guides/lookahead-bias/ Run the strategy twice: once on the full data, and once with everything after a chosen moment removed or changed. Every decision made before that moment must be identical in both runs. If one decision moves, the strategy used data it could not have had. ### What lookahead bias looks like Lookahead bias means a decision in the backtest depends on something that did not exist yet. It rarely looks like cheating in the code. It is usually one innocent line, and it makes the equity curve look better than any live result could be. We learned this the hard way. In our own research, some of the best-looking intraday results turned out to be leaks of this kind, which is why these checks now run on every strategy. These are the usual places it hides: - A signal computed from a bar’s close and filled at that same close, or anywhere inside that same bar. - A higher-timeframe value, such as a 15-minute bar or the daily high, read before that bar has finished. - A session level, such as the day’s VWAP or volume profile, taken from the finished session instead of the session so far. - An indicator normalised with the mean or range of the whole dataset. - A front-month contract picked with volume from later in the same day. ### Four tests that find it 1. Cut the future. Pick several moments across the history. For each one, run the strategy again on the data up to that moment only, and compare the decision logs. Anything before the cut must match exactly. 2. Change the future. Keep the data up to a chosen moment and scramble or reshape what comes after it. Earlier decisions must not move. This catches code that treats missing data differently from real data, which can slip past the first test. 3. Fill after the signal. A signal from a completed bar can fill at the next bar’s open at the earliest. If your backtester fills at the signal price, every result is too good. 4. Replay bar by bar. Feed the strategy one completed bar at a time through an event loop, the way a live engine would, and compare trades, fills, exits and profit with the fast backtest. Any difference means the fast version knew something a live engine would not. ### What these tests do not catch They find leaks in the code and the data pipeline. They do not find selection bias: a strategy can be perfectly causal and still be the luckiest of a thousand variants. That takes the overfitting checks. Test more than one moment, too. A leak in a rarely used branch only shows up at the moments that reach that branch. ### How BenchAlgo handles it - Strategies only receive completed bars, one event at a time. Reading at or past the current moment raises an error that voids the run, even if the code catches it. - The forming session profile and the session VWAP are computed through the current moment only. - The harness owns fills and the choice of contract. A strategy cannot pick either one. - Every run is checked for leaks of later data. A failed check blocks an idea from becoming a Lead or Validated. Sources: [Pine Script docs: repainting and lookahead in request.security()](https://www.tradingview.com/pine-script-docs/concepts/repainting/) ## How to tell if a backtest is overfit URL: https://benchalgo.com/guides/backtest-overfitting/ Judge the strategy on data it was not tuned on, test the settings next to the chosen one, and count every variant you tried. An overfit backtest looks best at one exact setting on one stretch of history and falls apart everywhere else. ### Why good backtests are often overfit Try enough variants on the same history and one of them will look great by luck alone. The more variants you try, the better the best one looks, even when none of them has an edge. Cheap compute and AI-written code make this easier to do by accident. ### Walk-forward testing Tune the settings on one slice of history, then judge them on the next slice, which the tuning never saw. Move forward and repeat until the history runs out. Only the results on the unseen slices count. theta_i = argmax over theta of Pi(theta; B_= x}) / (1 + K), where x is the idea’s net, X_k the net of random twin k and K the number of twins. With 400 twins, if 12 of them do at least as well as the idea, p = (1 + 12) / (1 + 400), about 0.03. The one added above and below keeps p from ever reaching zero. Test the two parts apart as well: random times with your sides, and your times with random sides. An idea can have good timing and no directional skill, and that is useful to know. ### What it cannot tell you Passing is not proof. Test twenty ideas and one will pass at p = 0.05 by luck alone, which is why the number of trials matters. White’s Reality Check, linked below, was built for this problem of reusing the same data many times. ### How BenchAlgo handles it The harness draws many random twins for every run and turns the comparison into a p-value. It also replaces your entry times with random ones and, separately, your direction with a coin flip. These are among the checks that can reject an idea on their own. Sources: [White (2000): A Reality Check for Data Snooping, Econometrica](https://www.jstor.org/stable/2999444) ## Backtest validation checklist: twelve checks URL: https://benchalgo.com/guides/backtest-validation-checklist/ Before you trust a backtest, check twelve things: the overall summary, drawdown, consistency by period, the trade distribution, worse costs, resampled history, neighbouring settings, walk-forward, overfitting, lookahead, the recent trades and the written specification. ### 1. Scorecard The summary: the score, the verdict, net profit, drawdown and the key checks in one place. S = min(sum over k of w_k s_k, c): a weighted score, capped by c when a key check fails. ### 2. Equity and drawdown Net profit after costs over time, and the deepest drawdowns with how long each took to recover. DD_t = max over s <= t of E_s, minus E_t: the drop of equity E below its running peak. ### 3. Period consistency Profit by month, week and year, so one lucky stretch cannot hide a weak idea. p+ = #{m : pi_m > 0} / M: the share of months with a profit. ### 4. Trade distribution How often the idea trades, how big the winners and losers are, whether the win rate beats the break-even for that win-loss ratio, and how trades end: at the target, at the stop or on time. kappa = m4 / m2^2 - 3: the excess kurtosis of trade results (fat tails show up here). ### 5. Cost stress The same idea with worse fees and slippage. Thin edges disappear here. Pi(c) = sum over trades of (g_i - 2cv - f): gross result minus c ticks of slippage per side at tick value v, minus the fee. ### 6. Monte Carlo History is resampled many times, in blocks of trading days, to show how bad luck could have looked. p5 = Q_0.05(E_T^(b)): the 5th percentile of final net E_T over the resampled paths b. ### 7. Parameter stability Neighbouring settings are tested too. An idea that works at one exact setting only is fragile. r = |N+| / |N|: the share of neighbouring settings that stay profitable, also under stress. ### 8. Walk-forward The idea is tuned on one slice of history and judged on the next one, again and again. theta_i = argmax over theta of Pi(theta; B_