How to backtest a trading strategy written by AI
Short answer
Let the AI write the strategy as code, but keep market data and results away from it, run the code in a sandbox, let the backtester decide fills and costs, and count every attempt as a trial. The model then cannot fit to the data it is tested on.
Why AI-written strategies need stricter tests
A model can write a hundred variants in the time it takes you to write one. Each one is another trial on the same history, so the best of them looks better and better without being any more real. A model that sees the results will also fit to them, even when nobody asks it to.
The code itself deserves care too. It is generated text you did not read line by line, and it may import things or reach for data it should not have.
Four rules
- Keep the data away from the model. Give it the strategy API and short feedback, not prices, trades or equity curves.
- Sandbox the code. Run it with no files, no network and a time limit, and check it before it runs.
- Take fills and costs out of its hands. The strategy only states what it wants to do. The backtester decides what happened.
- Count every attempt. Failed and crashed runs count as trials. Otherwise a long session with the AI hides behind one good result.
Your AI or ours
The tests are the same either way. The AI never sees market data, trades or curves, only short feedback on what passed.
| Aspect | OpenRouter | Claude Code or Codex |
|---|---|---|
| Runs on | Our server | Your computer |
| Pays for the AI | Your credits | Your own subscription |
| Connect | Pick a model for each chat | Run st-mcp connect once |
| Backtests | Credits | Credits |
Backtests use credits either way. The Monthly plan includes 9,900 credits a month.
Connect Claude Code or Codex
- Get the command. The web app gives you a one-line command for your account.
- Run it in a terminal. Paste it on your computer and keep the window open. It finds Claude Code and Codex and tells the app which ones it found.
- Chat in the app. Each turn runs your CLI once, headless, with its tools switched off, in a new empty folder that is removed afterwards. It runs on your own subscription.
st-mcp connect
09:41:02 Connected. The chat in the app now runs on Claude Code and Codex on this computer.You can also work from your own project folder. The st-mcp tools give your AI three tools: check_strategy, which is free, run_backtest, which uses credits, and get_result.
What the AI gets back
A short feedback note: the verdict, a rounded score and which checks failed. It never sees market data, trades or curves, and strategy code runs only in our sandbox.
Every attempt becomes a node on your strategy tree, with its score, P&L and verdict. Rejected ideas stay on the tree, so neither you nor the AI tries them again.