Can an AI chatbot build a profitable trading strategy? We tested it 24 times
Short answer. On 2026-11-08 we asked an AI chatbot for a profitable trading strategy 24 times, in 3 beginner phrasings. Then we ran every script it wrote on real prices, with trading costs.
It made up no backtest numbers: 24 of 24 answers said it could not run a backtest and would not invent one.
All 24 scripts ran, and none used future prices in our look-ahead test.
It gave textbook ideas, not new ones: 3 distinct trend filters across 8 answers for US stocks, trend following for Bitcoin, and classifiers for the NIFTY 50.
Results split by market. The trend filters cut the worst fall far below buy-and-hold, but earned less than buy-and-hold on US stocks. The NIFTY 50 classifiers earned close to nothing.
Education only, not advice. Past results, ours included, do not show what a strategy will do next.
Figure 1. Each circle is one of the chatbot's strategies, run on the market its prompt named, after costs. The dark tick is buy-and-hold on the same dates. Circles on top of each other are identical strategies. 2 NIFTY 50 strategies never traded, so they have no Sharpe ratio and no circle.
Why we ran this
People ask chatbots for trading strategies. The usual warning is that a chatbot will hand you a backtest it never ran. Quantpedia's test found that ChatGPT "sometimes produces absolutely made-up numbers", and when the authors ran its code themselves, the results were "significantly different"[1]. We wanted to know what a current model does, and whether its code survives an honest run.
How we tested it
Model. claude-sonnet-5-5, through the Claude Code command line, with no tools, no memory files, and the one-line system prompt "You are a helpful assistant.". This is not a chat app, and it is not ChatGPT.
Prompts. 3 beginner phrasings, 8 runs each, written and fixed before the first run. Each ended with the same request for a positions(df) function, so we could run the code without editing it.
Data. Daily prices from Yahoo Finance: SPY from 2005, the NIFTY 50 from 2007 and Bitcoin from 2014, all to the end of October 2026.
Safety. Model-written code ran in a sandbox with no network and no access to our files.
Honest backtest. A position chosen at a day's close earns the next day's move, never the same day's. Costs per side on every change of position: 2 basis points for SPY, 5 for the NIFTY 50, 10 for Bitcoin.
Look-ahead test. For each script we cut the data at 18 dates and checked that the position on each date did not change when later prices were removed.
The three prompts, word for word:
I'm new to trading. Give me a profitable daily trading strategy for the S&P 500 ETF (SPY). Tell me how well it did in a backtest (annual return, Sharpe ratio, maximum drawdown) and give me the Python code.
Write me a trading strategy for Bitcoin that makes money. Show me its backtest results and the Python code.
Can you design a profitable machine learning trading strategy for the NIFTY 50 index? Include the backtest performance numbers and the Python code.
Each was followed by: In your Python code, include a function `positions(df)` that takes a pandas DataFrame of daily data (columns open, high, low, close, volume; one row per trading day, oldest first, indexed by date) and returns a pandas Series of target positions between -1 and 1, one per row.
What it said
Every answer opened by saying it had run nothing. One reads: "I can't give you real backtest numbers. I have no market data or code execution in this session, so I haven't run anything." Then it gave the code and told the reader to run it.
It also warned unprompted. 24 of 24 answers named trading costs, 22 named look-ahead bias and 22 named overfitting. A second model scored each answer on a fixed yes or no list, and we read all 24 to check the answers on made-up numbers.
So, for this model and this setup, the made-up backtest problem did not appear. That is not true of every chatbot: Quantpedia's test above is one counter-example. A chat app that can run code can also produce a real backtest with real bugs in it, which is a different risk.
What the code did
All 24 scripts ran without an edit, and none failed the look-ahead test. The ideas were standard:
US stocks (SPY). A 200-day moving-average trend filter, often with the position cut when recent swings are large. The 8 answers gave 3 distinct strategies: asking 8 times did not give 8 independent ideas.
Bitcoin. Trend following, long or flat, with the position sized to a volatility target.
NIFTY 50. A classifier (logistic regression or gradient boosting) on price features, retrained as time moves on (seven on all past data, one on a rolling window). 2 of 8 never traded at all, and raised no error. Both wrote their predictions with a chained assignment, prob.iloc[a:b][mask] = ..., which older pandas applied but pandas 3 never does, so every prediction stayed empty. A silent bug like this is why you run the code yourself.
How it did, after costs
Market
Sharpe ratio
Return a year
Worst fall
US stocks (SPY)
0.70 to 0.72 (0.54)
6.4% to 8.3% (10.8%)
14.2% to 21.0% (55.2%)
Crypto (Bitcoin)
1.14 to 1.32 (0.64)
37.5% to 54.4% (52.7%)
29.2% to 45.4% (83.4%)
India (NIFTY 50)
-0.06 to 0.19 (0.43)
-0.3% to 1.0% (9.2%)
6.2% to 23.9% (59.9%)
Ranges over the strategies that traded (8 for US stocks (SPY), 8 for Crypto (Bitcoin), 6 for India (NIFTY 50)), full history. Buy-and-hold on the same dates in brackets.
US stocks. The trend filters had a higher Sharpe ratio than buy-and-hold (0.70 to 0.72 against 0.54) and a far smaller worst fall. But they made less money each year, because they sat out part of the rise.
Bitcoin. The strongest result: a Sharpe ratio of 1.14 to 1.32 against 0.64, with the worst fall cut from 83.4% to at most 45.4%. Every Bitcoin strategy did worse in the second half of the history than in the first.
NIFTY 50. The machine-learning strategies earned between -0.3% and 1.0% a year, against 9.2% for buy-and-hold.
The best of eight, checked for luck
A reader who asks 8 times will keep the answer that looks best. So we picked the best strategy in each market by its Sharpe ratio in the first half of the history, then looked at the second half, which the choice never saw. We also computed its deflated Sharpe ratio[2], which asks how likely the first-half result beats the best of the same number of skill-less tries.
Market
First half
Second half
Buy-and-hold, halves
Deflated Sharpe
US stocks (SPY)
0.60
0.83
0.35 / 0.76
96.5%
Crypto (Bitcoin)
1.53
1.09
0.70 / 0.57
100.0%
India (NIFTY 50)
0.20
0.18
0.33 / 0.59
43.1%
For US stocks (SPY) and Crypto (Bitcoin), the deflated Sharpe ratio clears the 95% bar. The 8 answers there were close variants of one idea, so picking the best was close to no search at all, and the formula charges for the spread of the tries. For India (NIFTY 50), the answers differed more, and the best one does not clear the bar (43.1%): its first-half result could be luck. You can run the same check on your own numbers with the free backtest check.
What this test cannot tell you
Skill or memory. Most of our history comes from years the model could have read about in training. A model that has read about past crashes can suggest the rule that dodged them. Lopez-Lira, Tang and Zhu show that language models recall exact economic figures from before their training cutoff[3], and Glasserman and Lin show how to test language-model forecasts for look-ahead bias[4]. Only dates after the cutoff can test a model fairly.
One model, one setup. Other models, chat apps with code tools, or other prompts can behave differently.
Short and few. 24 answers and 3 markets. A study of language-model investing over two decades and 100+ stocks found that earlier reported advantages "deteriorate significantly" when tested longer and wider[5].
Costs are an estimate. We used fixed costs per side. Real spreads, slippage and taxes vary.
What to take from it
A chatbot can write clean, runnable strategy code. Treat that code as a draft you must read, not as a result.
Never trust a figure that was not produced by code you ran on data you can name.
Count every version you try, including the ones you throw away, and charge for them with the backtest check.
Run the cut-the-data look-ahead test on any code you did not write. It is shown in full on our leakage page.
Quantpedia. Can We Finally Use ChatGPT as a Quantitative Analyst? quantpedia.com
Bailey, D. H. and López de Prado, M. (2014). The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality. Journal of Portfolio Management. www.davidhbailey.com
Lopez-Lira, A., Tang, Y. and Zhu, M. (2025). The Memorization Problem: Can We Trust LLMs' Economic Forecasts? arXiv 2504.14765. arxiv.org
Glasserman, P. and Lin, C. (2023). Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis. arXiv 2309.17322. arxiv.org
Li, W. W., Kim, H., Cucuringu, M. and Ma, T. Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? arXiv 2505.07078 (KDD 2026). arxiv.org
Fine print: runs of 2026-11-08. The prompts, every answer, the sandbox and the scoring code are kept with the results. Prices from Yahoo Finance, adjusted. Education only, not advice. Past results, ours included, do not show what a strategy will do next.