Library
AI can help with quant work: it writes code fast, reads text at scale, and fits patterns no person could. It also makes the oldest mistakes in quant finance faster and harder to see. This guide lists ten of them, what published research says about each, and how to check your own work. Where we make a claim of our own, we ran the test and show the code.
A chatbot without data and a tool to run code cannot backtest anything. In our own test of 24 requests, the model refused to invent figures every time, but another test caught a chatbot making numbers up[1]. Even working code is a draft until you run it on data you can name. Our 24-run test.
Accuracy that high on daily prices almost always means a leak: an input that sees the future, a shuffled split, or a trade credited with the move that made its signal. In our runs one centred moving average lifted accuracy from 52.7% to 82.3%. Five leaks, measured on four markets.
The best of many random tries always looks good. Bailey, Borwein, López de Prado and Zhu show that with 5 years of data, trying more than about 45 independent versions makes an annualised Sharpe ratio of 1 likely from luck alone[8]. The deflated Sharpe ratio charges for every try[7]. Harvey, Liu and Zhu argue a new finding needs a t-statistic above 3.0, not 2.0, because so many have been tested[9]. A flat "halve the Sharpe ratio" rule is not enough either: Harvey and Liu show the right cut depends on the Sharpe ratio and the number of tests[10]. Check your own count.
Predicting tomorrow's price as today's price scores an R² near 1 on price levels, in every market we tried, and is useless for trading. Judge a model on returns or direction, after costs, against that do-nothing forecast. The price-level illusion.
Where machine learning does help, the edges are small. Gu, Kelly and Xiu found trees and neural networks beat older regression models on US stocks, with monthly stock-level R² of 0.33% to 0.40%, and large economic gains from those small numbers[2]. Claims that bigger models simply find more have been challenged: Nagel shows that one such model's forecasts reduce to a volatility-timed momentum strategy[13].
Lopez-Lira and Tang found that GPT-4 scores of news headlines predicted stock returns over the next one to two days. Their own paper also reports that a strategy trading on them is unprofitable at 20 basis points of round-trip cost, and that its Sharpe ratio fell from 6.54 in the last quarter of 2021 to 1.22 over January to May 2024, which they read as a sign of fading predictability as these models spread[3]. Novy-Marx and Velikov find that trading costs reduce the profit of every anomaly strategy they test, and that few high-turnover ones stay significant after costs[14].
A language model may simply remember last year. Lopez-Lira, Tang and Zhu found models recall exact economic and financial figures from before their training cutoff, and none after it, and that telling them to ignore what they know does not stop it[4]. Glasserman and Lin, and Sarkar and Vafa, show ways to test for this look-ahead bias[5][6]. Test only on dates after the model's cutoff. Models also extrapolate recent returns the way people do[11].
Markets change. A study of language-model investing strategies over two decades and more than 100 stocks found earlier reported advantages "deteriorate significantly"; the strategies were too cautious in rising markets and too aggressive in falling ones[12]. In our chatbot test, every Bitcoin strategy did worse in the second half of the history than in the first.
An agent acts on its own outputs, so its mistakes compound with no one watching. Nof1, the team that runs Alpha Arena, a public contest in which language models trade, writes that "the current generation of LLMs are shockingly bad at navigating financial markets"[16]. Of its research agents, Man Group writes: "we don't expect this to act as a replacement for human researchers"[15]. If you experiment with one, give it hard limits it cannot change: maximum position, maximum daily loss, a cap on trades per hour, a kill switch, and paper trading first.
Regulators have acted against firms that overstated their AI. In 2024 the US SEC charged two investment advisers over false claims about using AI[17]. ESMA has told EU firms that use AI to stay accountable for it[18], and SEBI has set rules for algorithmic trading by retail investors in India[19]. Ask any product for its method, its live record and its costs.
Thuztra Learn teaches the maths of quant finance from zero, in short lessons, for readers who are unsure of it. Three questions in the app build your quant roadmap, free. Membership is a paid subscription. See what Thuztra Learn is and what it costs, or see the whole Core quant curriculum.
Get the app and answer the three questions:
Download on the Microsoft Store Download for Android