Take eight closing prices and one rule: hold after any day the price rose. Book each day's gain on the same day the signal appears, and the rule makes 20 points. Book it on the next day, which is the only day you could really have traded, and the same rule loses 8. Nothing changed but one day of timing. Most of backtesting is like that: a short list of questions about when you knew what, what you paid, and how many rules you tried first.
To test a trading rule on the past, you run a backtest: a fixed, written rule run over past prices to see what it would have done. If the idea is new to you, read what a backtest is, in plain words first. It walks one rule through ten numbers by hand.
This page is the method. It has eight steps. Each step names the mistake it prevents and a check you can run yourself. The steps work the same for US or European shares, a stock index, a currency pair or a crypto coin, and in any tool you like: a spreadsheet, Python, or code an AI assistant wrote for you. For the wider set of pages on the subject, see the backtesting hub.
Every number below is computed on this page, and we checked each one in Python. This page is education about method. It is not advice to buy or sell anything, and a good backtest does not tell you the future.
Step 1. Write the rule down before you look
A rule is a sentence precise enough that two people would place the same trades from it. "Buy when the trend is up" is not a rule. "At each day's close, if the close is above its 50-day average, hold the index for the next day; otherwise hold cash" is a rule. For more on this, read what a trading strategy is.
Write it down, with the date, before you run anything. Write down what result would make you drop the idea, too.
What goes wrong. You look at a chart, see a pattern, and write a rule that fits it. The test then "confirms" what you already saw in the same data. That is not a test.
Check. Could you show someone a dated note that names the rule and your pass mark, written before the first run?
Step 2. Choose the data, and know what is missing from it
Name the market, the dates and the bar size, such as daily closes of an index from January 2010 to December 2024. The window matters. A rule tested only through one long rise has never seen a fall.
Three gaps catch most people:
- Things that died are missing. A list of today's index members leaves out every firm that was dropped, bought or went bust along the way. Testing today's survivors over the past makes the past look kinder than it was. That is survivorship bias. The fix is a list of what was in the market on each past date: see survivorship bias and the universe that never existed.
- Prices jump when nothing happened. A stock split gives each holder more shares at a lower price, and the holder is no richer. Apple's investor page says its stock "split on a 4-for-1 basis on August 28, 2020". In raw prices that looks like a fall of about 75% in a day. A rule run on unadjusted prices "sees" a crash that never cost anyone anything. Dividends cause a smaller version of the same jump. Read corporate actions and the adjustment chain.
- Time stamps lie. Know what time each price was taken and in which time zone. In a market that trades through the night, a "daily close" is a time someone chose. Choose it, write it down, and use the same one everywhere.
Check. Pick two dates at random. Is every instrument that existed then in your data? Does any price move more than 30% in a day? If so, find out why before you go on.
Step 3. Decide when the signal is known and when the trade fills
This is the step that turns most broken backtests into beautiful ones. A signal computed from a day's close is known only after the close. So the earliest trade it can drive fills on the next bar.
A worked example. Eight daily closes:
200, 204, 201, 206, 203, 209, 207, 212.
The day-to-day changes are +4, −3, +5, −3, +6, −2, +5. The rule: after a day the price rose, hold for one day.
- Wrong (same bar). Book each rising day's own gain: 4 + 5 + 6 + 5 = 20 points. The rule can never lose, because it only "holds" on days that went up. It used each day's close to decide to hold that same day.
- Right (next bar). The signal at the close of a rising day earns the next day's change. The rises on days 2, 4 and 6 are followed by −3, −3 and −2. That is −8 points. The last rise, on day 8, has no next day in the data, so it earns nothing.
Holding all the way, from 200 to 212, gained 12. So the same rule looks brilliant or useless depending on one day of timing. The error has a name, look-ahead bias: using information that did not exist yet when the decision was made.
Check. Shift every signal one bar later and rerun. If a good result falls apart, it was probably look-ahead. For a deeper treatment, see point-in-time data and look-ahead bias.
Step 4. Charge every cost
Every trade pays something:
- The spread: the gap between the best price to sell (the bid) and the best price to buy (the ask). With a bid of 100.00 and an ask of 100.04, buying at the ask and selling at the bid loses 0.04 on a round trip, about 0.04% of the price, before any move at all. See the bid-ask spread.
- Fees and commissions, which the broker or exchange charges. These are part of transaction costs.
- Slippage: the gap between the price your test assumed and the price you would really have got, which grows with trade size and in fast markets.
A worked example. A rule makes 50 round trips a year. Before costs, each trip earns 0.15% of the position on average. Each trip costs 0.10% in spread, fees and slippage. Adding up without compounding:
- gross: 50 × 0.15% = 7.5% of the position a year;
- costs: 50 × 0.10% = 5.0%;
- net: 2.5%.
Now suppose the real cost is 0.15% a trip. Net becomes zero. At 0.20% it is −2.5% a year. The break-even cost is 0.15%, the same as the gross edge per trip. For any rule that trades often, the cost estimate decides the answer.
Check. Rerun with costs doubled. If the result goes from good to bad, the rule's edge is smaller than your uncertainty about costs.
Step 5. Keep data the rule has never seen
Split the window in two before you start. Design and tune on the first part (in-sample). Then run the finished rule once on the second part (out-of-sample), and accept what it says.
A stronger form is walk-forward analysis: tune on a window, test on the next stretch, slide forward, and repeat. With daily data from 2010 to 2024, one layout is:
| Tune on | Then test on |
|---|---|
| 2010 to 2014 | 2015 |
| 2011 to 2015 | 2016 |
| 2012 to 2016 | 2017 |
| and so on | up to 2024 |
Join the test years end to end. That joined record is the only part of the result made of decisions taken without hindsight.
What goes wrong. Peeking. You look at the out-of-sample result, change one setting, and look again. After a few rounds the "unseen" data has been seen, and it is now in-sample too.
Check. How many times did you run the rule on the test window? The honest answer is once.
Step 6. Count every version you try
Each time you change a setting and rerun, you have tried a new rule. Keep a count of every version, including the ones you threw away. The reason is that the best of many worthless rules still looks good.
A common score for a backtest is the Sharpe ratio: average return divided by how much the return moves around, scaled to a year. A rule with no edge has a true Sharpe ratio of zero, but its measured one will land above or below zero by luck.
A worked example. Try 20 rules that all have no edge, each on 5 years of daily data, and keep the best. Bailey and López de Prado give an approximation for the expected best of N independent tries in their paper on the deflated Sharpe ratio. For 20 tries it is about 1.90 times the spread of the luck. With independent, normally distributed daily returns, that spread is about 1 divided by the square root of the number of years, so 0.447 for 5 years. The expected best measured Sharpe ratio is then about 1.90 × 0.447 ≈ 0.85, from no skill at all.
We checked this by simulation: 2,000 runs of 20 zero-edge rules over 5 years. The best rule's Sharpe ratio averaged 0.83, and in 23% of runs it was above 1. More tries push it higher. The same approximation gives about 1.02 for 50 tries and 1.13 for 100.
This is called backtest overfitting. Bailey, Borwein, López de Prado and Zhu put it plainly in their paper on the subject: "The higher the number of configurations tried, the greater is the probability that the backtest is overfit." Academic research has the same problem. Harvey, Liu and Zhu argue that a newly found return factor should clear a t-ratio above 3.0, not the usual 2.0, because so many have been tried. A t-ratio measures how many standard errors a result sits from zero.
Check. Write the number of versions you tried next to the result. Then put both into our free deflated Sharpe ratio calculator, which asks how much of the result the search alone explains. The full maths is in backtest overfitting.
Step 7. Read the result with its error bars
A single number hides how unsure it is. Read four things together.
The Sharpe ratio and its standard error. The standard error measures how far an estimate could be from the truth by chance. Under the same assumptions as Step 6, the deflated Sharpe ratio paper gives a formula for it. With normal returns (no skew, normal tails) and a ratio between zero and one, it works out to about 1 divided by the square root of the number of years. Take a measured Sharpe ratio of 0.8 over 3 years of daily data. The standard error is about 0.58. A rough 95% range is 0.8 ± 1.96 × 0.58, which is from −0.33 to 1.93. That range includes zero, so 3 years cannot tell this rule apart from no edge. Fat tails and lopsided returns widen it further. Our Sharpe ratio calculator is free if you want to try your own numbers.
The number of trades. Say 18 of 30 trades won, a 60% win rate. Its standard error is the square root of 0.6 × 0.4 ÷ 30, about 0.089. A rough 95% range runs from 42% to 78%. Thirty trades say very little.
The worst fall. The maximum drawdown is the largest drop from a peak to a later low. Expect it to be deep even for a rule with a real edge. We simulated a rule with a true Sharpe ratio of 0.5 (5% a year average, 10% a year of movement, independent normal daily log returns) over 10 years. In 2,000 runs, the median worst drawdown was about 20%. About one run in ten fell 30% or more. In more than four runs in five it passed 15%. Ask whether you could hold through that.
A plain benchmark. Compare against simply holding the market over the same dates, with the same costs.
Check. Do you know the standard error, the trade count, the worst drawdown and the benchmark? If any one is missing, the result is not yet readable.
Step 8. Paper trade before money
Paper trading means running the rule live, on prices as they arrive, with no money at stake. It is the first test the past cannot have shaped, because the data did not exist when you wrote the rule.
Write down each signal at the time it appears, the price you would have got, and the price you really could have got. Then compare the live record with the backtest over the same weeks. Big gaps point back to Steps 3 and 4.
Check. Run it long enough to see a fair number of trades. Until then, the paper record is only a test of your plumbing.
Three questions people ask
Is backtesting accurate? It is accurate about one thing only: what a rule would have done on that data, under the costs and timing you assumed. Each step above narrows the gap between that and what could really have been traded. No step closes the gap to the future.
Why does backtesting not work for so many people? In our view, the usual cause is not the method. It is skipping Step 3, Step 4 or Step 6: a signal traded on the same bar, costs left out, or the best of many versions reported as if it were the only one.
Can an AI assistant write the backtest for me? It can write code fast. It can also put a same-bar fill into that code without telling you. Check its code against Step 3 the same way you would check your own. Our member page an AI reviewer hunts look-ahead runs one such review.
The checklist, on one screen
The eight steps, as nine checks. Step 2 gives two of them.
- The rule and the pass mark are written and dated before the first run.
- The data names its market, dates, bar size, time zone and close time.
- The universe includes what was delisted; prices are adjusted for splits and dividends.
- Every signal trades on the bar after it is known.
- Spread, fees and slippage are charged; the result survives doubled costs.
- A test window was set aside and run once, or walk-forward was used.
- The number of versions tried is written next to the result.
- The result shows its standard error, trade count, worst drawdown and a plain benchmark.
- The rule is paper traded before any money is used.
Where Thuztra Learn fits in backtesting, and where it does not
Thuztra Learn is our product. It teaches the maths behind these steps in short lessons. The Level 0 module "What quants actually do" includes a backtest done by hand. The module "Statistical inference for research" covers backtest overfitting, testing many ideas at once and cross-validation for time series. The module "Market data and research infrastructure" covers point-in-time data, survivorship and corporate actions.
Thuztra Learn is not a backtesting platform. It does not connect to a broker, and it does not give trading signals. Membership is a paid subscription: see what Thuztra Learn is and what it costs.
Fine print: Apple is named only as the source of one stock split date, and has no link to Thuztra. The worked examples are for teaching method. Past results, real or simulated, do not tell you the future. Education only, not advice.
In short
A backtest is only as honest as its worst step. Write the rule first, trade one bar after the signal, charge real costs, keep some data unseen, count every version you tried, and read the result with its error bars. A result that survives all eight steps is still a test of the past, not a promise about the future.
Terms used here
- Backtest
- Look-ahead bias
- Survivorship bias
- Transaction costs
- Slippage
- Bid-ask spread
- Walk-forward analysis
- Maximum drawdown
- Deflated Sharpe ratio
Related
What a backtest is, in plain words
Ten numbers, one written rule, and the whole test done by hand. Then the harder half: why a past that was kind to a rule proves very little.
What a trading strategy is
A rule written plainly enough that two people would do the same thing on the same day. One toy rule walked down six numbers, and the gap between a rule and a hunch.
Backtest overfitting
How many configurations a backtest of a given length can support, how far the Sharpe threshold moves once we admit the search, and how to test whether the selection itself generalises.
Point-in-time data and look-ahead bias
A vendor's current fundamentals table is not what was knowable then. The three dates every datum carries, the join that respects them, and why no lag rule fixes a restatement.
Survivorship bias and the universe that never existed
Today's index members run backwards through history are not a universe. The three separate deletions, the arithmetic of the overstatement, and why deflating a Sharpe does nothing.
Corporate actions and the adjustment chain
A dividend puts a hole in a raw price series that no trader experienced. The factor algebra, why price and volume must move together, and the invariant that catches a double adjustment.
Cross-validation when the rows are not exchangeable
Why ordinary k-fold leaks on financial data, what purging and embargoing remove, and what walk-forward and combinatorial schemes buy at what cost.
The Sharpe ratio
Excess return divided by volatility, the square-root-of-time scaling, the standard error of the estimate, and the four ways the number lies about a real strategy.
More on these topics
- Knowing when to stop adding parameters
- What a single decision tree actually fits
- Quantitative finance projects you can build, check and explain
- An AI reviewer hunts look-ahead
- Backtesting a risk model, and the power problem
- Backtesting as the regulator requires it
Topics
- backtesting
- evidence
- getting started
- overfitting
- costs