Lesson 6 of 7

Backtests and overfitting

Learning goal

Explain what a backtest is, why testing many tweaks can fool you, and how to build a rulebook that is robust.

Explanation

A backtest runs your rules on past data to see what would have happened. It is a useful test. It is not a promise about the future.

The danger is overfitting. If you keep changing settings until the past looks perfect, you are teaching your rules to match the quirks of that one period. Those quirks will not repeat. The rules look strong in the past and fail on new data.

Think of a student who memorizes last year's exam. The score on last year's paper is perfect. The score on this year's paper is not.

Two kinds of data exist for this reason:

Practice periods. You can see them and run as many tests as you like.
Unseen periods. Your fund is scored on data you never saw, kept in sealed windows on the server.

Many tweaks fool you because luck is easy to find. If each setting has a small chance of looking great by pure luck, trying many settings makes a lucky winner almost certain.

The run counter on the platform shows how many tests you have made. Runs are unlimited, and the scoring table gives no points for or against the count. Use the counter as feedback. A high count tells you that your "best" settings may be the luckiest, not the strongest.

Robustness means your rules work across many conditions, not just one. A robust rulebook has a simple logic you can explain, and it does not collapse when you nudge a setting.

Worked example

Suppose each rulebook has a 10% chance of looking great on practice data by luck alone, and the rulebooks are independent.

Try 1 rulebook: 10% chance of a lucky winner.
Try 20 rulebooks: chance that at least one looks great = 1 − 0.9^20 = 1 − 0.12 = 88%.

You will almost certainly find a "winner," even with no skill.

The neighbor test. Check the settings next to your favorite.

Lookback N	13	26	39
Rulebook A: CAGR	9%	10%	9%
Rulebook B: CAGR	2%	15%	1%

Rulebook B has the highest peak, but only at N = 26. A small change destroys it. That is a sign of a lucky fit. Rulebook A holds up. It is robust. Prefer A.

What this means in the challenge

In Part 2 you may test as often as you want on the practice decade, then you seal the fund. A sealed fund is final. Your Part 2 score comes from three hidden five-year windows, scored by the median. The median means one lucky or unlucky window cannot decide your result. The windows stay sealed until the final, when you see how your fund did in each.

The practice sandbox (1960 to 1969) opens at portal launch. Use it to learn how each block behaves before the round. In the round, use the practice decade to check that your fund makes sense, not to chase the highest number.

Practical rules: change one block at a time. Write down why you made each change. Keep your rulebook simple. Run the neighbor test on your final settings. Finalists must explain every rule, so a rulebook you cannot explain is a liability.

Self-check

Why does the challenge score you on periods you cannot see?
Show the answer
Answer: Because a fund can be fitted to the past. Unseen periods show whether the rules work on data the student never tuned them to.
Rulebook A has CAGRs of 9%, 10%, 9% for three neighboring lookbacks. Rulebook B has 2%, 15%, 1%. Which is more robust, and why?
Show the answer
Answer: Rulebook A. Its results barely change when the lookback moves. Rulebook B works only at one exact setting, which suggests a lucky fit.
Why does using the median across three windows help?
Show the answer
Answer: One very good or very bad window cannot decide your score alone. The median reflects the typical result across all three.