1
Overfitting
The system captures historical noise or researcher choices instead of repeatable behavior.
Lesson · Backtesting and Trading Systems
Overfitting, Sensitivity, Monte Carlo, and Stress Testing
Test nearby parameters, resample trade sequences, increase costs, remove favorable periods, and verify that the result is not dependent on one precise historical path.
Core concepts
What to understand
Keep each definition operational so it can be implemented and tested.
2
Sensitivity
Small parameter changes should not completely reverse the result without a market reason.
3
Stress and resampling
Test unfavorable costs, order sequence, missing winners, and regime changes.
Interactive lab
Robustness Testing Lab
Inspect a parameter surface, resample trade order, and stress execution costs. Compare peak performance with stable performance.
Robustness Testing Lab
Deterministic synthetic educational data · no live market connection
BACKTEST MODEL
Process
Try to break the system
Robustness testing searches for failure conditions, not prettier results.
Perturb
Change parameters, delays, prices, and costs within reasonable ranges.
Resample
Change trade sequence and selected historical blocks.
Decide
Keep, simplify, constrain, or reject the system based on stability.
Knowledge check
Check your understanding
Select one answer for each question. Feedback appears immediately.
1. A single narrow parameter peak is usually:
A sharp isolated optimum can result from fitting noise.
2. Monte Carlo trade reordering evaluates:
The same trade distribution can produce different paths and drawdowns.
3. Why increase costs in a stress test?
Live spreads and slippage can exceed average historical assumptions.
4. A selected lookback returns +8.0%; the average of its neighbouring lookbacks is +1.5%. The share of the result the neighbourhood keeps is:
1.5 ÷ 8.0 = 0.19, so 19% of the result survives a small change in the parameter — the mixed sample's narrow peak. The trending sample's plateau keeps 72%.
Practical rules
- Prefer simpler rules and broad parameter stability.
- Stress costs, delays, and missing favorable events.
- Record all experiments, not only the selected result.
Common mistakes
- Choosing the single best parameter cell.
- Running many trials without accounting for selection.
- Calling one historical sequence the expected future path.
References
Official reading
The labs are simplified simulations. These sources cover historical performance, data, costs, or execution limitations.
FINRA · Smart Beta—What You Need to Know
Explains that historical backtesting is hypothetical and does not predict future performance.
FINRA Rule 2210 · Communications with the Public
Includes restrictions on performance projections and misleading performance claims.