1
In-sample
Develop rules and estimate parameters.
Lesson · Backtesting and Trading Systems
In-Sample, Validation, and Out-of-Sample Data
Separate rule development, parameter selection, and final evaluation so the last dataset remains unseen until the system is fixed.
Core concepts
What to understand
Keep each definition operational so it can be implemented and tested.
2
Validation
Compare alternatives and make limited design decisions.
3
Out-of-sample
Evaluate the frozen system once on data not used for selection.
Interactive lab
Data Split and Parameter Selection Lab
Optimize a parameter in training, inspect validation, then reveal the untouched out-of-sample result.
Data Split and Parameter Selection Lab
Deterministic synthetic educational data · no live market connection
BACKTEST MODEL
Process
Protect the final test
Repeatedly checking out-of-sample data turns it into training data.
Develop
Use in-sample data for hypotheses and parameter candidates.
Select
Use validation data for controlled comparison.
Freeze and test
Lock the system before revealing out-of-sample performance.
Knowledge check
Check your understanding
Select one answer for each question. Feedback appears immediately.
1. Which dataset should select the final parameter?
The final test should remain unavailable until the system is frozen.
2. Checking out-of-sample performance after every change causes:
The supposedly unseen data begins influencing rule selection.
3. A large in-sample/out-of-sample gap suggests:
The selected rule may be fitting noise or conditions unique to development data.
4. 260 bars are split 50 / 25 / 25. The in-sample, validation and out-of-sample segments hold:
260 × 50% = 130; 260 × 25% = 65; the remaining 65 are out of sample — and the split is set before any candidate is scored, in chronological order.
Practical rules
- Set split dates before optimization.
- Keep the final test inaccessible during development.
- Report every split, including weak results.
Common mistakes
- Choosing the split that gives the best chart.
- Retuning after each final-test failure.
- Using random splits that break time order for time-series trading.
References
Official reading
The labs are simplified simulations. These sources cover historical performance, data, costs, or execution limitations.
FINRA · Smart Beta—What You Need to Know
Explains that historical backtesting is hypothetical and does not predict future performance.
FINRA Rule 2210 · Communications with the Public
Includes restrictions on performance projections and misleading performance claims.