1
Completeness
Verify expected instruments, timestamps, fields, and sessions.
Lesson · Backtesting and Trading Systems
Historical Data and Data Quality
Detect missing bars, duplicates, stale values, timestamp errors, and price outliers before they create false signals or distorted returns.
Core concepts
What to understand
Keep each definition operational so it can be implemented and tested.
2
Consistency
Check duplicates, ordering, units, timezone, and price relationships.
3
Plausibility
Flag stale bars, extreme jumps, negative values, and impossible OHLC records.
Interactive lab
Historical Data Quality Lab
Inject and clean common data defects. Observe how one bad record can change signals and performance.
Historical Data Quality Lab
Deterministic synthetic educational data · no live market connection
BACKTEST MODEL
Process
Validate before calculating
Cleaning rules must be documented and reproducible.
Profile
Measure missing, duplicate, stale, and extreme records.
Repair or exclude
Apply a documented treatment without inventing unknown prices.
Recalculate
Confirm that indicators and signals use the validated dataset.
Knowledge check
Check your understanding
Select one answer for each question. Feedback appears immediately.
1. A single extreme bad price is most likely to distort:
One bad price can affect every calculation that uses that bar or its return.
2. What should happen to an unexplained missing bar?
Missing data requires an explicit, instrument-appropriate rule.
3. Why verify timestamps?
Incorrect ordering or timezone can expose future data or create wrong bars.
4. A rule counts closes above the highest close of the previous 12 bars. One bar in the file is missing. How many bars can the rule not score because of it?
Each of the twelve bars after a hole has the hole inside its lookback, so twelve bars cannot be scored on the raw series — the lab prints this count under its chart.
Practical rules
- Run automated validation before every backtest.
- Preserve raw data and record every transformation.
- Treat suspicious records as defects until verified.
Common mistakes
- Correcting data manually without an audit trail.
- Assuming all vendor files use the same timezone.
- Ignoring delisted or temporarily halted instruments.
References
Official reading
The labs are simplified simulations. These sources cover historical performance, data, costs, or execution limitations.
CME Group · Historical Market Data
Describes historical market data used for market analysis and strategy validation.
FINRA · Smart Beta—What You Need to Know
Explains that historical backtesting is hypothetical and does not predict future performance.