1
Event time
The time the market event occurred, distinct from the time the system received it.
Lesson · Algorithmic and Automated Trading
Market Data Ingestion and Normalization
Receive historical and streaming data, validate timestamps and symbols, remove duplicates, handle late events, and expose one normalized market-data model.
Core concepts
What to understand
Keep the boundaries, state, and authority explicit.
2
Normalization
Convert provider-specific messages into one internal schema with consistent units and identifiers.
3
Data health
Measure gaps, duplicates, lateness, sequence errors, and feed freshness continuously.
Interactive lab
Streaming Data Quality Lab
Inject duplicates, delays, missing events, and out-of-order messages. Compare raw and normalized strategy input.
Streaming Data Quality Lab
Deterministic synthetic educational data · no broker or live-market connection
SYSTEM MODEL
Process
Validate before publishing
Downstream strategies should not reimplement provider-specific cleanup.
Receive
Authenticate and capture the original provider message.
Validate
Check schema, sequence, timestamp, price, volume, and freshness.
Publish
Emit a normalized event with provenance and quality status.
Knowledge check
Check your understanding
Select one answer for each question. Feedback appears immediately, and a wrong choice is answered on its own terms before the rule is restated.
1. Why preserve event time and receive time?
The difference reveals transport delay and supports correct event ordering.
2. A duplicate trade event can cause:
Repeated input may update indicators and strategy state more than once.
3. Normalization should occur:
A common model keeps downstream behavior consistent and testable.
4. An event is stamped 10:15:02.000 by the provider and received at 10:15:09.500; the maximum feed age is 5 s. How late is it, and what does the normalizer do?
Lateness = receive − event = 7.5 s. The record is kept, in event order, with a quality flag; the 5 s limit is exceeded, so the strategy is degraded until fresh data arrives.
Practical rules
- Store provider timestamps and system receive timestamps.
- Deduplicate with stable event identifiers or sequence rules.
- Stop or degrade trading when data freshness exceeds a defined limit.
Common mistakes
- Treating arrival order as market order.
- Silently filling missing values without quality flags.
- Mixing adjusted historical prices with unadjusted live prices.
References
Official reading
These sources cover algorithmic controls, market access, broker APIs, system operations, or security.
Alpaca · WebSocket Market Data
Official example of receiving real-time stock, option, crypto, and news updates over a streaming API.
FINRA · Regulatory Notice 15-09
Official guidance on supervision, testing, controls, documentation, and monitoring for algorithmic trading strategies.