Every quant trader who has ever run a backtest that worked beautifully in simulation and catastrophically in production has asked the same question: What went wrong?

The answer, in roughly 40% of cases, is data quality. Specifically: unadjusted prices that inflate historical returns, timestamp misalignments that create phantom arbitrage signals, and outliers that warp statistical models. These are not edge cases. They are the default state of raw market data.

Raw exchange feeds are designed for real-time trading, not historical analysis. They prioritize speed and completeness of the live stream over the kind of cleaning and normalization that makes backtesting reliable. When you pull raw OHLCV data directly from a primary exchange feed, you inherit a set of data pathologies that silently corrupt your strategy evaluation.

This article dissects the five dimensions of TickDB's historical data cleaning and alignment pipeline. For each dimension, we compare the raw data state against the cleaned output, explain the engineering logic behind the transformation, and show what the difference means for your backtest results.

The Five Dimensions of Data Cleaning

Before diving into each dimension, here is the full landscape:

Dimension Raw Data Problem TickDB Solution Impact on Backtest
Corporate Actions Unadjusted prices create artificial jumps at split/ex-dividend dates Backward-adjusted price series Accurate historical return calculation
Timestamp Alignment Exchange timestamps may use local time, UTC, or epoch without documentation Normalized to UTC with millisecond precision Synchronized multi-asset analysis
Anomaly Detection Bad prints, exchange errors, and stale data pass through unmodified Multi-stage outlier detection and flagging Removes signal contamination
Data Completeness Gaps from exchange outages, non-trading periods, or API limitations Gap detection + interpolation or explicit NaN marking Prevents false regime detection
Cross-Venue Harmonization Different venues use different conventions for open/close, settlement Unified OHLCV construction methodology Reliable cross-market comparison

We will examine each in depth.

Dimension 1: Corporate Action Adjustment (复权)

The Problem with Unadjusted Prices

When a stock splits 2-for-1, its nominal price drops by 50% overnight. If you calculate returns using unadjusted close prices, the split date shows a catastrophic loss that never actually occurred in any trader's portfolio. The same pathology applies to dividend payments, bonus issues, and rights offerings.

Raw exchange data typically delivers prices as they occurred at each moment — nominal prices. For backtesting, this creates a systematic bias: historical returns before the split date are inflated by a factor equal to the cumulative split ratio.

Consider a stock that split 4-for-1 in 2020 and has not split since. An unadjusted price series shows a $400 high in 2018. In reality, that high represented $100 in effective value per share. Any strategy that targets a specific price level — such as "buy when price crosses above $200" — will produce entirely incorrect historical signals on unadjusted data.

TickDB's Adjustment Methodology

TickDB applies backward-adjusted pricing to all OHLCV (kline) data. The adjustment process works as follows:

  1. Identify all corporate action events: Splits, dividends, bonus issues, and rights offerings are catalogued from SEC filings and exchange announcements.
  2. Calculate cumulative adjustment factors: For each corporate action, compute a multiplier that converts the nominal historical price to the equivalent current-day price.
  3. Apply adjustment backward: Every historical bar's OHLC fields are multiplied by the cumulative adjustment factor applicable at that date.
  4. Preserve volume integrity: Volume is adjusted in the opposite direction (divided by the split ratio) to maintain the economic interpretation of volume as share count.

This produces a price series where the economic value of a share position is consistent across all time periods. A $100 move in 2018 represents the same economic move as a $100 move in 2024.

Practical Comparison

Date Raw Close Adjusted Close Split Factor
2023-01-10 $145.20 $145.20 1.0
2023-06-01 $180.50 $180.50 1.0
2023-06-02 (pre-split) $72.10 $180.25 2.5
2023-06-03 (post-split) $28.80 $144.00 2.5

After adjustment, the price series is continuous. The 50% "drop" on split day disappears, and the return series correctly reflects economic value.

Code Example: Fetching Adjusted Kline Data

import os
import requests

# Load API key from environment variable
API_KEY = os.environ.get("TICKDB_API_KEY")
if not API_KEY:
    raise EnvironmentError("Set TICKDB_API_KEY environment variable")

headers = {"X-API-Key": API_KEY}

# Fetch backward-adjusted daily kline for AAPL
params = {
    "symbol": "AAPL.US",
    "interval": "1d",
    "limit": 500
}

# Note: TickDB kline endpoint returns pre-adjusted data by default
# For forward-adjusted series, specify adjustment_type in params
response = requests.get(
    "https://api.tickdb.ai/v1/market/kline",
    headers=headers,
    params=params,
    timeout=(3.05, 10)
)

data = response.json()
if data.get("code") == 0:
    klines = data["data"]
    print(f"Fetched {len(klines)} adjusted kline records for AAPL.US")
else:
    print(f"Error {data['code']}: {data.get('message')}")

Backtest Impact

With unadjusted data, a simple moving average crossover strategy on AAPL from 2015–2023 would show a return series that spikes artificially at every split date. After adjustment, the return series is smooth, and performance attribution correctly reflects the periods of genuine price appreciation rather than accounting artifacts.

Dimension 2: Timestamp Alignment and Normalization

The Problem of Heterogeneous Timestamps

Financial markets operate across dozens of time zones and use at least three distinct timestamp conventions:

  • Exchange local time: NYSE uses Eastern Time (ET), but may or may not observe daylight saving time depending on historical era.
  • UTC: Many modern APIs default to UTC.
  • Unix epoch (milliseconds or seconds): Common in low-latency trading systems and some exchange feeds.

When you combine data from multiple sources — for example, US equity OHLCV alongside futures data from CME and crypto data from Binance — timestamp misalignment creates phantom signals. A arbitrage strategy that detects a price discrepancy between SPY and ES futures might be matching a 16:00 ET bar against a 21:00 UTC bar from the previous calendar day, creating a fabricated signal that would not exist in live trading.

TickDB's Timestamp Normalization

TickDB normalizes all timestamps to UTC using the following pipeline:

  1. Source identification: Each data source is tagged with its native timezone and timestamp resolution.
  2. Conversion to UTC: Timestamps are converted using the IANA timezone database (tzdata), which correctly handles historical daylight saving time transitions.
  3. Resolution standardization: Timestamps are stored with millisecond precision, with sub-millisecond noise rounded consistently.
  4. Bar boundary alignment: For kline data, the open time of each bar is anchored to the UTC boundary corresponding to the specified interval (e.g., 00:00 UTC for a 1D bar).

This means that when you fetch a 1-hour kline for AAPL and a 1-hour kline for TSLA, the bars are aligned to the same UTC boundaries. You can directly compare the 14:00–15:00 UTC bar across both securities without any offset calculation.

Critical Implication for Multi-Asset Strategies

The following table illustrates how the same calendar moment maps to different timestamps across raw sources:

Asset Exchange Raw Timestamp UTC Normalized
AAPL NYSE 2024-03-15 16:00:00 ET 2024-03-15 20:00:00 UTC
ES1! CME 2024-03-15 20:00:00 UTC 2024-03-15 20:00:00 UTC
BTCUSDT Binance 1710532800000 (epoch ms) 2024-03-15 20:00:00 UTC

Without normalization, a strategy that triggers on "price at NYSE market close" would fire at three different actual moments if applied to raw AAPL, ES, and BTC data simultaneously. With normalized timestamps, the trigger fires once — at the correct moment.

Dimension 3: Anomaly Detection and Flagging

What Constitutes an Anomaly in Market Data?

Anomalies in raw market data fall into three categories:

  1. Bad prints: Individual trades or quotes that violate physical or market-structure constraints — negative prices, trades at $0, volume that exceeds the total shares outstanding, or prices that move by 99% in a single millisecond.

  2. Exchange errors and corrections: Data vendors sometimes propagate corrected prices with a delay, or exchange maintenance windows generate placeholder values that get distributed as live data.

  3. Stale data propagation: When an exchange feed stops updating (due to outage or connectivity issues), the last known price continues to be broadcast. Without detection logic, this stale price appears as a period of zero volatility followed by a catastrophic jump when fresh data resumes.

TickDB's Multi-Stage Detection Pipeline

TickDB applies a layered anomaly detection system:

Stage 1 — Physical Constraint Checks

  • Price must be positive and within a physiologically plausible range relative to the security type (e.g., within 10 standard deviations of the trailing 30-day mean for equities).
  • Volume must not exceed the security's average daily volume by more than a configurable multiple (default: 50x).
  • For OHLCV bars: High ≥ Low; Open and Close within [Low, High].

Stage 2 — Statistical Outlier Detection

  • A rolling Z-score is computed for price returns and volume. Bars exceeding a Z-score threshold (default: 5.0) are flagged.
  • The window size adapts based on market volatility regime — wider windows during high-VIX periods to avoid false positives during genuine market stress.

Stage 3 — Contextual Consistency Checks

  • Price gaps between consecutive bars are compared against the security's typical gap distribution. An overnight gap exceeding 3x the 30-day average true range is flagged.
  • Opening price relative to previous close is checked against the security's typical open-to-close behavior.

Stage 4 — Flagging and Handling

  • Flagged bars are marked with a quality indicator rather than silently deleted.
  • The kline response includes a qualified field that signals whether a bar passed all checks.
  • For severe anomalies (e.g., clearly corrupted data), the bar may be replaced with null values rather than a fabricated number.

Comparison: Raw vs. Cleaned

Timestamp Raw Close Raw Volume Adjusted Close Quality Flag
2024-01-03 09:30:00 $185.42 12,450 $185.42 ✅ Pass
2024-01-03 09:30:01 $0.00 0 null ⚠️ Flagged
2024-01-03 09:30:02 $185.43 8,200 $185.43 ✅ Pass
2024-01-03 09:30:03 $185.43 8,100 $185.43 ✅ Pass

The $0.00 print at 09:30:01 — likely an exchange error — is flagged and nulled. A naive backtest on raw data would treat this as a 100% price drop and generate a catastrophic stop-loss signal.

Backtest Impact

Strategies that rely on volatility estimates — such as mean-reversion or options Greeks — are particularly sensitive to anomaly contamination. A single bad print with a 40% price spike, if unfiltered, will inflate the estimated volatility by a factor that causes the strategy to under-trade (wider stops, smaller position sizes) for the subsequent 20 trading days until the rolling window rolls off the anomaly.

Dimension 4: Data Completeness and Gap Handling

The Nature of Data Gaps

Market data gaps arise from multiple sources:

  • Exchange maintenance windows: Most exchanges have daily maintenance periods (typically 1–5 minutes) where no data is transmitted.
  • Trading halts: Reg SHO-mandated circuit breaker halts can pause trading for minutes to hours.
  • API rate limits and outages: Third-party data vendors may have gaps due to rate limiting or service interruptions.
  • Non-trading days: weekends, holidays, and early/late close days create systematic gaps.

Raw data typically represents these gaps as missing records. A backtest that iterates over "every 5-minute bar" will either crash when it hits a gap or silently skip the missing period, creating a subtle survivorship bias in time-series calculations.

TickDB's Completeness Pipeline

  1. Gap detection: For each continuous subscription or data feed, gaps are detected by comparing the observed sequence of timestamps against the expected sequence for the given interval.

  2. Gap classification:

    • Expected non-trading gap: Weekend, holiday, or market-defined early close. These are treated as intentional absence — no interpolation.
    • Unexpected gap: Exchange outage, API failure, or data vendor issue. These are flagged with a gap_reason field.

  3. Bar construction for unexpected gaps:

    • For short gaps (≤ 3 bars for intraday intervals), TickDB may construct a placeholder bar using the last known close as all four OHLC values, with volume set to 0. This preserves the time-series structure for iteration but clearly marks the bar as synthetic.
    • For longer gaps, the gap is left as explicit null records to prevent fabricated data from contaminating the series.

  4. Completeness metadata: Each kline response includes a complete boolean field indicating whether the bar represents a continuous trading period or a constructed placeholder.

Implications for Intraday Strategy Design

If you are running a mean-reversion strategy on 5-minute bars, a gap from 15:45 to 15:50 (due to an exchange feed dropout) will produce a synthetic bar that shows zero price movement. A strategy that calculates "current deviation from the 20-bar moving average" will understate the true deviation after the gap, potentially missing a trading signal.

With completeness metadata, you can explicitly handle gaps:

import os
import requests

API_KEY = os.environ.get("TICKDB_API_KEY")
headers = {"X-API-Key": API_KEY}

params = {
    "symbol": "AAPL.US",
    "interval": "5m",
    "start": 1710529800000,  # 2024-03-15 15:30 UTC
    "limit": 100
}

response = requests.get(
    "https://api.tickdb.ai/v1/market/kline",
    headers=headers,
    params=params,
    timeout=(3.05, 10)
)

data = response.json()
if data.get("code") == 0:
    klines = data["data"]
    for bar in klines:
        is_complete = bar.get("complete", True)
        close_price = bar["close"]
        if not is_complete:
            print(f"⚠️ Gap detected at {bar['open_time']}: synthetic bar, price={close_price}")
        else:
            print(f"✅ Bar at {bar['open_time']}: close={close_price}")

Dimension 5: Cross-Venue Harmonization

The Problem of Inconsistent Conventions

Different exchanges use different conventions for constructing OHLCV bars, even for the same underlying asset:

  • Opening auction vs. continuous open: Some exchanges (e.g., NYSE) include an opening auction in the "open" price of the first regular-session bar. Others define "open" as the first trade of continuous trading.
  • Closing price sources: Some exchanges use the closing auction price as the official close; others use the last trade.
  • Intraday bar boundaries: CME futures use a 23-hour trading day with a 1-hour maintenance window. Crypto exchanges use a 24-hour rolling window. Equities use 6.5-hour trading sessions.
  • Pre- and post-market data: US equities have official pre-market and after-hours sessions, but some data vendors include them in the same "daily" bar while others separate them.

When you combine data from multiple venues — for example, analyzing the relationship between ES futures and SPY ETF — these convention differences create systematic biases in any derived metric (correlations, betas, spread dynamics).

TickDB's Harmonization Approach

TickDB applies a unified bar construction methodology across all supported markets:

  1. Venue-specific pre-processing: Raw feeds are first processed according to their native conventions to extract the raw open, high, low, close, and volume.
  2. Standardization layer: A market-specific configuration layer applies the appropriate convention mapping:
    • Equities: Opening/closing auction prices used for official open/close; continuous trading for high/low.
    • Futures: Settlement price used for close; exchange-defined session hours.
    • Crypto: 24-hour rolling window from 00:00 UTC.
  3. Cross-venue alignment: For markets that trade on the same underlying (e.g., SPY and ES futures), TickDB aligns bar boundaries to a common reference time, ensuring that "the 14:00–15:00 UTC bar" means the same thing across both products.

Practical Comparison: CME Futures vs. Crypto

Property CME ES Futures Binance BTCUSDT
Raw session hours 23:00–22:00 UTC (rolls daily) 24/7
TickDB harmonized session Aligned to 00:00 UTC daily reset Aligned to 00:00 UTC daily reset
Close definition Settlement price Last trade
Settlement time 22:00 UTC N/A

After harmonization, a daily bar for ES and a daily bar for BTCUSDT both represent the 00:00–23:59:59.999 UTC window. Cross-asset correlation and spread analysis can proceed without correction factors.

Synthesis: Why the Cleaning Pipeline Matters for Your Backtest

The five dimensions above are not independent. They compound:

  • Timestamp misalignment × unadjusted prices = A backtest that calculates returns using bars from the wrong time periods at incorrect price levels.
  • Anomaly contamination × gap unhandling = A volatility model that overestimates risk, causing the strategy to under-allocate capital during the periods when it should be most aggressive.
  • Cross-venue inconsistency × no harmonization = A correlation matrix that understates the true diversification benefit of a multi-asset portfolio, leading to excessive concentration.

The practical consequence: a strategy that shows a Sharpe ratio of 1.8 on raw data might score 0.9 on cleaned, aligned data. The gap is not a bug in the strategy — it is the raw data lying to you.

TickDB's cleaning pipeline is designed to eliminate these five sources of systematic data bias, giving you a backtesting environment that more closely reflects the conditions a live strategy would face.

Next Steps

If you're evaluating data quality for your backtesting pipeline, explore TickDB's kline endpoint documentation to understand the full schema, including adjustment, completeness, and quality metadata fields.

If you want to test the cleaned data against your own raw source:

  1. Sign up at tickdb.ai (free, no credit card required)
  2. Generate an API key in the dashboard
  3. Set the TICKDB_API_KEY environment variable
  4. Run the code examples in this article to compare raw vs. cleaned data

If you need 10+ years of cleaned, aligned US equity OHLCV data for strategy backtesting, reach out to enterprise@tickdb.ai for institutional data plans covering historical depth, cross-venue coverage, and custom adjustment methodologies.

If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your AI tool's marketplace to integrate TickDB data retrieval directly into your workflow.

This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Backtested results are inherently limited by the quality of input data, the assumptions embedded in the strategy logic, and the market conditions of the historical period studied.