The Data Gap That Breaks Your Backtest

The strategy looks flawless on paper. Sharpe ratio of 1.42. Max drawdown under 8%. Win rate of 61%. You run it live for three weeks and watch it hemorrhage money while your backtest sailed through simulated conditions.

What went wrong?

In most cases, the answer is hiding in the gaps.

Market data is messy. Trading halts create voids. Lunch breaks in Asian markets partition the day into disconnected blocks. Network disconnections during WebSocket streaming truncate your time series. A 5-minute gap might look like "no price movement" in your dataframe — but for your backtest engine, it becomes an invisible bridge that compresses volatility, distorts momentum signals, and fabricates continuous return streams that never existed in reality.

This is the missing data problem in quantitative trading. It is not a corner case. It is the default condition of any real-world dataset.

This article dissects the three primary imputation strategies — forward fill, linear interpolation, and deletion — examines their mathematical implications on backtest results, and provides production-grade Python implementations you can deploy today.

Why Missing K-Lines Matter More Than You Think

A missing candle is not neutral. It is an active decision about how to represent reality, and every choice carries statistical consequences.

The Mechanics of a Missing Candle

When a K-line is absent from your dataset, the downstream effects ripple through your entire pipeline:

Effect Mechanism Consequence
Return distortion Gap return = (P_t+1 - P_t) / P_t Apparent jump includes unobserved intra-gap price movement
Volatility compression σ(Gap period) ≈ 0 in your series Realized volatility underestimated by 15–40% depending on gap frequency
Momentum signal decay Rolling windows include the gap Moving averages flatten artificially; momentum signals weaken
Technical indicator corruption RSI, MACD, Bollinger Bands all assume continuous sampling Calculations produce values that do not correspond to any actual market state

Types of Missing Data in K-Line Series

Before choosing an imputation strategy, you must identify the mechanism that created the gap. Different causes demand different treatments.

Gap Type Cause Characteristics Best Strategy
Scheduled absence Lunch breaks (HK, China markets), weekend closure Predictable, consistent duration, no trading activity Forward fill or deletion
Trading halt Circuit breaker, news suspension, exchange halt Unpredictable, variable duration, no trading activity Forward fill with warning flag
Network dropout WebSocket disconnection, API timeout Irregular, potentially includes trading activity Linear interpolation or deletion
API pagination artifact Incorrect limit/offset in REST fetch Consistent pattern, usually detectable Deletion + pipeline fix
Symbol delisting/suspension Corporate action, market rules Extended duration, no reactivation Deletion (do not impute)

The critical distinction is whether trading could have occurred during the gap. A lunch break is a genuine market-closed period. A network dropout is a data collection failure that may have obscured real price action.

Strategy 1: Forward Fill (Last Observation Carried Forward)

The Logic

Forward fill (FFILL) propagates the last known candle's OHLCV values forward until a new candle appears. The implicit assumption: if no trade occurred, the price did not change.

When to Use It

Forward fill is the correct choice when:

  • The market was genuinely closed during the gap (scheduled breaks, trading halts).
  • You are calculating indicator-based signals that tolerate stale prices (e.g., moving average crossover).
  • You need to maintain a continuous dataframe index for alignment with other time series.

The Math

P_t = P_(t-1)  for all t in Gap
Return_t = 0   for all t in Gap

This means the gap period contributes zero return and zero volatility. In a 5-day trading halt scenario, the strategy will show 5 days of flat returns that never occurred in live trading — but also never exposed the portfolio to gap-day risk.

Production-Grade Implementation

import os
import time
import random
import requests
import pandas as pd
from datetime import datetime, timedelta

# ⚠️ For production HFT workloads, use aiohttp/asyncio
# This implementation targets backtesting pipelines, not live trading

class KLineForwardFiller:
    """
    Fetch K-line data from TickDB and apply forward-fill imputation
    to handle scheduled market closures and trading halts.
    
    Critical behavior: gaps where trading COULD have occurred
    (network dropouts) are flagged, not silently filled.
    """
    
    BASE_URL = "https://api.tickdb.ai/v1/market/kline"
    
    def __init__(self, api_key: str = None):
        self.api_key = api_key or os.environ.get("TICKDB_API_KEY")
        if not self.api_key:
            raise ValueError(
                "TickDB API key required. "
                "Set TICKDB_API_KEY environment variable."
            )
        self.headers = {"X-API-Key": self.api_key}
    
    def fetch_with_retry(
        self,
        symbol: str,
        interval: str = "1h",
        limit: int = 1000,
        start_time: int = None,
        max_retries: int = 5
    ) -> dict:
        """
        Fetch K-line data with exponential backoff + jitter.
        Handles rate limits (code 3001) by respecting Retry-After header.
        """
        params = {
            "symbol": symbol,
            "interval": interval,
            "limit": limit
        }
        if start_time:
            params["start_time"] = start_time
        
        for attempt in range(max_retries):
            try:
                response = requests.get(
                    self.BASE_URL,
                    headers=self.headers,
                    params=params,
                    timeout=(3.05, 27)  # (connect, read) timeout
                )
                data = response.json()
                
                # ⚠️ Standard TickDB error handling
                code = data.get("code", 0)
                if code == 0:
                    return data.get("data", [])
                
                if code == 3001:
                    # Rate limited — respect Retry-After
                    retry_after = int(
                        response.headers.get("Retry-After", 5)
                    )
                    print(
                        f"Rate limited (attempt {attempt + 1}). "
                        f"Retrying in {retry_after}s..."
                    )
                    time.sleep(retry_after)
                    continue
                
                if code in (1001, 1002):
                    raise ValueError(
                        "Invalid API key — check TICKDB_API_KEY"
                    )
                if code == 2002:
                    raise KeyError(
                        f"Symbol {symbol} not found"
                    )
                
                raise RuntimeError(
                    f"API error {code}: {data.get('message')}"
                )
                
            except requests.exceptions.Timeout:
                delay = min(2 ** attempt * 0.5, 30)
                jitter = random.uniform(0, delay * 0.1)
                time.sleep(delay + jitter)
                continue
        
        raise RuntimeError(
            f"Failed after {max_retries} attempts"
        )
    
    def fill_forward(
        self,
        df: pd.DataFrame,
        max_gap_minutes: int = 30,
        flag_suspicious: bool = True
    ) -> pd.DataFrame:
        """
        Apply forward-fill imputation to K-line DataFrame.
        
        Args:
            df: DataFrame with columns [timestamp, open, high, low, close, volume]
            max_gap_minutes: Gap duration threshold — gaps longer than this
                             are flagged as suspicious (potential data loss).
            flag_suspicious: If True, adds 'gap_flag' column for long gaps.
        
        Returns:
            DataFrame with missing rows forward-filled and gap flags added.
        """
        df = df.copy()
        df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
        df = df.set_index('timestamp')
        df = df.sort_index()
        
        # Generate complete time index at the specified interval
        full_range = pd.date_range(
            start=df.index.min(),
            end=df.index.max(),
            freq='1h'  # Match your interval parameter
        )
        
        # Reindex to insert missing rows
        df_filled = df.reindex(full_range)
        gap_mask = df_filled['close'].isna()
        
        # ⚠️ Engineering warning: FFILL assumes market closure.
        # Do NOT use this on gaps where trading may have occurred.
        df_filled = df_filled.ffill()
        
        if flag_suspicious and gap_mask.any():
            # Flag suspiciously long gaps for manual review
            gap_durations = []
            in_gap = False
            gap_start = None
            
            for ts in full_range:
                if ts in df.index:
                    if in_gap:
                        duration = (ts - gap_start).total_seconds() / 60
                        gap_durations.append(duration)
                    in_gap = False
                else:
                    if not in_gap:
                        gap_start = ts
                    in_gap = True
            
            df_filled['gap_flag'] = [
                d > max_gap_minutes if pd.notna(d) else False
                for d in gap_durations
            ] + [False]  # Handle edge case
        
        return df_filled.reset_index().rename(
            columns={'index': 'timestamp'}
        )


# Usage example
if __name__ == "__main__":
    filler = KLineForwardFiller()
    
    raw_data = filler.fetch_with_retry(
        symbol="AAPL.US",
        interval="1h",
        limit=500
    )
    
    df = pd.DataFrame(raw_data)
    df_filled = filler.fill_forward(df, max_gap_minutes=30)
    
    print(f"Original rows: {len(df)}")
    print(f"After fill: {len(df_filled)}")
    print(f"Rows with suspicious gaps: {df_filled['gap_flag'].sum()}")

Backtest Bias from Forward Fill

Metric Effect of FFILL Direction
Realized volatility Understated Underestimate
Maximum drawdown Understated Underestimate
Strategy Sharpe ratio Overstated Overestimate
Turnover Understated (fewer trading days) Underestimate

The bias magnitude scales with gap frequency. In markets with daily lunch breaks (HK, China A-shares), a 1-hour lunch gap on a 5-minute interval creates 12 artificial candles per day. Over a 1-year backtest, that is 3,000 imputed rows — approximately 30% of the dataset for a 5-minute strategy.

Strategy 2: Linear Interpolation

The Logic

Linear interpolation (LERP) estimates missing values as a weighted average between the last known value and the next known value. The implicit assumption: price moved at a constant rate during the gap.

When to Use It

Linear interpolation is appropriate when:

  • You suspect data collection failure during the gap (network dropouts).
  • The gap duration is short (under 5 periods for typical intervals).
  • You are calculating returns-based metrics and need to preserve return continuity.

The Math

P_mid = P_before + (P_after - P_before) * (gap_position / gap_length)

For a 3-period gap with P_before = 100 and P_after = 106:

Period 1: 100 + (6 * 1/3) = 102
Period 2: 100 + (6 * 2/3) = 104
Period 3: 100 + (6 * 3/3) = 106

Critical Warning: Why LERP is Dangerous for K-Lines

Before showing the code, understand the fundamental problem with linear interpolation on OHLCV data.

Open, High, Low, Close are not continuous variables.

Linear interpolating between two closes produces artificial OHLC values that violate market mechanics:

# What LERP might produce for a gap from $100 to $106:
# Before gap: close = 100, high = 102, low = 98
# After gap:  close = 106, high = 108, low = 104
# Interpolated (incorrect):
#   Period 2: close = 103, high = 103, low = 103  # All equal — impossible
#   Period 3: close = 105, high = 106, low = 104  # High < close — violates OHLC logic

Only the close price should be linearly interpolated. High, low, and volume require different treatment — typically bounded interpolation or deletion.

Production-Grade Implementation

import numpy as np
import pandas as pd
from typing import Literal

class KLineInterpolator:
    """
    Apply linear interpolation to K-line close prices while
    handling OHLC constraints properly.
    
    ⚠️ CRITICAL: High, Low, Volume use bounded interpolation
    or are set to NA. Do NOT naively linear-interpolate OHLC.
    """
    
    def __init__(self, max_interpolation_gaps: int = 5):
        """
        Args:
            max_interpolation_gaps: Maximum gap length to interpolate.
                                   Longer gaps are flagged and deleted.
                                   Prevents interpolating across trading halts.
        """
        self.max_gaps = max_interpolation_gaps
    
    def identify_gaps(self, df: pd.DataFrame, freq: str = '5T') -> pd.Series:
        """
        Identify gap locations and lengths in the time series.
        Returns a Series with gap lengths (0 = no gap).
        """
        df = df.copy()
        df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
        df = df.set_index('timestamp')
        df = df.sort_index()
        
        full_range = pd.date_range(
            start=df.index.min(),
            end=df.index.max(),
            freq=freq
        )
        
        # Insert missing rows as NaN
        df_reindexed = df.reindex(full_range)
        
        # Calculate gap lengths (consecutive NaN counts)
        gap_lengths = df_reindexed['close'].isna().astype(int)
        gap_group_id = (gap_lengths != gap_lengths.shift()).cumsum()
        gap_lengths_grouped = gap_lengths.groupby(gap_group_id).transform(
            'sum' * gap_lengths.astype(bool)  # Will recalculate below
        )
        
        # Proper gap length calculation
        gap_info = []
        in_gap = False
        gap_count = 0
        
        for idx in full_range:
            if idx in df.index:
                if in_gap:
                    gap_info.append(gap_count)
                in_gap = False
                gap_count = 0
            else:
                in_gap = True
                gap_count += 1
        
        if in_gap:
            gap_info.append(gap_count)
        
        return pd.Series(
            gap_info + [0] * (len(full_range) - len(gap_info)),
            index=full_range[:len(gap_info + [0] * (len(full_range) - len(gap_info)))]
        )
    
    def interpolate_close_only(
        self,
        df: pd.DataFrame,
        freq: str = '5T'
    ) -> pd.DataFrame:
        """
        Linear interpolate close prices only.
        
        High and Low are set to the interpolated close value.
        Volume is set to 0 for interpolated rows.
        
        This preserves return continuity while avoiding OHLC violations.
        """
        df = df.copy()
        df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
        df = df.set_index('timestamp')
        df = df.sort_index()
        
        full_range = pd.date_range(
            start=df.index.min(),
            end=df.index.max(),
            freq=freq
        )
        
        df_reindexed = df.reindex(full_range)
        
        # Identify gap lengths
        gap_lengths = self.identify_gaps(df_reindexed, freq)
        
        # Linear interpolate close only
        df_reindexed['close'] = df_reindexed['close'].interpolate(
            method='linear'
        )
        
        # For High: max of interpolated close and surrounding highs (bounded)
        # For Low: min of interpolated close and surrounding lows (bounded)
        # This is a simplified approach; production systems may use more sophisticated bounds
        df_reindexed['high'] = df_reindexed['close'].copy()
        df_reindexed['low'] = df_reindexed['close'].copy()
        df_reindexed['volume'] = df_reindexed['volume'].fillna(0)
        
        # Flag rows that were interpolated
        df_reindexed['interpolated'] = gap_lengths > 0
        
        # Flag rows that exceed our interpolation threshold
        df_reindexed['exceeds_threshold'] = gap_lengths > self.max_gaps
        
        return df_reindexed.reset_index().rename(
            columns={'index': 'timestamp'}
        )
    
    def get_clean_dataframe(
        self,
        df: pd.DataFrame,
        freq: str = '5T',
        drop_long_gaps: bool = True
    ) -> pd.DataFrame:
        """
        Return a clean DataFrame with appropriate imputation applied.
        
        Args:
            df: Raw K-line DataFrame
            freq: Frequency string for index generation
            drop_long_gaps: If True, rows with gaps > max_gaps are excluded
                           from the final output (recommended).
        
        Returns:
            Clean DataFrame with interpolation applied.
        """
        result = self.interpolate_close_only(df, freq)
        
        if drop_long_gaps:
            # Remove rows that came from gaps exceeding our threshold
            # These represent trading halts or data collection failures
            result = result[~result['exceeds_threshold']]
        
        return result


# Usage example
if __name__ == "__main__":
    interpolator = KLineInterpolator(max_interpolation_gaps=5)
    
    # Assuming df_raw is your fetched K-line data
    df_clean = interpolator.get_clean_dataframe(
        df_raw,
        freq='5T',
        drop_long_gaps=True
    )
    
    interpolated_count = df_clean['interpolated'].sum()
    total_count = len(df_clean)
    
    print(
        f"Interpolated {interpolated_count} of {total_count} rows "
        f"({100 * interpolated_count / total_count:.2f}%)"
    )

Backtest Bias from Linear Interpolation

Metric Effect of LERP Direction
Realized volatility Approximates true volatility Unbiased (if gap is short)
Maximum drawdown Slightly understated Underestimate
Strategy Sharpe ratio Unbiased for short gaps Neutral
Indicator accuracy Degraded for momentum strategies Underestimate

Linear interpolation introduces mean-reversion bias. Prices rarely move linearly — they overshoot, reverse, and trend. Forcing a linear path between two known prices artificially suppresses the volatility that occurred inside the gap.

Strategy 3: Deletion (Pairwise or Listwise)

The Logic

Deletion removes rows with missing data rather than imputing them. Two variants:

  • Pairwise deletion: Exclude a row only from calculations that require the missing column.
  • Listwise deletion: Exclude the entire row from all calculations.

In backtesting, listwise deletion is the standard approach — you exclude the gap period entirely from your return series and signal generation.

When to Use It

Deletion is the correct choice when:

  • The gap represents a genuine market closure (do not model returns during closed markets).
  • The gap duration exceeds your tolerance for imputation error.
  • Your strategy's signals are sensitive to price path assumptions (momentum, mean-reversion).
  • You are calculating metrics that should not include non-trading periods (e.g., annual return).

Production-Grade Implementation

import pandas as pd
from typing import List, Optional

class KLineGapPruner:
    """
    Remove K-line gaps from DataFrame based on configurable rules.
    
    Supports three modes:
    1. 'listwise': Remove all rows in gap periods
    2. 'conservative': Remove gaps longer than threshold only
    3. 'aggressive': Remove ANY gap, regardless of length
    """
    
    def __init__(
        self,
        mode: Literal['listwise', 'conservative', 'aggressive'] = 'conservative',
        max_acceptable_gap_minutes: int = 120
    ):
        self.mode = mode
        self.max_gap = max_acceptable_gap_minutes
    
    def detect_gaps(
        self,
        df: pd.DataFrame,
        freq: str = '5T'
    ) -> pd.DataFrame:
        """
        Detect and annotate gaps in the K-line series.
        
        Returns DataFrame with columns:
        - timestamp: Row timestamp
        - is_gap: Boolean indicating if this row falls in a gap
        - gap_duration_minutes: Length of the gap containing this row
        """
        df = df.copy()
        df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
        df = df.set_index('timestamp')
        df = df.sort_index()
        
        full_range = pd.date_range(
            start=df.index.min(),
            end=df.index.max(),
            freq=freq
        )
        
        # Create gap detection DataFrame
        detection = pd.DataFrame(index=full_range)
        detection['has_data'] = detection.index.isin(df.index)
        
        # Calculate gap start/end markers
        detection['gap_start'] = (~detection['has_data']) & (
            detection['has_data'].shift(1).fillna(False)
        )
        detection['gap_end'] = (~detection['has_data']) & (
            detection['has_data'].shift(-1).fillna(False)
        )
        
        # Forward-fill to propagate gap start/end markers
        detection['current_gap_start'] = detection['gap_start'].cumsum()
        
        # Calculate gap durations
        gap_info = []
        current_gap_id = None
        gap_start_ts = None
        gap_length = 0
        
        for ts in full_range:
            if detection.loc[ts, 'has_data']:
                if current_gap_id is not None:
                    # End of gap — record duration for all gap rows
                    gap_info.append({
                        'gap_id': current_gap_id,
                        'gap_start': gap_start_ts,
                        'gap_duration_minutes': gap_length * self._freq_to_minutes(freq)
                    })
                    current_gap_id = None
                gap_info.append({
                    'gap_id': None,
                    'gap_start': None,
                    'gap_duration_minutes': 0
                })
            else:
                if current_gap_id is None:
                    current_gap_id = detection.loc[ts, 'current_gap_start']
                    gap_start_ts = ts
                    gap_length = 1
                else:
                    gap_length += 1
                gap_info.append({
                    'gap_id': current_gap_id,
                    'gap_start': gap_start_ts,
                    'gap_duration_minutes': 0  # Will be backfilled
                })
        
        gap_df = pd.DataFrame(gap_info, index=full_range)
        detection = detection.join(gap_df)
        
        # Backfill gap duration
        detection['gap_duration_minutes'] = detection.groupby('gap_id')[
            'gap_duration_minutes'
        ].transform(
            lambda x: x.replace(0, np.nan).ffill().bfill()
        )
        
        detection['is_gap'] = ~detection['has_data']
        
        return detection.reset_index().rename(
            columns={'index': 'timestamp'}
        )
    
    def _freq_to_minutes(self, freq: str) -> int:
        """Convert pandas frequency string to minutes."""
        freq_map = {
            '1T': 1, '5T': 5, '10T': 10, '15T': 15,
            '30T': 30, '1H': 60, '2H': 120, '4H': 240,
            '1D': 1440
        }
        return freq_map.get(freq, 5)  # Default to 5 minutes
    
    def prune(
        self,
        df: pd.DataFrame,
        freq: str = '5T',
        return_gap_report: bool = False
    ) -> pd.DataFrame:
        """
        Remove gap rows according to the configured mode.
        
        Args:
            df: Raw K-line DataFrame
            freq: Frequency string for index generation
            return_gap_report: If True, return (pruned_df, gap_report)
        
        Returns:
            Pruned DataFrame, or (pruned_df, gap_report) if return_gap_report=True
        """
        df = df.copy()
        df['timestamp'] = pd.to_datetime(df['timestamp'], unit='ms')
        df = df.set_index('timestamp')
        df = df.sort_index()
        
        gaps = self.detect_gaps(df.reset_index(), freq)
        
        if self.mode == 'listwise':
            # Remove all gap rows (regardless of duration)
            gap_timestamps = gaps[gaps['is_gap']]['timestamp']
            df_pruned = df.drop(index=gap_timestamps)
            
        elif self.mode == 'conservative':
            # Remove only gaps exceeding threshold
            long_gap_timestamps = gaps[
                gaps['is_gap'] & 
                (gaps['gap_duration_minutes'] > self.max_gap)
            ]['timestamp']
            df_pruned = df.drop(index=long_gap_timestamps)
            
        elif self.mode == 'aggressive':
            # Remove gaps of ANY length
            # Use full time range — keep only rows with data
            df_pruned = df.copy()  # Already filtered to only data rows
        
        gap_report = gaps.groupby('gap_duration_minutes').size().reset_index(
            name='count'
        ).sort_values('gap_duration_minutes')
        
        if return_gap_report:
            return df_pruned.reset_index(), gap_report
        return df_pruned.reset_index()

Backtest Bias from Deletion

Metric Effect of Deletion Direction
Realized volatility Accurate (excludes non-trading periods) Unbiased
Maximum drawdown Accurate Unbiased
Strategy Sharpe ratio Unbiased Neutral
Turnover Accurate Unbiased
Time series continuity Broken (may affect rolling window calculations) Requires careful handling

Deletion preserves the most accurate return and volatility estimates but requires care when using rolling window calculations. A 20-period moving average computed on a dataset with deletions will have effective lookback that varies with gap frequency — your window is sometimes 20 actual periods, sometimes 20 calendar periods with fewer trading periods.

Comparative Analysis: When Each Strategy Wins

Side-by-Side Comparison

Criterion Forward Fill Linear Interpolation Deletion
Market closure gaps ✅ Correct ❌ Introduces false returns ✅ Correct
Network dropout gaps ❌ Conceals missing data ✅ Preserves continuity ✅ Preserves accuracy
Volatility preservation ❌ Understated ✅ Approximate ✅ Accurate
Return continuity ❌ Zero return in gap ✅ Smoothed returns ❌ Discontinuous
Implementation complexity Low Medium Medium
Backtest bias Overstates Sharpe Mixed Minimal
Recommended for Scheduled breaks, halts Short data gaps All cases (primary)

Decision Flowchart

Is the gap a scheduled market closure (lunch, weekend, halt)?
├── YES → Use DELETION (correct) or FORWARD FILL (if you need continuous index)
└── NO → Is the gap short (< 5 periods) and likely a data collection failure?
         ├── YES → Use LINEAR INTERPOLATION on close price only
         └── NO → Use DELETION (do not impute across unknown market conditions)

Quantitative Bias Example

Consider a momentum strategy on HK stock data with a 12:00–13:00 lunch break gap (12 periods on a 5-minute interval). Over 1 year:

Metric Forward Fill Linear Interpolation Deletion
Trading days in backtest 252 252 252
Total periods 75,600 75,600 64,800 (after deletion)
Implied annual volatility 18.4% 21.2% 24.1%
Strategy Sharpe (example) 1.42 1.28 1.15

The forward-filled strategy shows a 1.42 Sharpe because it never experienced the uncertainty of the lunch period. The deleted strategy shows 1.15 — closer to what live trading would produce. The 0.27 Sharpe difference is pure backtest fiction.

Best Practices for Production Systems

1. Always Log Gap Metadata

Before filling or deleting, record:

  • Gap start/end timestamps
  • Gap duration
  • Likely cause (based on exchange schedule vs. anomaly detection)
# After fetching data, immediately log gaps
gap_report, _ = gap_pruner.prune(
    df_raw, freq='5T', return_gap_report=True
)
print(f"Gap distribution:\n{gap_report}")
# If large gaps exist (>60 min), alert for manual review

2. Validate Fill Strategy Against Exchange Schedule

Cross-reference detected gaps against known exchange closures:

# HKEX lunch break: 12:00–13:00 HKT
# NYSE regular hours: 09:30–16:00 ET
# NYSE early close: 13:00 ET ( Wednesdays after Thanksgiving, Christmas Eve)

EXCHANGE_SCHEDULES = {
    'HKEX': [(12, 0), (13, 0)],  # Lunch break
    'NYSE': [(9, 30), (16, 0)],
}

def is_expected_gap(timestamp, exchange):
    hour = timestamp.hour
    minute = timestamp.minute
    for start_h, start_m in EXCHANGE_SCHEDULES[exchange]:
        end_h, end_m = start_h, start_m + 60  # 1-hour break
        if (hour == start_h and minute >= start_m) or \
           (hour == end_h and minute < end_m):
            return True
    return False

3. Separate Backtest Data from Live Data Pipelines

Never use the same filling logic for historical backtesting and live trading. Live data should be handled differently:

  • Backtest: Apply consistent imputation to all historical data, document the strategy, use deletion for most cases.
  • Live: Forward fill is acceptable for display dashboards; never forward fill live strategy signals without explicit gap handling.

4. Run Sensitivity Analysis

Before trusting any backtest result, vary your imputation strategy and observe the effect on performance:

def sensitivity_analysis(df_raw, strategy_fn, param_name, param_values):
    """
    Run backtest across multiple imputation parameter values.
    Report performance range to quantify model uncertainty.
    """
    results = []
    
    for value in param_values:
        df_test = strategy_fn(df_raw, **{param_name: value})
        metrics = run_backtest(df_test)
        results.append({param_name: value, **metrics})
    
    df_results = pd.DataFrame(results)
    
    # Report sensitivity range
    sharpe_range = (
        df_results['sharpe'].max() - df_results['sharpe'].min()
    )
    print(
        f"Sharpe sensitivity to {param_name}: {sharpe_range:.3f}"
    )
    print(f"Full range: {df_results['sharpe'].min():.2f} – {df_results['sharpe'].max():.2f}")
    
    return df_results

If Sharpe ratio swings by more than 0.2 across reasonable imputation choices, your strategy's edge is likely smaller than reported — or nonexistent.

Conclusion

Missing K-line data is not a data quality problem you can fix once and forget. It is a modeling decision that must be made deliberately, documented rigorously, and tested for sensitivity.

The hierarchy of correctness:

  1. Deletion is the most honest approach — it does not fabricate data that does not exist.
  2. Forward fill is appropriate for scheduled closures where you need a continuous index.
  3. Linear interpolation is a last resort for short data collection gaps — and only on close prices.

Whatever strategy you choose, run sensitivity analysis. Report your imputation method in any backtest disclosure. And if your Sharpe ratio drops by more than 0.2 points when you switch from forward fill to deletion, treat that gap as a warning sign — not a data problem to smooth away.

Next Steps

If you're building a backtesting pipeline and need reliable historical K-line data with proper market closure metadata:

  1. Sign up at tickdb.ai (free API key, no credit card required)
  2. Use the /v1/market/kline endpoint with the appropriate interval for your strategy
  3. Apply the gap detection and deletion logic from this article before running your backtest

If you're comparing data vendors for K-line completeness:

  • Visit tickdb.ai for a feature comparison showing data coverage across US equities, HK stocks, crypto, and other asset classes.
  • The depth and kline endpoints support 10+ years of cleaned, aligned OHLCV data — suitable for cross-cycle backtesting.

If you use AI coding assistants:

Search for and install the tickdb-market-data SKILL in your AI tool's marketplace for direct access to market data integration in your development workflow.

This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Backtest results are historical simulations subject to inherent limitations including look-ahead bias, survivorship bias, and data snooping. Always validate strategies on out-of-sample data before live deployment.