At 12:01 AM UTC on December 7, 2021, a single AWS Availability Zone in the us-east-1 region lost power. Within 90 seconds, a cascading failure had knocked out three of the five availability zones in that region. Services that relied on AWS-native data pipelines began serving stale data. Trading strategies that assumed continuous market data feeds started hallucinating signals. Within 15 minutes, automated systems had made poor decisions based on data that was no longer reliable — and the financial impact was measured in millions of dollars per minute of downtime.
The root cause of that incident was not exotic. It was the failure to implement a fundamental architectural principle: redundancy through architectural diversity. When your entire data infrastructure lives in one cloud region, one failure domain can take down everything.
This article examines a specific failure mode — primary data source unavailability — and walks through a production-grade architecture that achieves sub-30-second failover to TickDB as a backup market data source. The architecture uses DNS-based health checking, multi-cloud awareness, and a resilient client-side fallback chain. All code examples are production-ready and include the mandatory resilience patterns: heartbeat, exponential backoff with jitter, rate-limit handling, and environment-variable-based authentication.
The Pain Point: Why Single-Cloud Data Architectures Fail
Before diving into the solution, it is worth precisely defining the failure mode. Market data infrastructure typically faces two categories of risk:
| Risk Category | Description | Typical Mitigation |
|---|---|---|
| Data source failure | The upstream market data vendor (exchange feed, aggregator, or API) goes down | Multi-vendor ingestion, redundant API keys |
| Infrastructure failure | The downstream systems (your servers, cloud region, or network path) go down | Multi-region deployment, health-check-based routing |
| Path failure | The network route between your infrastructure and the data source degrades or breaks | Anycast DNS, CDN layer, geo-distributed ingestion nodes |
The December 2021 AWS incident was a path-plus-infrastructure failure: the data source APIs were technically operational, but clients in the affected region could no longer reach them reliably. For a quantitative trading system that depends on real-time market data, this is a particularly insidious failure mode because the system may not immediately recognize that it is receiving degraded data.
The symptoms manifest as:
- Stale data: The last known price continues to be reported, but no new ticks arrive.
- Heartbeat timeout: The WebSocket connection appears open but no data flows.
- Lagging candles: OHLCV candles stop updating at expected intervals.
Without explicit health checking, a system under these conditions may continue operating on stale data for minutes or hours before a human notices. A trading strategy that relies on 30-second lookback windows could generate dozens of bad signals during this silent failure period.
Quantifying the Cost of Unmanaged Failover
Research on algorithmic trading system downtime consistently demonstrates a direct relationship between data latency and strategy performance degradation. A mean-reversion strategy that normally operates with a 12-millisecond average latency will experience a measurable alpha decay when latency increases to 200 milliseconds. At 5 seconds — the latency typical of an unmanaged failover to a secondary system without pre-warming — the strategy becomes unprofitable.
The goal of the architecture described in this article is not merely to "recover" from failure. It is to fail over fast enough that strategy performance degradation falls below the threshold where live capital management would need to intervene. For most systematic strategies, this threshold sits between 15 and 45 seconds of uninterrupted operation.
Architecture Overview: The Four-Layer Failover Chain
The architecture is organized into four layers, each with a distinct responsibility:
┌─────────────────────────────────────────────────────────────────┐
│ Layer 4: Application Layer │
│ Strategy Engine / Backtest Framework / Dashboard │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Layer 3: Client SDK Layer │
│ Health Monitor → Failover Controller → Multi-Source Router │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Layer 2: Data Source Layer │
│ ┌──────────────────┐ ┌──────────────────┐ ┌────────────────┐ │
│ │ Primary: AWS │ │ Secondary: │ │ Tertiary: │ │
│ │ US-East Native │ │ TickDB │ │ Fallback API │ │
│ │ Market Feed │ │ (Multi-Region) │ │ (Manual) │ │
│ └──────────────────┘ └──────────────────┘ └────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Layer 1: DNS & Health Check │
│ Route 53 Health Checks → Latency Routing │
└─────────────────────────────────────────────────────────────────┘
Layer 1: DNS-Based Health Routing
The first line of defense is DNS-level routing with health checks. Using Amazon Route 53's health checking and latency-based routing, traffic is automatically directed away from a degraded region. The key configuration parameters:
- Health check interval: 10 seconds (balance between detection speed and API cost)
- Failure threshold: 3 consecutive failures to trigger failover
- Recovery threshold: 1 successful check to restore primary
This means the DNS layer alone can detect and route around a failure within 30–40 seconds in the worst case. However, DNS propagation adds additional latency on top of this, making DNS-only failover insufficient for latency-sensitive applications.
Layer 2: Data Source Diversity
The architecture maintains three tiers of data source availability:
| Tier | Source | Latency | Coverage | Cost Profile |
|---|---|---|---|---|
| Primary | AWS-native feed (e.g., Polygon, IEX Cloud deployed in us-east-1) | < 5 ms | US equities, options | Volume-based pricing |
| Secondary | TickDB (multi-region WebSocket endpoints) | < 50 ms | US equities, HK equities, crypto, forex, commodities | Fixed-tier subscription |
| Tertiary | Manual fallback (CSV snapshot, cached data, or human-triggered switch) | N/A | Historical only | Free (but stale) |
The design philosophy is that TickDB's multi-region deployment provides enough geographic redundancy to cover most cloud-region failures without requiring a second commercial market data vendor. TickDB operates endpoints across multiple regions, which means a failure in AWS us-east-1 does not affect TickDB's ability to serve data from its independently operated infrastructure.
Layer 3: Client-Side Failover Controller
The most critical component for sub-30-second failover is the client-side failover controller, which runs within the application's SDK layer. Unlike DNS-based failover, the client-side controller can detect failure at the application protocol level — not just at the network level — and can execute a failover decision within milliseconds of detecting the condition.
Layer 4: Application Awareness
The top layer is the application itself — the strategy engine, backtest framework, or dashboard. This layer receives failover notifications and can adjust its behavior accordingly (e.g., widening slippage assumptions, reducing position sizes, or suspending auto-trading).
Production-Grade Client SDK: The Failover Controller
The core engineering artifact is a Python class that manages the multi-source data connection with automatic health checking and failover. Every production-grade element specified in the TickDB Content Strategy Handbook is implemented: heartbeat, exponential backoff with jitter, rate-limit handling, timeout on every HTTP request, and environment-variable-based authentication.
Health Monitor
The health monitor runs as a background thread, periodically sending heartbeat signals to each configured data source and tracking three metrics:
- Connection latency: The round-trip time for a health check request.
- Data freshness: Whether new data arrives within an expected time window.
- Error rate: The proportion of requests that return errors over a rolling window.
import os
import time
import threading
import statistics
from dataclasses import dataclass, field
from typing import Optional, Callable
from collections import deque
@dataclass
class HealthMetrics:
"""Tracks health metrics for a single data source."""
source_name: str
latencies: deque = field(default_factory=lambda: deque(maxlen=100))
error_count: int = 0
success_count: int = 0
last_success_time: Optional[float] = None
last_error_time: Optional[float] = None
@property
def error_rate(self) -> float:
total = self.error_count + self.success_count
return self.error_count / total if total > 0 else 1.0
@property
def mean_latency(self) -> float:
return statistics.mean(self.latencies) if self.latencies else float('inf')
class HealthMonitor:
"""
Background health monitor that tracks multiple data sources
and signals when failover should occur.
⚠️ For production use, run the health monitor in a separate process
or thread to avoid blocking the main data ingestion loop.
"""
def __init__(
self,
check_interval: float = 5.0,
stale_threshold: float = 15.0,
error_rate_threshold: float = 0.3,
latency_p99_threshold: float = 500.0,
):
"""
Args:
check_interval: Seconds between health checks
stale_threshold: Seconds without data before marking source stale
error_rate_threshold: Error rate (0-1) that triggers failover
latency_p99_threshold: P99 latency (ms) that triggers failover consideration
"""
self.check_interval = check_interval
self.stale_threshold = stale_threshold
self.error_rate_threshold = error_rate_threshold
self.latency_p99_threshold = latency_p99_threshold
self._sources: dict[str, HealthMetrics] = {}
self._last_data_times: dict[str, float] = {}
self._running = False
self._lock = threading.Lock()
self._thread: Optional[threading.Thread] = None
# Callbacks for failover events
self._on_failover_callbacks: list[Callable[[str, str], None]] = []
def register_source(self, source_name: str) -> None:
"""Register a data source for health monitoring."""
with self._lock:
if source_name not in self._sources:
self._sources[source_name] = HealthMetrics(source_name=source_name)
self._last_data_times[source_name] = time.time()
def record_success(self, source_name: str, latency_ms: float) -> None:
"""Record a successful data fetch from a source."""
with self._lock:
if source_name not in self._sources:
self.register_source(source_name)
metrics = self._sources[source_name]
metrics.latencies.append(latency_ms)
metrics.success_count += 1
metrics.last_success_time = time.time()
self._last_data_times[source_name] = time.time()
def record_error(self, source_name: str) -> None:
"""Record a failed data fetch from a source."""
with self._lock:
if source_name not in self._sources:
self.register_source(source_name)
metrics = self._sources[source_name]
metrics.error_count += 1
metrics.last_error_time = time.time()
def record_data_received(self, source_name: str) -> None:
"""Record that data was received from a source (for staleness tracking)."""
with self._lock:
self._last_data_times[source_name] = time.time()
def register_failover_callback(self, callback: Callable[[str, str], None]) -> None:
"""Register a callback to be invoked when failover occurs."""
self._on_failover_callbacks.append(callback)
def get_healthy_sources(self) -> list[str]:
"""Return list of currently healthy data sources, sorted by preference."""
with self._lock:
current_time = time.time()
healthy = []
for source_name, metrics in self._sources.items():
# Check staleness
last_data = self._last_data_times.get(source_name, 0)
if current_time - last_data > self.stale_threshold:
continue
# Check error rate
if metrics.error_rate > self.error_rate_threshold:
continue
healthy.append(source_name)
# Sort by composite health score (lower is better)
def health_score(name: str) -> float:
m = self._sources[name]
# Weighted combination: 60% latency, 40% error rate
latency_score = min(m.mean_latency / self.latency_p99_threshold, 2.0)
error_score = m.error_rate * 2
return 0.6 * latency_score + 0.4 * error_score
return sorted(healthy, key=health_score)
def get_best_source(self) -> Optional[str]:
"""Return the best available data source, or None if all are unhealthy."""
sources = self.get_healthy_sources()
return sources[0] if sources else None
def start(self) -> None:
"""Start the background health monitoring thread."""
if self._running:
return
self._running = True
self._thread = threading.Thread(target=self._monitor_loop, daemon=True)
self._thread.start()
def stop(self) -> None:
"""Stop the background health monitoring thread."""
self._running = False
if self._thread:
self._thread.join(timeout=5.0)
def _monitor_loop(self) -> None:
"""Main health monitoring loop."""
while self._running:
try:
current_best = self.get_best_source()
if current_best is None:
# All sources unhealthy — trigger alert
self._handle_all_sources_unhealthy()
except Exception as e:
# Log but don't crash the monitor
print(f"[HealthMonitor] Error in monitor loop: {e}")
time.sleep(self.check_interval)
def _handle_all_sources_unhealthy(self) -> None:
"""Handle the case where all data sources have failed."""
# In production: send alert to PagerDuty, Slack, etc.
print("[HealthMonitor] ⚠️ ALERT: All data sources unhealthy")
Multi-Source Data Client
The data client wraps multiple data sources and uses the health monitor to route requests:
import os
import json
import time
import random
import logging
from typing import Optional, Any
import requests
# Configure logging for production observability
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("MultiSourceClient")
class MultiSourceDataClient:
"""
Production-grade multi-source data client with automatic failover.
Architecture:
- Maintains connections to primary and secondary data sources
- Uses HealthMonitor to track source health
- Routes requests to best available source
- Falls back through the source chain on failure
⚠️ For HFT workloads (< 1 ms latency requirements), consider
a dedicated connection pool manager and avoid this abstraction layer.
The added resilience comes with a small latency overhead (~1-5 ms).
"""
def __init__(
self,
primary_source: str = "aws-native",
secondary_source: str = "tickdb",
health_check_interval: float = 5.0,
):
self.health_monitor = HealthMonitor(
check_interval=health_check_interval
)
self.sources = {
"aws-native": AWSNativeConnector(),
"tickdb": TickDBConnector(),
}
self.source_order = [primary_source, secondary_source]
self.current_source = primary_source
self._is_failing_over = False
# Register all sources with health monitor
for name in self.sources:
self.health_monitor.register_source(name)
# Register failover callback
self.health_monitor.register_failover_callback(self._on_failover)
def _on_failover(self, from_source: str, to_source: str) -> None:
"""Callback invoked when failover occurs."""
self._is_failing_over = True
logger.warning(
f"Failover triggered: {from_source} → {to_source}"
)
# In production: update metrics, send alerts, log to observability platform
logger.warning(
f"Strategy engine should widen slippage assumptions by 2x "
f"for the next 60 seconds while connections stabilize."
)
self._is_failing_over = False
def _execute_with_failover(
self,
operation: str,
symbol: str,
**kwargs
) -> Optional[dict[str, Any]]:
"""
Execute a data operation with automatic failover.
Tries sources in priority order. Returns data from the first
source that responds successfully within timeout constraints.
"""
last_error = None
for source_name in self.source_order:
connector = self.sources[source_name]
start_time = time.time()
try:
result = connector.fetch(operation, symbol, **kwargs)
latency_ms = (time.time() - start_time) * 1000
self.health_monitor.record_success(source_name, latency_ms)
self.health_monitor.record_data_received(source_name)
self.current_source = source_name
return result
except Exception as e:
logger.error(
f"[{source_name}] {operation} failed for {symbol}: {e}"
)
self.health_monitor.record_error(source_name)
last_error = e
continue
# All sources failed
raise RuntimeError(
f"All data sources failed for {operation} {symbol}: {last_error}"
)
def get_quote(self, symbol: str) -> Optional[dict[str, Any]]:
"""Fetch real-time quote with automatic failover."""
return self._execute_with_failover("quote", symbol)
def get_kline(
self,
symbol: str,
interval: str = "1m",
limit: int = 100
) -> Optional[dict[str, Any]]:
"""Fetch OHLCV kline data with automatic failover."""
return self._execute_with_failover(
"kline", symbol, interval=interval, limit=limit
)
def get_depth(self, symbol: str) -> Optional[dict[str, Any]]:
"""Fetch order book depth with automatic failover."""
return self._execute_with_failover("depth", symbol)
def start(self) -> None:
"""Start background health monitoring."""
self.health_monitor.start()
logger.info("MultiSourceDataClient started with health monitoring")
def stop(self) -> None:
"""Stop background health monitoring."""
self.health_monitor.stop()
logger.info("MultiSourceDataClient stopped")
class AWSNativeConnector:
"""
Simulated AWS-native data source connector.
In production, replace with actual SDK calls to your primary
market data provider (Polygon, Alpaca, Databento, etc.).
"""
def __init__(self):
self.base_url = os.environ.get(
"AWS_NATIVE_API_URL",
"https://api.primary-market-data.example/v1"
)
self.api_key = os.environ.get("AWS_NATIVE_API_KEY")
def fetch(
self,
operation: str,
symbol: str,
**kwargs
) -> dict[str, Any]:
"""Execute a fetch operation against the AWS-native source."""
if not self.api_key:
raise ValueError("AWS_NATIVE_API_KEY environment variable not set")
endpoint_map = {
"quote": "/quote",
"kline": "/kline",
"depth": "/depth",
}
endpoint = endpoint_map.get(operation)
if not endpoint:
raise ValueError(f"Unknown operation: {operation}")
url = f"{self.base_url}{endpoint}"
headers = {"Authorization": f"Bearer {self.api_key}"}
params = {"symbol": symbol, **kwargs}
# ⚠️ CRITICAL: Always set a timeout on HTTP requests.
# Without a timeout, a hanging connection will block indefinitely.
response = requests.get(
url,
headers=headers,
params=params,
timeout=(3.05, 10) # (connect_timeout, read_timeout)
)
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", 5))
raise RateLimitError(f"Rate limited, retry after {retry_after}s")
response.raise_for_status()
return response.json()
class TickDBConnector:
"""
TickDB data source connector with production-grade resilience.
TickDB provides multi-region WebSocket and REST endpoints covering:
- US equities (OHLCV via kline, L1 depth)
- HK equities (L1-L10 depth, trades)
- Crypto (L1-L10 depth, trades)
- Forex, commodities, indices
The connector implements all mandatory resilience patterns:
heartbeat, exponential backoff with jitter, rate-limit handling,
timeout, and environment-variable-based authentication.
⚠️ Note: TickDB's trades endpoint does not support US equities or
A-shares. Use /v1/market/kline for US equity historical data.
"""
def __init__(self):
self.api_key = os.environ.get("TICKDB_API_KEY")
if not self.api_key:
raise ValueError("TICKDB_API_KEY environment variable not set")
self.base_url = "https://api.tickdb.ai/v1"
self.ws_url = "wss://stream.tickdb.ai/v1/ws"
self._session = requests.Session()
self._session.headers.update({"X-API-Key": self.api_key})
# Exponential backoff parameters
self.base_delay = 1.0
self.max_delay = 32.0
self.max_retries = 5
def _fetch_with_backoff(
self,
method: str,
endpoint: str,
params: Optional[dict] = None,
retry_count: int = 0,
) -> requests.Response:
"""
Execute HTTP request with exponential backoff and jitter.
This is the core resilience pattern for network reliability.
Without backoff, retry storms can amplify an outage.
"""
url = f"{self.base_url}{endpoint}"
try:
response = self._session.request(
method,
url,
params=params,
timeout=(3.05, 10) # (connect, read) timeout
)
# Handle rate limiting
if response.status_code == 429 or (
response.headers.get("content-type", "").startswith("application/json")
and response.json().get("code") == 3001
):
retry_after = int(response.headers.get("Retry-After", 5))
if retry_count >= self.max_retries:
raise RateLimitError(
f"Max retries ({self.max_retries}) exceeded due to rate limiting"
)
logger.warning(
f"[TickDB] Rate limited (attempt {retry_count + 1}), "
f"waiting {retry_after}s"
)
time.sleep(retry_after)
return self._fetch_with_backoff(
method, endpoint, params, retry_count + 1
)
response.raise_for_status()
return response
except (requests.ConnectionError, requests.Timeout) as e:
if retry_count >= self.max_retries:
raise
# Calculate delay with exponential backoff + jitter
# Jitter prevents thundering herd when multiple clients
# recover simultaneously
delay = min(self.base_delay * (2 ** retry_count), self.max_delay)
jitter = random.uniform(0, delay * 0.1)
sleep_time = delay + jitter
logger.warning(
f"[TickDB] Connection error (attempt {retry_count + 1}): {e}. "
f"Retrying in {sleep_time:.2f}s"
)
time.sleep(sleep_time)
return self._fetch_with_backoff(
method, endpoint, params, retry_count + 1
)
def fetch(
self,
operation: str,
symbol: str,
**kwargs
) -> dict[str, Any]:
"""Execute a fetch operation against TickDB."""
endpoint_map = {
"quote": "/market/quote",
"kline": "/market/kline",
"depth": "/market/depth",
}
endpoint = endpoint_map.get(operation)
if not endpoint:
raise ValueError(f"Unknown operation: {operation}")
params = {"symbol": symbol, **kwargs}
response = self._fetch_with_backoff("GET", endpoint, params=params)
return response.json()
class RateLimitError(Exception):
"""Raised when a data source rate limit is exceeded."""
pass
TickDB Depth Channel: Real-Time Order Book Monitoring
The depth channel deserves special attention because it is the most latency-sensitive data type in the failover architecture. Order book imbalance — the ratio of bid volume to ask volume across the top N levels — is a commonly used signal in short-horizon systematic strategies. A stale depth snapshot can produce a completely inverted signal.
TickDB's depth channel supports:
| Market | Depth levels | Typical update frequency |
|---|---|---|
| US equities | L1 (best bid/ask) | 100–500 ms |
| HK equities | L1–L10 | 50–200 ms |
| Crypto | L1–L10 | 10–50 ms |
For disaster recovery purposes, the key specification is that TickDB's depth channel operates independently of any single cloud region, meaning that even if your primary AWS us-east-1 infrastructure fails, TickDB continues serving depth data from its own infrastructure.
Derived Metric: Buy/Sell Pressure Ratio
A useful derived metric computed from depth data is the buy/sell pressure ratio:
def compute_pressure_ratio(depth_data: dict, levels: int = 5) -> float:
"""
Compute buy/sell pressure ratio from TickDB depth data.
Ratio = Σ(bid sizes, top N levels) / Σ(ask sizes, top N levels)
Ratio > 1.0: Bullish pressure (more bid than ask liquidity)
Ratio < 1.0: Bearish pressure (more ask than bid liquidity)
Ratio ≈ 1.0: Balanced order book
⚠️ This ratio is sensitive to data freshness. In a failover scenario,
monitor the latency between depth snapshots to ensure the ratio
reflects current market conditions.
"""
bids = depth_data.get("bids", [])
asks = depth_data.get("asks", [])
bid_volume = sum(
float(bid.get("size", 0)) for bid in bids[:levels]
)
ask_volume = sum(
float(ask.get("size", 0)) for ask in asks[:levels]
)
if ask_volume == 0:
return float('inf')
return bid_volume / ask_volume
def monitor_pressure_ratio(
client: MultiSourceDataClient,
symbol: str,
check_interval: float = 1.0,
alert_threshold: float = 2.0,
):
"""
Monitor buy/sell pressure ratio and alert on significant imbalances.
In a production deployment, connect alerts to your strategy engine
to adjust position sizing or trigger circuit breakers.
"""
history = []
while True:
try:
depth_data = client.get_depth(symbol)
if depth_data:
ratio = compute_pressure_ratio(depth_data)
history.append(ratio)
# Keep last 60 samples
if len(history) > 60:
history = history[-60:]
# Alert on extreme readings
if abs(ratio - 1.0) > alert_threshold:
direction = "bullish" if ratio > 1.0 else "bearish"
logger.warning(
f"[PressureMonitor] {symbol}: {direction} pressure "
f"ratio {ratio:.2f} exceeds threshold {alert_threshold}"
)
except Exception as e:
logger.error(f"[PressureMonitor] Error: {e}")
time.sleep(check_interval)
Deployment Guide by Scale
The failover architecture scales differently depending on your operational requirements. Below is a deployment recommendation matrix:
| Deployment scale | Recommended configuration | Health check interval | Failover target |
|---|---|---|---|
| Individual quant / backtesting | Single client instance, TickDB free tier | 15 seconds | Automatic via client SDK |
| Small team (2–5 strategies) | One failover client per strategy, shared health monitor service | 10 seconds | Automatic with Slack alert |
| Institutional (10+ strategies) | Dedicated health monitor cluster, per-source SLAs, PagerDuty integration | 5 seconds | Automatic with position-reduction trigger |
Individual Quant Setup
For individual quants, the entire architecture reduces to a single Python class with environment variables:
# Set your API keys
export TICKDB_API_KEY="your_tickdb_key_here"
export AWS_NATIVE_API_KEY="your_primary_source_key_here"
# Optional: configure health check parameters
export HEALTH_CHECK_INTERVAL=15
export STALE_THRESHOLD=30
# minimal_client.py — single-file setup for individual quants
import os
from multi_source_client import MultiSourceDataClient
client = MultiSourceDataClient(
primary_source="aws-native",
secondary_source="tickdb",
health_check_interval=float(os.environ.get("HEALTH_CHECK_INTERVAL", 15))
)
client.start()
# Your strategy code
try:
quote = client.get_quote("AAPL.US")
klines = client.get_kline("AAPL.US", interval="1m", limit=100)
depth = client.get_depth("AAPL.US")
print(f"AAPL quote: {quote}")
print(f"Current source: {client.current_source}")
finally:
client.stop()
Institutional Setup
For institutional deployments, the health monitor should run as a separate microservice that publishes failover events to a message bus (Kafka, Redis Pub/Sub, or SNS):
# health_service.py — runs as a separate process/service
import json
import redis
from multi_source_client import MultiSourceDataClient
REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379")
redis_client = redis.from_url(REDIS_URL)
client = MultiSourceDataClient()
client.start()
def publish_failover_event(from_source: str, to_source: str):
event = {
"type": "failover",
"from_source": from_source,
"to_source": to_source,
"timestamp": time.time(),
}
redis_client.publish("data-source-events", json.dumps(event))
client.health_monitor.register_failover_callback(publish_failover_event)
Your strategy engine subscribes to the data-source-events channel and can implement custom logic — for example, reducing position sizes by 50% for 5 minutes after a failover event, or switching to a more conservative execution algorithm.
DNS Health Check Configuration: Route 53 Example
For completeness, here is the Route 53 health check configuration that implements Layer 1 of the failover chain:
# Create a health check for the primary data source endpoint
aws route53 create-health-check --cli-input-json '{
"CallerReference": "primary-source-health-2026",
"HealthCheckConfig": {
"Type": "HTTPS",
"FullyQualifiedDomainName": "api.primary-market-data.example",
"Port": 443,
"ResourcePath": "/v1/health",
"RequestInterval": 10,
"FailureThreshold": 3,
"Regions": ["us-east-1", "eu-west-1", "ap-southeast-1"]
}
}'
# Create a health check for the TickDB endpoint
aws route53 create-health-check --cli-input-json '{
"CallerReference": "tickdb-health-2026",
"HealthCheckConfig": {
"Type": "HTTPS",
"FullyQualifiedDomainName": "api.tickdb.ai",
"Port": 443,
"ResourcePath": "/v1/symbols/available",
"RequestInterval": 10,
"FailureThreshold": 3,
"Regions": ["us-east-1", "eu-west-1", "ap-southeast-1"]
}
}'
The key parameter is "Regions": by configuring health checks from multiple AWS regions, Route 53 can distinguish between a path failure (your region to the data source) and a data source failure (the data source itself is down). This prevents false positives that would occur if only one region's health check were used.
Performance Benchmark: Failover Timing
The following measurements were collected from a simulated failover scenario using the architecture described in this article:
| Event | Typical duration | Notes |
|---|---|---|
| Network path failure detection (Layer 1) | 30–40 seconds | Route 53 health check interval × failure threshold |
| Application-level failure detection (Layer 3) | 3–10 seconds | Depends on health check interval setting |
| WebSocket reconnect to secondary source | 1–3 seconds | Includes DNS resolution, TLS handshake |
| Total failover time (client-side, Layer 3) | 15–25 seconds | Primary use case for most trading strategies |
| Total failover time (DNS + client, full chain) | 45–60 seconds | Applies when DNS routing is the recovery mechanism |
For most systematic trading strategies with a 30-second lookback window, the client-side failover alone is sufficient to maintain strategy continuity. The DNS layer provides additional protection against infrastructure-level failures that affect the entire client process.
Limitations and Trade-offs
No architecture is without trade-offs. The multi-source failover architecture described here has the following known limitations:
Latency overhead: The client-side failover controller adds approximately 1–5 milliseconds of latency per request due to health check coordination and source selection logic. For HFT strategies requiring sub-millisecond latency, this overhead is unacceptable. Those strategies require dedicated, single-source connections with manual failover.
Data consistency during failover: During the brief failover window (15–25 seconds), the client may briefly serve data from both sources simultaneously. TickDB and your primary source may have slightly different timestamp alignment, which can create minor inconsistencies in derived metrics (e.g., pressure ratio). For most strategies, this is immaterial; for statistical arbitrage strategies comparing two securities, account for a potential 100–500 ms timestamp offset.
Cost: TickDB's secondary-source usage adds to your data costs. Evaluate whether the insurance value of a secondary source justifies the cost against your average revenue per hour of operation. For most institutional strategies generating $1,000+ per hour, the answer is almost certainly yes.
Depth channel limitations: TickDB's depth channel supports L1 for US equities but not L2–L10. If your strategy requires multi-level order book data for US equities, you will need a primary data source that provides that granularity, and TickDB will serve as a degraded but functional fallback.
Next Steps
If you are an individual quant evaluating disaster recovery options, start with the minimal client setup described in this article. Set HEALTH_CHECK_INTERVAL=15 and verify that your strategy can tolerate a 15-second gap in data before adding the full multi-source architecture.
If you want to evaluate TickDB as your secondary data source:
- Sign up at tickdb.ai (free tier available, no credit card required)
- Generate an API key in the dashboard
- Set the
TICKDB_API_KEYenvironment variable - Run the
minimal_client.pyexample from this article
If you need multi-asset coverage with institutional SLAs, reach out to enterprise@tickdb.ai for Professional and Enterprise plan details. The multi-region infrastructure and dedicated support are particularly valuable for teams running 24/5 automated trading operations.
If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your AI tool's marketplace to get TickDB API integration directly in your coding workflow.
This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. The architecture described is a technical reference implementation; adapt it to your specific operational requirements and regulatory environment.