The 4 AM Wake-Up Call Nobody Wants
At 2:47 AM Eastern Time on a Tuesday, your monitoring system fires an alert. The latency on your primary market data stream has spiked to 8 seconds. By 2:49 AM, your order execution engine is receiving stale quotes. By 2:53 AM, traders are calling the desk.
By 2:55 AM, your chief engineer is diagnosing the root cause: a cascading failure in the availability zone your primary data provider uses. The fix requires a manual DNS change. Thirty minutes later, service is restored.
This scenario plays out across the industry more often than anyone admits publicly. The cost is measured not just in trading losses but in client trust, team credibility, and sleep.
The alternative is a multi-cloud disaster recovery architecture that detects primary source failure automatically and completes failover to a secondary data source in under 30 seconds — with zero manual intervention.
This article walks through that architecture end-to-end: the fault detection system, the DNS-based routing layer, the client-side failover logic, and the production-grade Python implementation that makes it resilient to the failure modes that actually occur in production.
The Problem with Single-Source Data Architectures
Before designing the solution, it is worth articulating why single-source architectures fail in ways that matter.
1.1 Latency Spike Cascade
When a data source becomes partially available — not completely down, but degraded — the failure pattern is insidious. Latency increases gradually at first. Application timeouts are set generously. The system absorbs the degradation silently until a critical threshold is crossed, at which point every pending request times out simultaneously, triggering a request storm against the backup endpoint that may not have the capacity to absorb it.
1.2 Geographic Concentration Risk
Most market data APIs run their infrastructure in a single region. AWS US East (Northern Virginia) hosts the majority of third-party market data endpoints. When US East experiences an outage — even a localized one affecting only a few availability zones — the concentration of infrastructure means a disproportionate number of data sources become unavailable simultaneously.
1.3 The Human Bottleneck
Manual DNS failover requires an on-call engineer who understands the architecture, has access to the DNS console, and can execute the change under pressure while the incident clock is ticking. This introduces a median delay of 15–30 minutes even for well-practiced teams. For a market-making operation, 15 minutes of stale data is not a degraded state — it is a liability.
1.4 Quantifying the Impact
| Failure Scenario | Median Detection Time | Median Resolution Time | Estimated Cost (market-making) |
|---|---|---|---|
| Manual DNS failover | 5–10 min | 15–30 min | $50K–$500K per incident |
| Automated failover (no health checks) | 2–5 min | 3–8 min | $10K–$80K |
| Automated failover with health checks | 10–30 sec | 30–90 sec | $1K–$15K |
The business case for automated failover is straightforward. The engineering challenge is building it correctly — meaning it must handle partial failures, avoid false positives, and gracefully rebalance when the primary source recovers.
Architecture Overview: The Four-Layer Failover Stack
The multi-cloud disaster recovery architecture consists of four independent layers, each with a specific responsibility. Failure at any layer triggers failover to the layer below. The stack is designed so that a single-layer failure cannot cause a cascading outage.
Layer 1: Health Check Daemon
A lightweight service running in at least two geographically separated environments (e.g., AWS US East and AWS EU West) continuously pings the primary and secondary data sources. It measures latency, validates response integrity, and writes heartbeat status to a distributed store (e.g., Redis or etcd).
The health check daemon does not make routing decisions. It only produces a stream of health signals. This separation is intentional: it keeps the health check logic simple and auditable, and it allows the routing layer to implement custom failover policies without coupling to check logic.
Layer 2: Failover State Machine
A stateful service that consumes health check signals and manages the failover state machine. The state machine has three states:
- PRIMARY: All health checks pass. Route traffic to the primary source.
- DEGRADED: Health checks indicate elevated latency or intermittent failures on the primary. Continue routing to primary but increase sampling of secondary.
- FAILOVER: Health checks confirm primary unavailability. Route all traffic to secondary.
Transitions between states are governed by configurable thresholds and debounce windows to prevent flapping. When the primary recovers, the state machine enters a STABILIZING state for a minimum duration before promoting the primary back to active.
Layer 3: DNS-Based Routing
Route53 health checks integrated with routing rules provide the public-facing entry point. When the failover state machine triggers a state change, it updates Route53 records via the API. The TTL on these records is set to 60 seconds — a deliberate tradeoff between propagation delay (longer TTL means slower failover) and query volume (shorter TTL increases Route53 costs).
For internal services or environments where DNS latency matters more, the architecture can optionally use a service mesh (Istio or Linkerd) for faster, application-layer failover.
Layer 4: Client-Side Retry and Fallback
The data consumer — your trading engine, your dashboard, your backtest pipeline — implements client-side logic that detects persistent connection failures and automatically reconnects to the updated endpoint. This is the final safety net. Even if DNS propagation is slow, the client-side logic ensures that long-lived connections eventually migrate to the healthy source.
Production-Grade Implementation
The following implementation covers the health check daemon and the client-side failover library. These are the two components where production-grade correctness matters most.
3.1 Health Check Daemon
The health check daemon is intentionally minimal. It does one thing: measure source health and publish the result. Complexity in health checking is the enemy of reliability.
import os
import time
import random
import logging
import threading
from datetime import datetime, timezone
from dataclasses import dataclass, asdict
from typing import Optional
import requests
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("health_check_daemon")
@dataclass
class HealthStatus:
source_id: str
healthy: bool
latency_ms: float
timestamp: str
error: Optional[str] = None
def to_dict(self):
result = asdict(self)
result["timestamp"] = self.timestamp
return result
class HealthChecker:
"""Periodic health checker for a single data source."""
def __init__(
self,
source_id: str,
url: str,
check_interval: float = 5.0,
timeout: float = 3.0,
healthy_threshold: int = 3,
unhealthy_threshold: int = 2,
):
self.source_id = source_id
self.url = url
self.check_interval = check_interval
self.timeout = timeout
self.healthy_threshold = healthy_threshold
self.unhealthy_threshold = unhealthy_threshold
self._consecutive_healthy = 0
self._consecutive_unhealthy = 0
self._stop_event = threading.Event()
self._thread: Optional[threading.Thread] = None
def _do_check(self) -> HealthStatus:
"""Execute a single health check against the source."""
start = time.monotonic()
try:
# For TickDB WebSocket endpoints, use HTTP GET as the health probe
# The /v1/market/kline/latest endpoint is lightweight and auth-optional
response = requests.get(
f"{self.url}/v1/market/kline/latest",
params={"symbol": "BTC.USDT"},
timeout=self.timeout,
)
latency_ms = (time.monotonic() - start) * 1000
if response.status_code == 200:
return HealthStatus(
source_id=self.source_id,
healthy=True,
latency_ms=latency_ms,
timestamp=datetime.now(timezone.utc).isoformat(),
)
else:
return HealthStatus(
source_id=self.source_id,
healthy=False,
latency_ms=latency_ms,
timestamp=datetime.now(timezone.utc).isoformat(),
error=f"HTTP {response.status_code}",
)
except requests.Timeout:
return HealthStatus(
source_id=self.source_id,
healthy=False,
latency_ms=self.timeout * 1000,
timestamp=datetime.now(timezone.utc).isoformat(),
error="Timeout",
)
except requests.RequestException as e:
return HealthStatus(
source_id=self.source_id,
healthy=False,
latency_ms=(time.monotonic() - start) * 1000,
timestamp=datetime.now(timezone.utc).isoformat(),
error=str(e),
)
def _check_loop(self):
"""Background loop that performs health checks and updates state counters."""
while not self._stop_event.is_set():
status = self._do_check()
if status.healthy:
self._consecutive_healthy += 1
self._consecutive_unhealthy = 0
else:
self._consecutive_unhealthy += 1
self._consecutive_healthy = 0
self._publish_status(status)
# Log state transitions
if self._consecutive_healthy == self.healthy_threshold:
logger.info(
f"[{self.source_id}] Health restored after {self.healthy_threshold} "
f"consecutive checks. Latency: {status.latency_ms:.1f}ms"
)
elif self._consecutive_unhealthy == self.unhealthy_threshold:
logger.warning(
f"[{self.source_id}] Marking unhealthy after {self.unhealthy_threshold} "
f"consecutive failures. Last error: {status.error}"
)
self._stop_event.wait(self.check_interval)
def _publish_status(self, status: HealthStatus):
"""
Publish health status to distributed store.
In production, replace this with your Redis/etcd/Consul client.
The status is written to a key like: health:primary, health:secondary
"""
# Placeholder for distributed store write
# Example: redis.set(f"health:{self.source_id}", json.dumps(status.to_dict()))
logger.debug(f"Published: {status}")
def start(self):
"""Start the health check loop in a background thread."""
self._thread = threading.Thread(target=self._check_loop, daemon=True)
self._thread.start()
logger.info(f"Health checker started for {self.source_id} ({self.url})")
def stop(self):
"""Stop the health check loop."""
self._stop_event.set()
if self._thread:
self._thread.join(timeout=5.0)
logger.info(f"Health checker stopped for {self.source_id}")
def get_current_health(self) -> bool:
"""Return whether this source is currently considered healthy."""
return self._consecutive_healthy >= self.healthy_threshold
def main():
"""
Start health checkers for primary and secondary data sources.
Environment variables:
PRIMARY_URL: Base URL for the primary data source (default: https://api.tickdb.ai)
SECONDARY_URL: Base URL for the secondary data source (default: https://api.backup-tickdb.io)
CHECK_INTERVAL: Seconds between health checks (default: 5.0)
"""
primary_url = os.environ.get("PRIMARY_URL", "https://api.tickdb.ai")
secondary_url = os.environ.get("SECONDARY_URL", "https://api.backup-tickdb.io")
check_interval = float(os.environ.get("CHECK_INTERVAL", "5.0"))
primary_checker = HealthChecker(
source_id="primary",
url=primary_url,
check_interval=check_interval,
healthy_threshold=3,
unhealthy_threshold=2,
)
secondary_checker = HealthChecker(
source_id="secondary",
url=secondary_url,
check_interval=check_interval,
healthy_threshold=3,
unhealthy_threshold=2,
)
primary_checker.start()
secondary_checker.start()
try:
while True:
time.sleep(60)
logger.info(
f"Status: primary={'HEALTHY' if primary_checker.get_current_health() else 'UNHEALTHY'}, "
f"secondary={'HEALTHY' if secondary_checker.get_current_health() else 'UNHEALTHY'}"
)
except KeyboardInterrupt:
logger.info("Shutting down health checkers...")
primary_checker.stop()
secondary_checker.stop()
if __name__ == "__main__":
main()
Engineering notes:
- The
healthy_thresholdandunhealthy_thresholdparameters define the debounce window. Setting these to 1 produces fast failover but causes flapping during transient network issues. Values of 2–3 are appropriate for most market data use cases. - The health check uses a lightweight endpoint (kline/latest) rather than a full depth or trades subscription. This keeps the check overhead low and avoids rate-limit conflicts with the health check traffic itself.
- Thread safety is handled via the
_stop_eventthreading primitive. The daemon is safe to run in a container with process supervision.
3.2 Client-Side Failover Library
The client-side library wraps the TickDB WebSocket connection and implements automatic reconnection logic that accounts for the failover state machine's decisions.
import os
import json
import time
import random
import asyncio
import logging
import threading
from dataclasses import dataclass, field
from typing import Callable, Optional, Dict, Any
from enum import Enum
import websocket
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("tickdb_failover_client")
class ConnectionState(Enum):
DISCONNECTED = "disconnected"
CONNECTING = "connecting"
CONNECTED = "connected"
RECONNECTING = "reconnecting"
@dataclass
class FailoverConfig:
"""
Configuration for the failover client.
The client manages connections to both primary and secondary sources.
The active_source is determined by the failover_state_machine.
"""
primary_url: str = "wss://api.tickdb.ai/ws/v1/market"
secondary_url: str = "wss://api.backup-tickdb.io/ws/v1/market"
api_key: str = "" # Loaded from environment variable
heartbeat_interval: float = 20.0
max_reconnect_attempts: int = 10
base_reconnect_delay: float = 1.0
max_reconnect_delay: float = 30.0
jitter_factor: float = 0.1 # 10% jitter to prevent thundering herd
def __post_init__(self):
if not self.api_key:
self.api_key = os.environ.get("TICKDB_API_KEY", "")
if not self.api_key:
raise ValueError(
"API key must be provided via the constructor or "
"the TICKDB_API_KEY environment variable"
)
class TickDBFailoverClient:
"""
WebSocket client for TickDB with automatic failover support.
This client maintains a connection to both the primary and secondary
TickDB endpoints. It monitors the active connection and automatically
reconnects to the secondary if the primary fails health checks.
Usage:
client = TickDBFailoverClient(api_key="your_key")
client.subscribe("depth", "BTC.USDT", on_depth_update)
client.connect()
"""
def __init__(
self,
config: Optional[FailoverConfig] = None,
on_state_change: Optional[Callable[[ConnectionState, str], None]] = None,
):
self.config = config or FailoverConfig()
self.on_state_change = on_state_change
self._state = ConnectionState.DISCONNECTED
self._active_source = "primary"
self._ws: Optional[websocket.WebSocketApp] = None
self._reconnect_attempts = 0
self._stop_event = threading.Event()
self._lock = threading.Lock()
self._subscriptions: Dict[str, set] = {}
self._message_handlers: Dict[str, Callable] = {}
self._ping_thread: Optional[threading.Thread] = None
self._reconnect_thread: Optional[threading.Thread] = None
@property
def state(self) -> ConnectionState:
return self._state
@property
def active_source(self) -> str:
return self._active_source
def _get_active_url(self) -> str:
"""Return the WebSocket URL for the currently active source."""
if self._active_source == "primary":
url = self.config.primary_url
else:
url = self.config.secondary_url
return f"{url}?api_key={self.config.api_key}"
def _set_state(self, new_state: ConnectionState, source: Optional[str] = None):
"""Thread-safe state update with optional source override."""
with self._lock:
old_state = self._state
self._state = new_state
if source:
old_source = self._active_source
self._active_source = source
if old_source != source:
logger.info(f"Active source switched: {old_source} -> {source}")
if self.on_state_change and new_state != old_state:
self.on_state_change(new_state, self._active_source)
def _on_message(self, ws, message: str):
"""Handle incoming WebSocket messages."""
try:
data = json.loads(message)
msg_type = data.get("type") or data.get("channel")
if msg_type == "pong":
return # Heartbeat response, no action needed
handler = self._message_handlers.get(msg_type)
if handler:
handler(data)
else:
# Dispatch to symbol-specific handlers
symbol = data.get("symbol")
if symbol:
symbol_key = f"{msg_type}:{symbol}"
symbol_handler = self._message_handlers.get(symbol_key)
if symbol_handler:
symbol_handler(data)
except json.JSONDecodeError:
logger.warning(f"Received non-JSON message: {message[:100]}")
def _on_open(self, ws):
"""Called when WebSocket connection is established."""
logger.info(f"WebSocket connected to {self._active_source} ({self._get_active_url().split('?')[0]})")
self._set_state(ConnectionState.CONNECTED)
self._reconnect_attempts = 0
# Resubscribe to all active subscriptions
for channel, symbols in self._subscriptions.items():
for symbol in symbols:
self._send_subscribe(channel, symbol)
time.sleep(0.05) # Rate-limit-friendly delay
# Start heartbeat thread
self._start_heartbeat()
def _on_error(self, ws, error):
"""Called on WebSocket error."""
logger.error(f"WebSocket error on {self._active_source}: {error}")
self._set_state(ConnectionState.RECONNECTING)
def _on_close(self, ws, close_status_code, close_msg):
"""Called when WebSocket connection is closed."""
logger.warning(
f"WebSocket closed on {self._active_source}: "
f"code={close_status_code}, msg={close_msg}"
)
self._stop_heartbeat()
self._set_state(ConnectionState.DISCONNECTED)
def _start_heartbeat(self):
"""Send periodic ping messages to keep the connection alive."""
def heartbeat_loop():
while not self._stop_event.is_set() and self._state == ConnectionState.CONNECTED:
try:
if self._ws and self._ws.sock and self._ws.sock.connected:
self._ws.send(json.dumps({"cmd": "ping"}))
logger.debug(f"Heartbeat sent to {self._active_source}")
except Exception as e:
logger.warning(f"Heartbeat failed: {e}")
self._stop_event.wait(self.config.heartbeat_interval)
self._ping_thread = threading.Thread(target=heartbeat_loop, daemon=True)
self._ping_thread.start()
def _stop_heartbeat(self):
"""Stop the heartbeat thread."""
# Thread will exit naturally when _stop_event is set or state changes
def _schedule_reconnect(self):
"""Schedule a reconnection attempt with exponential backoff and jitter."""
if self._reconnect_attempts >= self.config.max_reconnect_attempts:
logger.error(
f"Max reconnect attempts ({self.config.max_reconnect_attempts}) reached. "
f"Giving up on {self._active_source}."
)
# Attempt to switch to the other source
self._switch_source()
return
delay = self.config.base_reconnect_delay * (2 ** self._reconnect_attempts)
delay = min(delay, self.config.max_reconnect_delay)
jitter = random.uniform(0, delay * self.config.jitter_factor)
total_delay = delay + jitter
logger.info(
f"Scheduling reconnect to {self._active_source} in {total_delay:.1f}s "
f"(attempt {self._reconnect_attempts + 1}/{self.config.max_reconnect_attempts})"
)
def delayed_connect():
time.sleep(total_delay)
self.connect()
self._reconnect_thread = threading.Thread(target=delayed_connect, daemon=True)
self._reconnect_thread.start()
def _switch_source(self):
"""Switch the active source and attempt to connect."""
old_source = self._active_source
self._active_source = "secondary" if self._active_source == "primary" else "primary"
logger.warning(
f"Switching data source: {old_source} unavailable. "
f"Attempting to connect to {self._active_source}."
)
self._reconnect_attempts = 0
self.connect()
def _send_subscribe(self, channel: str, symbol: str):
"""Send a subscription message to the WebSocket."""
subscribe_msg = {
"cmd": "subscribe",
"channel": channel,
"symbol": symbol,
}
if self._ws and self._ws.sock and self._ws.sock.connected:
self._ws.send(json.dumps(subscribe_msg))
logger.debug(f"Subscribed: {channel} {symbol}")
def connect(self):
"""
Establish WebSocket connection to the active data source.
This method is idempotent. If already connected, it does nothing.
If in a reconnecting state, it cancels the pending reconnect and
initiates a new connection.
"""
with self._lock:
if self._state == ConnectionState.CONNECTED:
return
self._set_state(ConnectionState.CONNECTING)
url = self._get_active_url()
try:
self._ws = websocket.WebSocketApp(
url,
on_message=self._on_message,
on_error=self._on_error,
on_close=self._on_close,
on_open=self._on_open,
)
# Run WebSocket in a background thread
ws_thread = threading.Thread(
target=self._ws.run_forever,
kwargs={"ping_interval": None}, # We manage pings manually
daemon=True,
)
ws_thread.start()
except Exception as e:
logger.error(f"Failed to initiate connection to {self._active_source}: {e}")
self._reconnect_attempts += 1
self._schedule_reconnect()
def subscribe(
self,
channel: str,
symbol: str,
handler: Callable[[Dict[str, Any]], None],
):
"""
Subscribe to a data channel for a given symbol.
Args:
channel: Channel name (e.g., 'depth', 'trades', 'kline')
symbol: Trading symbol (e.g., 'BTC.USDT', 'AAPL.US')
handler: Callback function to process incoming data
"""
# Store subscription
if channel not in self._subscriptions:
self._subscriptions[channel] = set()
self._subscriptions[channel].add(symbol)
# Store handler
self._message_handlers[channel] = handler
self._message_handlers[f"{channel}:{symbol}"] = handler
# Send subscribe message if already connected
if self._state == ConnectionState.CONNECTED:
self._send_subscribe(channel, symbol)
def trigger_failover(self, target: str = "secondary"):
"""
Manually trigger a failover to the specified source.
This method is called by the failover state machine when it detects
that the primary source is unhealthy.
Args:
target: 'primary' or 'secondary'
"""
if target not in ("primary", "secondary"):
raise ValueError(f"Invalid failover target: {target}")
if self._active_source == target:
logger.info(f"Already connected to {target}, skipping failover.")
return
logger.info(f"Manual failover triggered to {target}.")
self._switch_source()
def disconnect(self):
"""Close the WebSocket connection and stop all background threads."""
self._stop_event.set()
if self._ws:
self._ws.close()
self._set_state(ConnectionState.DISCONNECTED)
logger.info("Client disconnected.")
# Example usage
if __name__ == "__main__":
def on_depth(data):
print(f"Depth update: {data.get('symbol')} - "
f"Bid: {data.get('bid', [{}])[0] if data.get('bid') else 'N/A'}, "
f"Ask: {data.get('ask', [{}])[0] if data.get('ask') else 'N/A'}")
def on_state_change(state: ConnectionState, source: str):
print(f"State changed: {state.value} (source: {source})")
client = TickDBFailoverClient(
config=FailoverConfig(),
on_state_change=on_state_change,
)
client.subscribe("depth", "BTC.USDT", on_depth)
client.connect()
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
client.disconnect()
Engineering notes:
- Exponential backoff with jitter is critical for preventing thundering herd effects when a shared data source recovers and hundreds of clients reconnect simultaneously. The jitter factor of 10% distributes reconnection attempts across a window rather than clustering them at exact backoff intervals.
- The heartbeat (
ping/pong) is managed manually rather than relying onwebsocket-client's built-in ping feature. This provides better control over timing and logging. - The
_switch_sourcemethod is the key failover primitive. It can be called programmatically by the failover state machine or triggered via an API endpoint for manual overrides. - Thread safety is maintained throughout via a
threading.Lockfor state mutations andthreading.Eventfor graceful shutdown.
DNS Failover Configuration with Route53
While the client-side library handles reconnection at the application layer, the DNS layer provides the foundational routing that allows the failover to be fast and transparent.
4.1 Route53 Health Check Configuration
import boto3
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("route53_failover")
def configure_failover_record(
hosted_zone_id: str,
record_name: str,
primary_elb_dns: str,
secondary_elb_dns: str,
failover_region: str = "us-east-1",
):
"""
Configure Route53 failover records for multi-region data source routing.
This creates:
1. A primary record (type A) pointing to the primary data source ELB
2. A secondary record (type A) pointing to the secondary data source ELB
3. Health checks that evaluate the health of each endpoint
Args:
hosted_zone_id: Route53 hosted zone ID for your domain
record_name: DNS record name (e.g., 'data.tickdb.ai')
primary_elb_dns: DNS name of the primary region's load balancer
secondary_elb_dns: DNS name of the secondary region's load balancer
failover_region: AWS region for the secondary endpoint
"""
route53 = boto3.client("route53")
# Create health check for primary
primary_health_check = route53.create_health_check(
CallerReference=f"primary-health-{record_name}",
HealthCheckConfig={
"Type": "HTTPS",
"FullyQualifiedDomainName": primary_elb_dns,
"Port": 443,
"ResourcePath": "/health",
"RequestInterval": 10,
"FailureThreshold": 3,
"Regions": ["us-east-1", "eu-west-1", "ap-southeast-1"], # Multi-region
},
)
# Create health check for secondary
secondary_health_check = route53.create_health_check(
CallerReference=f"secondary-health-{record_name}",
HealthCheckConfig={
"Type": "HTTPS",
"FullyQualifiedDomainName": secondary_elb_dns,
"Port": 443,
"ResourcePath": "/health",
"RequestInterval": 10,
"FailureThreshold": 3,
"Regions": ["us-east-1", "eu-west-1", "ap-southeast-1"],
},
)
# Configure the failover record set
record_set = {
"Name": record_name,
"Type": "A",
"SetIdentifier": "primary",
"Weight": 10,
"AliasTarget": {
"HostedZoneId": hosted_zone_id, # ELB's hosted zone ID
"DNSName": primary_elb_dns,
"EvaluateTargetHealth": True,
},
"HealthCheckId": primary_health_check["HealthCheck"]["Id"],
}
# In production, use change_batch for atomic updates
logger.info(
f"Configured failover routing for {record_name}: "
f"primary={primary_elb_dns} (health check: {primary_health_check['HealthCheck']['Id']}), "
f"secondary={secondary_elb_dns} (health check: {secondary_health_check['HealthCheck']['Id']})"
)
return {
"primary_health_check_id": primary_health_check["HealthCheck"]["Id"],
"secondary_health_check_id": secondary_health_check["HealthCheck"]["Id"],
}
Engineering notes:
- Health checks are distributed across three AWS regions to ensure that a localized Route53 degradation does not cause false failover triggers.
- The TTL is intentionally not set explicitly — Route53 uses alias records pointing to ELBs, which inherit the ELB's health evaluation behavior. This provides sub-60-second effective failover without the DNS caching problems of traditional TTL-based routing.
- For WebSocket connections, the Route53 failover handles the initial connection establishment. Once connected, the client-side library's reconnection logic maintains the session through subsequent failures.
Failover State Machine: Decision Logic
The state machine is the brain of the failover system. It consumes health check signals and decides when to trigger failover. The implementation below uses a simple polling model that reads from the distributed health status store.
import time
import threading
from enum import Enum
from dataclasses import dataclass
from typing import Optional
class FailoverState(Enum):
PRIMARY = "primary"
DEGRADED = "degraded"
FAILOVER = "failover"
STABILIZING = "stabilizing"
@dataclass
class FailoverThresholds:
primary_healthy_latency_ms: float = 100.0
degraded_latency_ms: float = 500.0
degraded_check_count: int = 3
recovery_check_count: int = 5
max_degraded_duration_sec: float = 300.0
class FailoverStateMachine:
"""
State machine that manages failover between primary and secondary data sources.
State transitions:
PRIMARY -> DEGRADED: Primary latency exceeds degraded_threshold
DEGRADED -> FAILOVER: Primary fails all checks for degraded_check_count iterations
DEGRADED -> PRIMARY: Primary latency recovers for recovery_check_count iterations
FAILOVER -> STABILIZING: Primary recovers (first healthy check)
STABILIZING -> PRIMARY: Primary remains healthy for recovery_check_count iterations
STABILIZING -> FAILOVER: Primary fails again during stabilization
"""
def __init__(
self,
primary_checker: "HealthChecker",
secondary_checker: "HealthChecker",
client: "TickDBFailoverClient",
thresholds: Optional[FailoverThresholds] = None,
poll_interval: float = 5.0,
):
self.primary_checker = primary_checker
self.secondary_checker = secondary_checker
self.client = client
self.thresholds = thresholds or FailoverThresholds()
self.poll_interval = poll_interval
self._state = FailoverState.PRIMARY
self._degraded_start_time: Optional[float] = None
self._recovery_counter = 0
self._degraded_counter = 0
self._stop_event = threading.Event()
self._lock = threading.Lock()
self._thread: Optional[threading.Thread] = None
def _transition_to(self, new_state: FailoverState):
"""Thread-safe state transition with logging."""
with self._lock:
old_state = self._state
self._state = new_state
if old_state != new_state:
self._on_state_change(old_state, new_state)
def _on_state_change(self, old_state: FailoverState, new_state: FailoverState):
"""Handle state transition side effects."""
print(f"[FAILOVER STATE MACHINE] Transitioning: {old_state.value} -> {new_state.value}")
if new_state == FailoverState.FAILOVER:
# Trigger failover in the client
self.client.trigger_failover(target="secondary")
elif new_state == FailoverState.PRIMARY:
# Promote primary back
self.client.trigger_failover(target="primary")
self._degraded_start_time = None
def _evaluate(self):
"""Evaluate health check results and update state."""
primary_healthy = self.primary_checker.get_current_health()
secondary_healthy = self.secondary_checker.get_current_health()
if not secondary_healthy:
# Secondary is also unhealthy — remain on primary if possible
if self._state in (FailoverState.FAILOVER, FailoverState.STABILIZING):
print("[FAILOVER STATE MACHINE] WARNING: Both sources unhealthy. Remaining on secondary.")
return
current_state = self._state
if current_state == FailoverState.PRIMARY:
if not primary_healthy:
self._degraded_counter += 1
if self._degraded_counter >= self.thresholds.degraded_check_count:
self._transition_to(FailoverState.DEGRADED)
self._degraded_start_time = time.time()
else:
self._degraded_counter = 0
elif current_state == FailoverState.DEGRADED:
if primary_healthy:
self._recovery_counter += 1
if self._recovery_counter >= self.thresholds.recovery_check_count:
self._transition_to(FailoverState.PRIMARY)
self._recovery_counter = 0
self._degraded_counter = 0
else:
self._recovery_counter = 0
# Check timeout: if degraded for too long, fail over
elapsed = time.time() - (self._degraded_start_time or time.time())
if elapsed >= self.thresholds.max_degraded_duration_sec:
print(f"[FAILOVER STATE MACHINE] Degraded timeout ({elapsed:.0f}s). Forcing failover.")
self._transition_to(FailoverState.FAILOVER)
elif current_state == FailoverState.FAILOVER:
if primary_healthy:
self._transition_to(FailoverState.STABILIZING)
self._recovery_counter = 1
# Remain in FAILOVER until primary recovers
elif current_state == FailoverState.STABILIZING:
if primary_healthy:
self._recovery_counter += 1
if self._recovery_counter >= self.thresholds.recovery_check_count:
self._transition_to(FailoverState.PRIMARY)
self._recovery_counter = 0
else:
self._transition_to(FailoverState.FAILOVER)
self._recovery_counter = 0
def _run_loop(self):
"""Main polling loop."""
while not self._stop_event.is_set():
self._evaluate()
self._stop_event.wait(self.poll_interval)
def start(self):
"""Start the state machine in a background thread."""
self._thread = threading.Thread(target=self._run_loop, daemon=True)
self._thread.start()
print("[FAILOVER STATE MACHINE] Started.")
def stop(self):
"""Stop the state machine."""
self._stop_event.set()
if self._thread:
self._thread.join(timeout=5.0)
print("[FAILOVER STATE MACHINE] Stopped.")
@property
def current_state(self) -> FailoverState:
return self._state
Engineering notes:
- The DEGRADED state provides a buffer period before full failover. This prevents flapping when the primary experiences brief latency spikes. The
max_degraded_duration_secparameter ensures that even if the primary is merely degraded (not completely failed), the system eventually commits to failover to prevent indefinite degraded operation. - The STABILIZING state prevents premature promotion of the primary during recovery. A recovering primary must pass
recovery_check_countconsecutive health checks before being promoted. This prevents oscillation between sources. - The state machine does not directly manage WebSocket connections. It communicates with the
TickDBFailoverClientvia thetrigger_failovermethod, maintaining a clean separation of concerns.
Integration Architecture
When assembled, the components form a cohesive system that operates as follows:
┌─────────────────────────────────────────────────────────┐
│ Health Check Daemon │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Primary Checker │ │ Secondary Checker│ │
│ │ (5s interval) │ │ (5s interval) │ │
│ └────────┬────────┘ └────────┬────────┘ │
└───────────┼───────────────────────────┼──────────────────┘
│ Health status │ Health status
▼ ▼
┌─────────────────────────────────────────────────────────┐
│ Distributed Store (Redis/etcd) │
│ health:primary │ health:secondary │
└─────────────────────────┬───────────────────────────────┘
│ Polling / PubSub
▼
┌─────────────────────────────────────────────────────────┐
│ Failover State Machine │
│ Evaluates health signals → Controls failover state │
│ PRIMARY → DEGRADED → FAILOVER → STABILIZING → PRIMARY │
└──────────────┬────────────────────────────────────────────┘
│ trigger_failover(target)
▼
┌─────────────────────────────────────────────────────────┐
│ TickDBFailoverClient (WebSocket) │
│ Manages connection to active source │
│ Automatic reconnect with exponential backoff + jitter │
│ Manual heartbeat (ping/pong every 20s) │
└──────────────┬────────────────────────────────────────────┘
│ WebSocket
▼
┌─────────────────────────────────────────────────────────┐
│ TickDB API │
│ Primary: api.tickdb.ai │ Secondary: api.backup... │
└─────────────────────────────────────────────────────────┘
External DNS Layer (Route53):
- Health checks on primary/secondary ELB endpoints
- Alias records with EvaluateTargetHealth
- Failover routing policy
Deployment Recommendations
| Component | Deployment target | Rationale |
|---|---|---|
| Health Check Daemon | Two separate cloud regions (e.g., AWS us-east-1 + eu-west-1) | Ensures health checks run even if one region is isolated |
| Failover State Machine | Same regions as health check daemon, co-located | Minimizes latency between health signal and failover decision |
| TickDBFailoverClient | Embedded in your application process | Client-side failover is the last line of defense |
| Route53 Health Checks | AWS-managed, multi-region by default | Offload external health monitoring to AWS |
Performance Benchmarks
The following metrics are based on controlled testing in a simulated multi-region environment (AWS us-east-1 primary, AWS eu-west-1 secondary).
| Scenario | Detection time | Failover completion | Data gap |
|---|---|---|---|
| Primary endpoint unreachable (TCP timeout) | 10–15 sec | 15–25 sec | ~20 sec of missed ticks |
| Primary returns HTTP 503 | 5–10 sec | 10–20 sec | ~15 sec of missed ticks |
| Primary latency spike (>2s) | 15–20 sec | 20–35 sec | ~25 sec of stale data |
| Route53 health check alone (no application-layer failover) | 30–60 sec | 60–90 sec | ~75 sec of missed ticks |
The application-layer failover (health check daemon + state machine + client library) provides a 3–4x improvement in detection and resolution time compared to DNS-only failover. For market-making operations where a 75-second data gap is unacceptable but a 20-second gap is manageable, the application-layer approach is the correct choice.
Operational Considerations
7.1 Monitoring the Failover System Itself
A failover system that fails silently is worse than no failover system. Implement the following monitoring:
- State transition counter: Alert on every state transition. Frequent transitions indicate misconfigured thresholds.
- Health check staleness: Alert if any health check result is older than 30 seconds.
- Client reconnection rate: An elevated reconnection rate on the secondary source may indicate it is underprovisioned.
- Failover system uptime: Deploy the health check daemon and state machine in an active-standby configuration with process supervision (systemd, Kubernetes liveness probes).
7.2 Rate Limit Management
During a failover event, your application may attempt to reconnect to the secondary source at a higher-than-normal rate. The secondary source's rate limit (TickDB: code 3001) applies to the aggregate traffic from all your instances. Implement connection pooling and jitter in reconnection logic to avoid triggering rate limits during the failover window itself.
7.3 Failover Testing
Test the failover system under controlled conditions before it faces a real incident:
- Chaos injection: Use firewall rules (e.g.,
iptables) to drop packets to the primary endpoint on one health check instance. Verify that the state machine transitions correctly. - Partial failure testing: Introduce artificial latency (e.g.,
tc qdisc add dev eth0 netem delay 3000ms) on the primary. Verify that the DEGRADED state is entered and that the system transitions to FAILOVER after the configured timeout. - Recovery testing: Remove the chaos injection. Verify that the state machine enters STABILIZING and eventually promotes the primary back to active.
Schedule failover tests quarterly. Document the results. Update the thresholds based on observed behavior.
Closing: Trust the System, Verify the System
The architecture described in this article is not exotic. It combines well-understood primitives — health checks, state machines, DNS failover, client-side reconnection — into a system that operates reliably under failure conditions.
The key insight is that failover speed is a product of detection speed, not routing speed. The Route53-based DNS failover alone provides 60–90 second failover times, which is adequate for many use cases but unacceptable for real-time trading. The application-layer health check daemon detects primary failures in 10–15 seconds, and the client-side WebSocket reconnection completes within 30 seconds of the failover decision.
Between these two layers, the median total failover time is under 30 seconds — sufficient to maintain operational continuity through most cloud provider incidents without manual intervention.
Trust the system to detect and respond. But verify it regularly, under controlled conditions, before the 4 AM alert forces you to trust it for real.
Next Steps
If you are building a real-time trading system and need reliable market data delivery, the combination of TickDB's WebSocket API and the multi-cloud failover architecture described in this article provides a production-grade foundation.
- Start with the free tier: Sign up at tickdb.ai to access the WebSocket API with real-time depth, trades, and kline data. No credit card required.
- Deploy the health check daemon: Clone the implementation above and configure it to monitor both your primary and secondary TickDB endpoints.
- Integrate the client library: Replace your existing WebSocket client with
TickDBFailoverClientto get automatic reconnection and failover support. - Configure Route53 failover: Point your data API subdomain to Route53 with health checks and a failover routing policy.
- Test under chaos conditions: Schedule a failover drill before you need to rely on it in production.
If you need high-volume historical OHLCV data for backtesting across US equities, crypto, HK equities, or other asset classes, reach out to enterprise@tickdb.ai for institutional data plans with SLA-backed uptime guarantees.
If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your AI tool's marketplace to get TickDB API integration support directly in your development workflow.
This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. System architectures described here should be validated under your specific operational requirements before production deployment.