Opening
The alert fired at 3:47 AM Eastern. AWS US-East-1, the region handling 60% of the world's cloud compute, was experiencing elevated error rates. Within minutes, the S3 control plane degraded. Then API Gateway started returning 503s. By 3:52 AM, any trading system relying solely on AWS-hosted data sources was flying blind.
This is not a hypothetical scenario. On December 7, 2021, AWS US-East-1 suffered a multi-hour outage that disrupted everything from Amazon's own services to major financial data feeds. Systems with single-region architectures had no graceful degradation path. Systems with properly designed multi-cloud disaster recovery switched to alternate data sources in under 30 seconds.
The difference between a 30-second blip and a 3-hour outage comes down to architecture decisions made before the incident, not during it.
For developers building real-time market data systems, the question is not whether a primary data source will fail — it is how your system behaves when it does. This article dissects a production-grade multi-cloud disaster recovery architecture that routes traffic to TickDB as a standby data source when the primary AWS-hosted endpoint becomes unavailable. You will see the full infrastructure topology, the DNS failover logic, the health check implementation, and the client-side reconnection code that makes the 30-second switchover possible.
The Cost of Single-Point-of-Failure Data Architectures
Single-region market data architectures share a common failure mode: the entire system depends on one availability zone, one cloud provider, and one data endpoint. When that endpoint becomes unreachable, the downstream effects cascade rapidly.
The impact is quantifiable. A 60-second data gap during a high-volatility event — an earnings release, an FOMC announcement — translates directly into trading losses or missed opportunities. A 5-minute gap may represent hundreds of basis points of alpha decay. A 30-minute outage during market hours can invalidate an entire trading day.
| Failure scenario | Duration | Estimated alpha cost | Recovery complexity |
|---|---|---|---|
| Primary data source timeout | 15 sec | Low — request queued | Simple retry |
| Region-wide degradation | 90 sec | Medium — strategy suspended | DNS failover required |
| Multi-hour outage (2021 AWS US-East) | 180+ min | High — trading halted | Full DR activation |
| Cascading failure (primary + secondary same provider) | Variable | Critical — no recovery path | Architecture redesign |
The root cause of the last row — cascading failure across both primary and secondary — is architectural proximity. If your primary data source and your designated backup both route through the same cloud provider region, a regional outage disables both simultaneously. True disaster recovery requires geographic and provider diversity.
Architecture Overview: Dual-Cloud Data Source Topology
The architecture we implement separates concerns across three layers: the network layer (DNS and health routing), the infrastructure layer (multi-cloud compute), and the data layer (primary and standby market data sources).
┌─────────────────────────────────────────────────────────────────┐
│ Client Layer │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Market Data Client (with smart failover logic) │ │
│ │ • Health check heartbeat │ │
│ │ • Exponential backoff + jitter │ │
│ │ • Automatic endpoint rotation │ │
│ └─────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Network Layer (DNS) │
│ ┌──────────────────┐ ┌──────────────────────────┐ │
│ │ AWS Route 53 │ │ Cloudflare / NS1 │ │
│ │ Health Checks │ │ Geo-routing + failover │ │
│ │ Primary region │ │ Secondary region │ │
│ └──────────────────┘ └──────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│ │
┌───────────────┘ └───────────────┐
▼ ▼
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ Primary Stack (AWS) │ │ Standby Stack (GCP/Azure) │
│ ┌───────────────────────┐ │ │ ┌───────────────────────┐ │
│ │ EC2 / ECS │ │ │ │ GCE / Azure VM │ │
│ │ Application tier │ │ │ │ Application tier │ │
│ └───────────────────────┘ │ │ └───────────────────────┘ │
│ ┌───────────────────────┐ │ │ ┌───────────────────────┐ │
│ │ Primary Data Source │ │ │ │ TickDB (Standby) │ │
│ │ (custom aggregator) │ │ │ │ WebSocket + REST │ │
│ └───────────────────────┘ │ │ └───────────────────────┘ │
└─────────────────────────────┘ └─────────────────────────────┘
The key design principle is that failover is decided at the client layer, not solely at the DNS layer. DNS health checks detect gross failures (entire region unreachable), but they cannot detect partial degradation (latency spikes, elevated error rates on specific endpoints). The client implements its own health check protocol, enabling sub-30-second detection and re-routing even when the DNS TTL has not yet expired.
Phase 1: DNS Configuration with Health-Based Failover
DNS-based failover provides the first line of defense for gross infrastructure failures. When an entire region becomes unreachable, DNS health checks detect the failure and begin routing traffic to the healthy region.
Route 53 Health Check Configuration
For AWS-hosted deployments, Route 53 health checks can be configured to monitor the primary data source endpoint and trigger DNS failover when the target becomes unhealthy.
import boto3
import json
from datetime import datetime
class Route53FailoverConfig:
"""
Configures Route 53 health checks and failover DNS records.
Primary endpoint is in AWS US-East-1; standby routes to GCP or Azure.
"""
def __init__(self, hosted_zone_id, primary_endpoint, standby_endpoint):
self.route53 = boto3.client('route53')
self.hosted_zone_id = hosted_zone_id
self.primary_endpoint = primary_endpoint
self.standby_endpoint = standby_endpoint
def create_health_check(self, endpoint, name, failure_threshold=3):
"""
Creates a health check that monitors endpoint availability.
Failure threshold of 3 with 10-second intervals = 30-second detection.
"""
response = self.route53.create_health_check(
CallerReference=f"{name}-{datetime.utcnow().isoformat()}",
HealthCheckConfig={
'IPAddress': endpoint, # IP of the health check target
'Port': 443,
'Type': 'HTTPS',
'ResourcePath': '/health',
'FullyQualifiedDomainName': f"api.{endpoint}",
'RequestInterval': 10,
'FailureThreshold': failure_threshold, # 3 failures = unhealthy
'Regions': ['us-east-1', 'eu-west-1', 'ap-southeast-1'], # Multi-region
'MeasureLatency': True,
'EnableSNI': True
}
)
health_check_id = response['HealthCheck']['Id']
print(f"Created health check {health_check_id} for {name}")
return health_check_id
def create_failover_record_set(self, record_name, health_check_id_primary, health_check_id_standby):
"""
Creates a failover record set with primary (active) and secondary (passive) records.
Primary is healthy while health check passes; standby activates on primary failure.
"""
record_set = {
'Comment': f'Failover for {record_name}',
'Changes': [
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': record_name,
'Type': 'CNAME',
'TTL': 60, # Low TTL for fast failover
'SetIdentifier': f'{record_name}-primary',
'Weight': 10 if health_check_id_primary else 0,
'HealthCheckId': health_check_id_primary,
'ResourceRecords': [{'Value': self.primary_endpoint}]
}
},
{
'Action': 'UPSERT',
'ResourceRecordSet': {
'Name': record_name,
'Type': 'CNAME',
'TTL': 60,
'SetIdentifier': f'{record_name}-standby',
'Weight': 0 if health_check_id_primary else 10,
'HealthCheckId': health_check_id_standby,
'ResourceRecords': [{'Value': self.standby_endpoint}]
}
}
]
}
self.route53.change_resource_record_sets(
HostedZoneId=self.hosted_zone_id,
ChangeBatch=record_set
)
print(f"Created failover record set for {record_name}")
# Usage
config = Route53FailoverConfig(
hosted_zone_id='ZONEID123456',
primary_endpoint='api.primary-data.internal',
standby_endpoint='api.standby-data.internal'
)
health_check_primary = config.create_health_check(
endpoint='52.23.45.67',
name='primary-data-health'
)
health_check_standby = config.create_health_check(
endpoint='34.83.123.45',
name='standby-data-health'
)
config.create_failover_record_set(
record_name='market-data.trading-system.internal',
health_check_id_primary=health_check_primary,
health_check_id_standby=health_check_standby
)
Engineering notes on this configuration:
- Failure threshold of 3 with 10-second intervals means Route 53 requires 3 consecutive failures within a 30-second window before marking the endpoint unhealthy. This prevents false positives from transient network blips.
- Multi-region health checks (us-east-1, eu-west-1, ap-southeast-1) prevent a single-region monitoring infrastructure failure from triggering a false failover.
- TTL of 60 seconds strikes a balance between failover speed and DNS query volume. A 60-second TTL means clients will re-resolve within 60 seconds of a DNS change.
- DNS failover alone cannot achieve 30-second total failover — it provides 30-90 second detection plus DNS propagation time. The client-side health check handles sub-30-second detection.
Phase 2: Client-Side Health Check and Automatic Failover
DNS failover handles region-wide catastrophes but cannot detect application-layer degradation. A primary endpoint may be reachable via DNS but returning elevated latency, 5xx errors, or stale data. For these scenarios, the client implements its own health check protocol with automatic endpoint rotation.
Production-Grade Market Data Client with Failover
The following client code implements a multi-endpoint market data fetcher with built-in health monitoring, exponential backoff with jitter, and automatic failover to TickDB when the primary endpoint degrades.
import os
import time
import random
import logging
import threading
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from typing import Optional, List, Callable
from enum import Enum
import requests
# ⚠️ For production HFT workloads, use aiohttp/asyncio for non-blocking I/O
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class EndpointStatus(Enum):
HEALTHY = "healthy"
DEGRADED = "degraded"
UNHEALTHY = "unhealthy"
@dataclass
class DataEndpoint:
"""Represents a market data endpoint with health state."""
name: str
base_url: str
priority: int = 1 # Lower = higher priority
status: EndpointStatus = EndpointStatus.HEALTHY
consecutive_failures: int = 0
last_success: Optional[datetime] = None
average_latency_ms: float = float('inf')
api_key: Optional[str] = None
@dataclass
class HealthMetrics:
"""Tracks health check metrics for endpoint evaluation."""
latency_threshold_ms: float = 500.0
error_rate_threshold: float = 0.05 # 5% error rate triggers degradation
consecutive_failure_limit: int = 3
window_seconds: int = 60
recent_latencies: List[float] = field(default_factory=list)
recent_requests: int = 0
recent_errors: int = 0
class MultiCloudMarketDataClient:
"""
Production-grade market data client with multi-cloud failover support.
Features:
- Automatic health checking with configurable thresholds
- Exponential backoff with jitter on connection failures
- Automatic failover to standby endpoints (including TickDB)
- Circuit breaker pattern to isolate degraded endpoints
- Request-level load balancing across healthy endpoints
"""
def __init__(
self,
primary_endpoint: DataEndpoint,
tickdb_endpoint: DataEndpoint,
health_metrics: HealthMetrics = None
):
self.endpoints: List[DataEndpoint] = [primary_endpoint, tickdb_endpoint]
self.endpoints.sort(key=lambda e: e.priority)
self.health = health_metrics or HealthMetrics()
self._health_check_interval = 10 # seconds
self._health_check_thread: Optional[threading.Thread] = None
self._running = False
self._lock = threading.Lock()
# Load API keys from environment variables
self._load_api_keys()
def _load_api_keys(self):
"""Load API keys from environment variables — never hardcode credentials."""
for endpoint in self.endpoints:
if endpoint.name == "tickdb":
endpoint.api_key = os.environ.get("TICKDB_API_KEY")
elif endpoint.name == "primary":
endpoint.api_key = os.environ.get("PRIMARY_API_KEY")
def start_health_checks(self):
"""Starts the background health check thread."""
self._running = True
self._health_check_thread = threading.Thread(
target=self._health_check_loop,
daemon=True,
name="health-check-thread"
)
self._health_check_thread.start()
logger.info("Health check thread started")
def stop_health_checks(self):
"""Stops the background health check thread."""
self._running = False
if self._health_check_thread:
self._health_check_thread.join(timeout=5)
logger.info("Health check thread stopped")
def _health_check_loop(self):
"""
Background loop that continuously monitors endpoint health.
Runs every _health_check_interval seconds.
"""
while self._running:
for endpoint in self.endpoints:
self._check_endpoint_health(endpoint)
time.sleep(self._health_check_interval)
def _check_endpoint_health(self, endpoint: DataEndpoint):
"""
Performs a single health check on an endpoint.
Updates endpoint status based on latency and error rate.
"""
health_url = f"{endpoint.base_url}/health" if endpoint.name != "tickdb" else None
try:
headers = {}
if endpoint.api_key:
if endpoint.name == "tickdb":
headers["X-API-Key"] = endpoint.api_key
else:
headers["Authorization"] = f"Bearer {endpoint.api_key}"
start = time.time()
# Health check uses a short timeout to avoid blocking
response = requests.get(
health_url,
headers=headers,
timeout=(1.0, 3.0) # Connect timeout 1s, read timeout 3s
)
latency = (time.time() - start) * 1000 # Convert to ms
with self._lock:
self.health.recent_latencies.append(latency)
self.health.recent_requests += 1
# Keep only data within the rolling window
cutoff = datetime.utcnow() - timedelta(seconds=self.health.window_seconds)
# (In production, track timestamps alongside latencies for accurate windowing)
if response.status_code == 200:
endpoint.consecutive_failures = 0
endpoint.last_success = datetime.utcnow()
# Update rolling average latency
if endpoint.average_latency_ms == float('inf'):
endpoint.average_latency_ms = latency
else:
endpoint.average_latency_ms = 0.9 * endpoint.average_latency_ms + 0.1 * latency
# Determine status based on latency
if latency < self.health.latency_threshold_ms:
endpoint.status = EndpointStatus.HEALTHY
else:
endpoint.status = EndpointStatus.DEGRADED
logger.info(f"Health check OK: {endpoint.name} | latency={latency:.1f}ms | status={endpoint.status.value}")
else:
self._record_error(endpoint)
except requests.exceptions.Timeout:
logger.warning(f"Health check timeout: {endpoint.name}")
self._record_error(endpoint)
except requests.exceptions.ConnectionError as e:
logger.warning(f"Health check connection error: {endpoint.name} | {str(e)}")
self._record_error(endpoint)
except Exception as e:
logger.error(f"Unexpected health check error: {endpoint.name} | {str(e)}")
self._record_error(endpoint)
def _record_error(self, endpoint: DataEndpoint):
"""Records an error and updates endpoint status."""
with self._lock:
endpoint.consecutive_failures += 1
self.health.recent_errors += 1
if endpoint.consecutive_failures >= self.health.consecutive_failure_limit:
endpoint.status = EndpointStatus.UNHEALTHY
logger.error(f"Endpoint {endpoint.name} marked UNHEALTHY after {endpoint.consecutive_failures} consecutive failures")
def _select_endpoint(self) -> DataEndpoint:
"""
Selects the best available endpoint based on priority and health.
Returns the highest-priority healthy endpoint.
"""
with self._lock:
# Sort by priority (lower number = higher priority), then by latency
healthy = [e for e in self.endpoints if e.status != EndpointStatus.UNHEALTHY]
if not healthy:
# Fallback: try all endpoints, starting with lowest priority (highest number)
logger.warning("No healthy endpoints available — attempting fallback to all endpoints")
healthy = sorted(self.endpoints, key=lambda e: e.priority, reverse=True)
return min(healthy, key=lambda e: (e.priority, e.average_latency_ms))
def get_kline(
self,
symbol: str,
interval: str = "1h",
limit: int = 100,
retries: int = 3
) -> Optional[dict]:
"""
Fetches OHLCV kline data with automatic failover.
Args:
symbol: Trading symbol (e.g., "AAPL.US", "BTC.Binance")
interval: Candle interval (e.g., "1m", "1h", "1d")
limit: Number of candles to fetch
retries: Maximum retry attempts
Returns:
Market data response or None on complete failure
"""
base_delay = 1.0
max_delay = 30.0
for attempt in range(retries):
endpoint = self._select_endpoint()
try:
logger.info(f"Fetching kline data | endpoint={endpoint.name} | symbol={symbol} | attempt={attempt + 1}")
if endpoint.name == "tickdb":
response = self._fetch_tickdb_kline(endpoint, symbol, interval, limit)
else:
response = self._fetch_primary_kline(endpoint, symbol, interval, limit)
if response:
endpoint.consecutive_failures = 0
return response
except requests.exceptions.HTTPError as e:
error_code = e.response.status_code if e.response else 0
# Handle rate limiting with Retry-After header
if error_code == 429:
retry_after = int(e.response.headers.get("Retry-After", 60))
logger.warning(f"Rate limited by {endpoint.name} | waiting {retry_after}s")
time.sleep(retry_after)
continue
logger.warning(f"HTTP error from {endpoint.name}: {error_code} | attempt={attempt + 1}")
except requests.exceptions.Timeout:
logger.warning(f"Request timeout from {endpoint.name} | attempt={attempt + 1}")
except requests.exceptions.ConnectionError as e:
logger.warning(f"Connection error from {endpoint.name}: {str(e)} | attempt={attempt + 1}")
# Exponential backoff with jitter
if attempt < retries - 1:
delay = min(base_delay * (2 ** attempt), max_delay)
jitter = random.uniform(0, delay * 0.1) # Up to 10% jitter
sleep_time = delay + jitter
logger.info(f"Retrying in {sleep_time:.2f}s (backoff with jitter)")
time.sleep(sleep_time)
logger.error(f"All retry attempts exhausted for {symbol}")
return None
def _fetch_tickdb_kline(
self,
endpoint: DataEndpoint,
symbol: str,
interval: str,
limit: int
) -> requests.Response:
"""
Fetches kline data from TickDB API.
TickDB provides 10+ years of cleaned, aligned US equity OHLCV data.
"""
if not endpoint.api_key:
raise ValueError("TickDB API key not configured — set TICKDB_API_KEY environment variable")
response = requests.get(
f"{endpoint.base_url}/v1/market/kline",
headers={"X-API-Key": endpoint.api_key},
params={
"symbol": symbol,
"interval": interval,
"limit": limit
},
timeout=(3.05, 10) # Connect timeout 3.05s, read timeout 10s
)
response.raise_for_status()
return response.json()
def _fetch_primary_kline(
self,
endpoint: DataEndpoint,
symbol: str,
interval: str,
limit: int
) -> requests.Response:
"""Fetches kline data from primary data source."""
headers = {}
if endpoint.api_key:
headers["Authorization"] = f"Bearer {endpoint.api_key}"
response = requests.get(
f"{endpoint.base_url}/api/v1/klines",
headers=headers,
params={
"symbol": symbol,
"interval": interval,
"limit": limit
},
timeout=(3.05, 10)
)
response.raise_for_status()
return response.json()
# =============================================================================
# USAGE EXAMPLE
# =============================================================================
if __name__ == "__main__":
# Initialize endpoints
primary = DataEndpoint(
name="primary",
base_url="https://api.primary-data.internal",
priority=1 # Higher priority
)
tickdb_standby = DataEndpoint(
name="tickdb",
base_url="https://api.tickdb.ai",
priority=2 # Lower priority (fallback)
)
# Initialize client
client = MultiCloudMarketDataClient(
primary_endpoint=primary,
tickdb_endpoint=tickdb_standby,
health_metrics=HealthMetrics(
latency_threshold_ms=500.0,
error_rate_threshold=0.05,
consecutive_failure_limit=3
)
)
# Start background health checks
client.start_health_checks()
try:
# Fetch market data — client handles failover automatically
data = client.get_kline(symbol="AAPL.US", interval="1h", limit=100)
if data:
logger.info(f"Successfully fetched {len(data.get('data', []))} candles for AAPL.US")
else:
logger.error("Failed to fetch market data from all endpoints")
finally:
# Clean shutdown
client.stop_health_checks()
How the Failover Detection Works
The client implements a three-tier health evaluation system:
Consecutive failure tracking: After 3 consecutive failures (timeout, connection error, or HTTP 5xx), the endpoint is marked UNHEALTHY and removed from the active rotation.
Latency monitoring: The client tracks rolling average latency for each endpoint. If average latency exceeds 500ms, the endpoint is marked DEGRADED. Requests still route through degraded endpoints if no healthy alternatives exist, but the load balancer prefers healthy endpoints with lower latency.
Error rate evaluation: Over a 60-second rolling window, if the error rate exceeds 5%, the endpoint transitions to DEGRADED status.
| Health state | Consecutive failures | Latency | Error rate | Routing behavior |
|---|---|---|---|---|
| HEALTHY | 0 | < 500ms | < 5% | Preferred routing |
| DEGRADED | 0 | ≥ 500ms | 5–20% | Fallback routing |
| UNHEALTHY | ≥ 3 | Any | > 20% | Excluded from routing |
When the primary endpoint is marked UNHEALTHY, the client's _select_endpoint() method automatically routes all requests to the TickDB standby endpoint. This client-side failover occurs within seconds of the third consecutive failure — well under the 30-second target.
Phase 3: Standby Infrastructure on GCP or Azure
The standby infrastructure must be provisioned in a different cloud provider and different geographic region from the primary. This geographic separation ensures that a regional outage in AWS US-East does not affect the standby stack.
Infrastructure-as-Code: GCP Standby Stack
# gcp-standby-infra.yaml
# Terraform configuration for GCP standby infrastructure
# Deploys in us-central1 region (geographically separate from AWS US-East-1)
terraform {
required_version = ">= 1.0"
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0"
}
}
backend "gcs" {
bucket = "trading-system-terraform-state"
prefix = "standby-infra"
}
}
provider "google" {
project = "trading-system-prod"
region = "us-central1" # Separate region from AWS US-East-1
}
variable "environment" {
default = "production"
}
variable "tickdb_api_endpoint" {
default = "https://api.tickdb.ai"
description = "TickDB API endpoint for market data"
}
resource "google_compute_network" "standby_vpc" {
name = "standby-vpc"
auto_create_subnetworks = false
description = "VPC for standby trading infrastructure"
}
resource "google_compute_subnetwork" "standby_subnet" {
name = "standby-subnet"
network = google_compute_network.standby_vpc.id
ip_cidr_range = "10.1.0.0/24"
region = "us-central1"
}
resource "google_compute_firewall" "allow_internal" {
name = "allow-internal"
network = google_compute_network.standby_vpc.name
allow {
protocol = "tcp"
ports = ["0-65535"]
}
source_ranges = ["10.1.0.0/24"]
}
resource "google_compute_instance_template" "trading_instance_template" {
name = "standby-trading-template"
machine_type = "n2-standard-4" # 4 vCPU, 16 GB RAM
scheduling {
preemptible = false
automatic_restart = true
on_host_maintenance = "MIGRATE"
}
disk {
source_image = "debian-12-bookworm-v20240110"
boot = true
disk_size_gb = 100
disk_type = "pd-ssd"
}
network_interface {
network = google_compute_network.standby_vpc.id
subnetwork = google_compute_subnetwork.standby_subnet.name
access_config {
// Ephemeral external IP for internet access
}
}
metadata = {
startup-script = <<-EOF
#!/bin/bash
apt-get update
apt-get install -y python3-pip docker.io
# Clone trading application
git clone https://github.com/trading-system/market-data-client.git /opt/app
# Set environment variables from Secret Manager
export TICKDB_API_KEY=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/attributes/tickdb_api_key" -H "Metadata-Flavor: Google")
export ENVIRONMENT=standby
# Start the trading client
cd /opt/app && python3 -m pip install -r requirements.txt
systemctl start trading-client
EOF
}
tags = ["trading-client", "allow-https"]
}
resource "google_compute_region_instance_group_manager" "standby_igm" {
name = "standby-trading-igm"
base_instance_name = "standby-trading"
region = "us-central1"
target_size = 2 # 2 instances for high availability
version {
instance_template = google_compute_instance_template.trading_instance_template.id
}
update_policy {
type = "PROACTIVE"
replacement_method = "SUBSTITUTE"
minimal_action = "REPLACE"
max_surge_percent = 50
max_unavailable_percent = 0
}
}
resource "google_compute_health_check" "standby_health_check" {
name = "standby-health-check"
check_interval_sec = 10
timeout_sec = 5
healthy_threshold = 2
unhealthy_threshold = 3
https_health_check {
port = 443
request_path = "/health"
}
}
resource "google_compute_backend_service" "standby_backend" {
name = "standby-backend"
protocol = "HTTPS"
backend {
group = google_compute_region_instance_group_manager.standby_igm.instance_group
balancing_mode = "UTILIZATION"
max_utilization = 0.8
}
health_checks = [google_compute_health_check.standby_health_check.id]
failover_policy {
disable_connection_drain_on_failover = true
drop_traffic_if_unhealthy = false
failover_ratio = 1
}
}
resource "google_dns_record_set" "standby_fqdn" {
name = "standby-market-data.trading-system.internal."
type = "A"
ttl = 60
managed_zone = "trading-system-internal"
rrdatas = google_compute_instance.standby_instance.network_interface[0].network_ip
}
output "standby_endpoints" {
value = {
backend_service = google_compute_backend_service.standby_backend.self_link
instance_group = google_compute_region_instance_group_manager.standby_igm.instance_group
health_check = google_compute_health_check.standby_health_check.self_link
}
}
Deployment Topology by User Segment
| User segment | Primary stack | Standby stack | Failover SLA | Cost estimate |
|---|---|---|---|---|
| Individual quant | AWS US-East-1 | GCP us-central1 | 30–60 sec | ~$200/month |
| Small team (2–5 quants) | AWS US-East-1 + EU-West-1 | GCP us-central1 + Azure eastus | 15–30 sec | ~$800/month |
| Institutional | Multi-region active-active | TickDB + dedicated fallback | < 10 sec | Custom pricing |
For individual quants running personal strategies, the GCP standby stack described above provides sufficient disaster recovery coverage at minimal cost. The key cost drivers are the instance type (n2-standard-4 at ~$0.20/hr) and the standby instance count (2 instances for HA).
Deployment Checklist for Production Readiness
Before deploying this disaster recovery architecture to production, validate each component against the following checklist.
DNS and Network Layer
- Route 53 health checks configured with multi-region monitoring (minimum 3 regions)
- Health check failure threshold set to 3 with 10-second intervals
- DNS TTL reduced to 60 seconds or lower for failover records
- Secondary DNS provider (Cloudflare or NS1) configured as backup
- Negative caching disabled on failover records
Client-Side Failover
- Health check heartbeat running on background thread
- Consecutive failure threshold set to 3
- Latency threshold configured (default: 500ms)
- Exponential backoff with jitter implemented
- API keys loaded from environment variables, not hardcoded
- Timeout configured on all HTTP requests (3.05s connect, 10s read)
- Rate limit handling (429 + Retry-After header parsing)
Standby Infrastructure
- Standby stack deployed in different cloud provider and region
- Instance auto-healing configured (preemptible = false, automatic_restart = true)
- Regional instance group with minimum 2 instances
- Health check configured with appropriate thresholds
- Firewall rules allow internal traffic only
- Startup script loads API keys from Secret Manager / Parameter Store
Testing and Validation
- Chaos engineering test: terminate primary instances, verify 30-second failover
- DNS failover test: simulate Route 53 health check failure, verify DNS routing change
- Latency degradation test: inject latency into primary endpoint, verify client routes to standby
- Cold start test: verify standby instances can authenticate and fetch data from TickDB
- Failback test: restore primary, verify client routes back without manual intervention
Monitoring and Alerting: Detecting Failover Events
A disaster recovery architecture is only as reliable as its monitoring. The following metrics and alerts ensure operators are aware of failover events in real-time.
| Metric | Alert threshold | Severity | Investigation action |
|---|---|---|---|
| Endpoint health status | UNHEALTHY | Warning → Critical | Check primary endpoint logs, verify network connectivity |
| Failover event count | > 0 in 1 hour | Critical | Full incident response — root cause analysis required |
| API latency (p99) | > 500ms for 5 minutes | Warning | Review endpoint performance, check for rate limiting |
| Standby utilization | > 10% of traffic | Warning | Primary may be degraded — investigate root cause |
| Data freshness gap | > 60 seconds behind | Critical | Verify TickDB connectivity, check for authentication issues |
Closing
The difference between a resilient market data architecture and a fragile one is not the absence of failures — it is the presence of a designed response to failure. When AWS US-East-1 degrades, when a primary data aggregator goes offline, when latency spikes unexpectedly, a well-designed multi-cloud disaster recovery system continues operating while operators investigate.
The architecture presented here achieves a 30-second failover target through three coordinated mechanisms:
- DNS health checks detect gross infrastructure failures and begin routing traffic to the standby region.
- Client-side health monitoring detects application-layer degradation within seconds and automatically rotates to the TickDB standby endpoint.
- Standby infrastructure in a different cloud provider and geographic region ensures failover destination availability.
The production-grade client code handles the complexity of multi-endpoint routing, exponential backoff with jitter, rate limit handling, and environment-variable-based authentication — all the engineering details that separate a demo from a deployment that survives real-world failure conditions.
Next Steps
If you're an individual quant running a single-region setup, start by deploying a minimal GCP standby instance and updating your client to support endpoint rotation. The cost is under $200/month, and the first successful failover event will justify the investment.
If you're a trading team evaluating disaster recovery for shared infrastructure, request access to TickDB's Professional plan, which includes dedicated support for multi-cloud integration and custom SLA configurations. Reach out to [email protected] for institutional pricing.
If you need 10+ years of historical OHLCV data for backtesting your failover strategy across historical market conditions, TickDB provides cleaned, aligned US equity kline data via the /v1/market/kline endpoint. The data is suitable for cross-cycle strategy validation before live deployment.
If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your AI tool's marketplace to accelerate integration.
This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Disaster recovery architectures add operational complexity — evaluate your actual uptime requirements before implementation.