When you're building a cross-market trading system, the naive approach quickly becomes a maintenance nightmare: one WebSocket connection for US equities, another for Hong Kong stocks, a third for crypto exchanges, each with its own authentication mechanism, message format, and reconnection logic. Your code base fragments along exchange boundaries. Your monitoring dashboards multiply. Your incident response runbooks need separate playbooks for every venue.
This is the problem TickDB's unified market data gateway was designed to solve. The architecture allows a single WebSocket connection to multiplex market data across six asset classes—US equities, Hong Kong stocks, A-shares, forex, cryptocurrencies, commodities, and indices—while presenting a normalized data model to the application layer. This article dissects how that works at the protocol, data model, and implementation levels.
The Protocol Fragmentation Problem
Before diving into the solution, it is worth establishing exactly why cross-market data aggregation is hard. Each exchange operates its own protocol:
| Exchange Type | Typical Protocol | Authentication | Message Format | Timestamp |
|---|---|---|---|---|
| US equities (e.g., Nasdaq) | FIX/ITCH | Session token | Binary / FIX | Nanoseconds, NYSE calendar |
| HK stocks (HKEX) | OMD-C | API key + signature | Binary | Microseconds, HK calendar |
| Crypto (Binance, OKX) | WebSocket JSON | HMAC signature in message | JSON | Milliseconds, UTC |
| Forex (ICAP) | N/A (broker-specific) | Variable | Variable | Variable |
The differences are not cosmetic. They touch every layer:
- Authentication: HMAC signatures, session tokens, API keys in headers vs. body vs. URL parameters.
- Message framing: Binary protocols require custom parsers; JSON requires schema normalization.
- Timestamps: Different precision, different time zones, different holiday calendars.
- Symbol formats:
AAPLvs.AAPL.USvs.USD/AUDvs.BTC-USDT. - Incremental updates: Some feeds send deltas; others send full snapshots; others send both with different trigger conditions.
A system that tries to integrate these directly ends up with N exchange adapters, each tested independently, each with its own failure modes. The operational burden scales linearly with the number of markets.
TickDB's Unified Gateway Architecture
TickDB solves this through a three-layer architecture:
┌─────────────────────────────────────────────────────────────────┐
│ Application Layer │
│ (Your strategy, dashboard, or backtesting engine) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Unified Normalization Layer │
│ - Unified symbol format │
│ - Normalized timestamp (UTC, milliseconds) │
│ - Standardized message schema (kline, depth, trades, ticker) │
│ - Timezone-aware datetime conversion │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Exchange Adapter Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ US Equity│ │ HK Stock │ │ Crypto │ │ Forex │ │
│ │ Adapter │ │ Adapter │ │ Adapter │ │ Adapter │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ (Protocol-specific handling, auth, parsing) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Exchange Gateways │
│ (Nasdaq / HKEX / Binance / Broker-specific feeds) │
└─────────────────────────────────────────────────────────────────┘
Layer 1: Exchange Adapter Layer
Each exchange has a dedicated adapter that handles:
- Protocol-specific authentication: Converts TickDB's unified auth token into the exchange's required format (HMAC, session token, signed URLs, etc.).
- Message parsing: Decodes binary feeds, validates checksums, and reconstructs structured data.
- Rate limit management: Each exchange enforces different rate limits; adapters manage backpressure independently.
- Heartbeat protocol: Normalizes ping/pong patterns across WebSocket, FIX, and proprietary protocols.
Layer 2: Unified Normalization Layer
This is the core of the gateway. All data flowing through this layer is transformed into a canonical model:
- Symbol normalization: Every symbol is mapped to a universal format
EXCHANGE:SYMBOL(e.g.,NASDAQ:AAPL,HKEX:0700,BINANCE:BTC-USDT). The application layer always uses this format; the adapter layer handles the reverse mapping for each exchange. - Timestamp normalization: All timestamps are converted to UTC, with millisecond precision. Holiday calendars are stored separately and referenced during filtering.
- Schema normalization: Kline, depth, trades, and ticker messages have standardized field names and types regardless of the source exchange.
Layer 3: Application Layer Interface
The application layer interacts exclusively with the normalized interface. It never needs to know whether AAPL data came from a US equity feed or whether BTC-USDT came from a crypto venue.
The Symbol Resolution System
Symbol mapping is one of the most operationally complex parts of cross-market integration. TickDB maintains a symbol registry that handles:
| Canonical Symbol | Exchange | Local Symbol | Asset Class |
|---|---|---|---|
NASDAQ:AAPL |
Nasdaq | AAPL |
US Equities |
HKEX:0700 |
HKEX | 00700 |
HK Stocks |
HKEX:9988 |
HKEX | 09988 |
HK Stocks |
BINANCE:BTC-USDT |
Binance | BTCUSDT |
Crypto |
OKX:BTC-USDT |
OKX | BTC-USDT |
Crypto |
The critical insight is that symbol resolution happens at subscription time, not at parse time. When you send:
{
"cmd": "subscribe",
"params": ["kline:NASDAQ:AAPL:1m", "depth:HKEX:0700:10"]
}
The gateway routes each subscription to the correct adapter, translates the canonical symbol to the exchange-specific local symbol, and establishes the appropriate feed channel. From your application's perspective, you are always working with EXCHANGE:SYMBOL format.
Timezone and Calendar Standardization
Market data is only useful if you can align it correctly across time zones. A 9:30 AM "open" on NYSE is not the same moment as a 9:30 AM open on HKEX. TickDB normalizes this through:
UTC normalization: All timestamps are stored and transmitted in UTC. When you receive a kline from HKEX, the
open_timeandclose_timefields are UTC milliseconds, not Hong Kong local time.Calendar metadata: Each market carries calendar metadata (
market_hours,timezone,holidays) that the application layer can query. This enables:- Accurate backtest period filtering (excluding weekends and holidays).
- Pre-market and after-hours window detection.
- Roll-over time calculation for futures-equivalent instruments.
Conversion utilities: The SDK provides timezone-aware conversion helpers:
from datetime import datetime, timezone
from tickdb import TickDB, MarketCalendar
client = TickDB(api_key=os.environ.get("TICKDB_API_KEY"))
# Get calendar metadata for US equities
calendar = client.markets.get_calendar("NASDAQ")
print(f"Timezone: {calendar.timezone}") # "America/New_York"
print(f"Open: {calendar.open_hour}:{calendar.open_minute}") # 9:30
print(f"Close: {calendar.close_hour}:{calendar.close_minute}") # 16:00
# Convert a UTC timestamp to market local time
utc_time = datetime.now(timezone.utc)
local_time = calendar.to_local_time(utc_time)
print(f"Market local time: {local_time}") # Adjusts for EDT/EST automatically
Production-Grade WebSocket Client
The following code implements a robust WebSocket client that connects to TickDB's unified gateway and subscribes to cross-market data. This is the reference implementation for production use:
import os
import json
import time
import random
import threading
import logging
from datetime import datetime, timezone
from websocket import create_connection, WebSocketException
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("tickdb_gateway")
class TickDBGateway:
"""
Production-grade unified gateway client for cross-market TickDB data.
Handles:
- Single WebSocket connection to multiple markets
- Automatic reconnection with exponential backoff and jitter
- Rate limit handling (code 3001 + Retry-After)
- Cross-market subscription management
- Heartbeat keepalive
"""
GATEWAY_URL = "wss://stream.tickdb.ai/ws"
PING_INTERVAL = 20 # seconds
MAX_RECONNECT_DELAY = 60 # seconds
BASE_RECONNECT_DELAY = 1 # seconds
def __init__(self, api_key: str):
if not api_key:
raise ValueError("TICKDB_API_KEY environment variable is not set")
self.api_key = api_key
self._ws = None
self._running = False
self._reconnect_thread = None
self._ping_thread = None
self._subscriptions = set()
self._last_pong_received = None
self._reconnect_count = 0
def connect(self) -> None:
"""
Establish WebSocket connection with TickDB unified gateway.
Authentication is via URL parameter as per TickDB spec.
"""
try:
auth_url = f"{self.GATEWAY_URL}?api_key={self.api_key}"
logger.info(f"Connecting to TickDB unified gateway...")
self._ws = create_connection(
auth_url,
ping_interval=None # We handle ping/pong manually
)
self._running = True
self._reconnect_count = 0
logger.info("Connected to TickDB unified gateway successfully")
self._start_ping_thread()
self._start_receive_loop()
except WebSocketException as e:
logger.error(f"Failed to connect: {e}")
self._schedule_reconnect()
def subscribe(self, channels: list[str]) -> dict:
"""
Subscribe to cross-market data channels.
Args:
channels: List of channel specs in format "type:EXCHANGE:SYMBOL:params"
Examples:
- "kline:NASDAQ:AAPL:1m"
- "depth:HKEX:0700:10"
- "trades:BINANCE:BTC-USDT"
- "ticker:NASDAQ:AAPL"
Returns:
Subscription confirmation response from gateway
"""
for channel in channels:
self._subscriptions.add(channel)
subscribe_msg = {
"cmd": "subscribe",
"params": channels
}
return self._send_and_wait(subscribe_msg)
def _send_and_wait(self, message: dict, timeout: float = 5.0) -> dict:
"""Send message and wait for acknowledgment."""
if not self._ws or not self._running:
raise ConnectionError("WebSocket is not connected")
import websocket
self._ws.send(json.dumps(message))
logger.info(f"Sent: {message.get('cmd', 'unknown')} - {message.get('params', [])}")
# Wait for response
start = time.time()
while time.time() - start < timeout:
try:
response = self._ws.recv()
data = json.loads(response)
return data
except websocket.WebSocketTimeoutException:
continue
raise TimeoutError(f"No response received within {timeout}s")
def _start_ping_thread(self) -> None:
"""Send periodic heartbeat pings to keep connection alive."""
def ping_loop():
while self._running:
time.sleep(self.PING_INTERVAL)
if self._running and self._ws:
try:
self._ws.send(json.dumps({"cmd": "ping"}))
logger.debug("Sent heartbeat ping")
except Exception as e:
logger.warning(f"Ping failed: {e}")
self._schedule_reconnect()
break
thread = threading.Thread(target=ping_loop, daemon=True)
thread.start()
def _start_receive_loop(self) -> None:
"""Main receive loop for processing incoming market data."""
def receive_loop():
while self._running:
try:
message = self._ws.recv()
self._handle_message(message)
except Exception as e:
if self._running:
logger.error(f"Receive error: {e}")
self._schedule_reconnect()
break
thread = threading.Thread(target=receive_loop, daemon=True)
thread.start()
def _handle_message(self, raw_message: str) -> None:
"""
Process incoming messages from the unified gateway.
Handles:
- kline: OHLCV candles with UTC timestamps
- depth: Order book snapshots (L1-L50 depending on market)
- trades: Individual trade executions
- ticker: 24hr summary statistics
- pong: Heartbeat response
- error: Error responses (including rate limits)
"""
try:
msg = json.loads(raw_message)
msg_type = msg.get("type", "unknown")
channel = msg.get("channel", "unknown")
if msg_type == "pong":
self._last_pong_received = datetime.now(timezone.utc)
logger.debug("Received heartbeat pong")
return
if msg_type == "error":
code = msg.get("code", 0)
if code == 3001:
# Rate limit exceeded — extract Retry-After
retry_after = int(msg.get("headers", {}).get("Retry-After", 5))
logger.warning(f"Rate limited. Retry after {retry_after}s")
time.sleep(retry_after)
# Re-subscribe to dropped channels
self._resubscribe_all()
else:
logger.error(f"Gateway error {code}: {msg.get('message')}")
return
# Normal market data message — process based on type
if msg_type in ("kline", "depth", "trades", "ticker"):
self._process_market_data(msg)
else:
logger.debug(f"Unhandled message type: {msg_type}")
except json.JSONDecodeError as e:
logger.error(f"Failed to parse message: {e}")
def _process_market_data(self, msg: dict) -> None:
"""
Process normalized market data messages.
All messages arrive in unified format regardless of source exchange:
- Timestamps are UTC milliseconds
- Symbol format is EXCHANGE:SYMBOL
- Fields are consistent across all markets
"""
msg_type = msg.get("type")
channel = msg.get("channel") # e.g., "kline:NASDAQ:AAPL:1m"
data = msg.get("data", {})
# Extract market from channel
# Format: type:EXCHANGE:SYMBOL[:params]
parts = channel.split(":")
exchange = parts[1] if len(parts) > 1 else "UNKNOWN"
symbol = parts[2] if len(parts) > 2 else "UNKNOWN"
# Log with exchange context for debugging
logger.info(
f"[{exchange}] {msg_type.upper()} {symbol}: "
f"{self._summarize_data(msg_type, data)}"
)
def _summarize_data(self, msg_type: str, data: dict) -> str:
"""Create a one-line summary of the data for logging."""
if msg_type == "kline":
return (
f"O:{data.get('open'):.2f} H:{data.get('high'):.2f} "
f"L:{data.get('low'):.2f} C:{data.get('close'):.2f} "
f"V:{data.get('volume'):,.0f}"
)
elif msg_type == "depth":
bids = len(data.get("bids", []))
asks = len(data.get("asks", []))
return f"bids:{bids} asks:{asks}"
elif msg_type == "trades":
return f"price:{data.get('price')} qty:{data.get('quantity')} side:{data.get('side')}"
elif msg_type == "ticker":
return f"last:{data.get('last')} change:{data.get('change_pct'):.2f}%"
return str(data)[:100]
def _resubscribe_all(self) -> None:
"""Re-establish all subscriptions after reconnection."""
if self._subscriptions:
logger.info(f"Re-subscribing to {len(self._subscriptions)} channels")
try:
self.subscribe(list(self._subscriptions))
except Exception as e:
logger.error(f"Re-subscription failed: {e}")
def _schedule_reconnect(self) -> None:
"""Schedule reconnection with exponential backoff and jitter."""
self._running = False
# Exponential backoff: delay = base * 2^attempt + jitter
delay = min(
self.BASE_RECONNECT_DELAY * (2 ** self._reconnect_count),
self.MAX_RECONNECT_DELAY
)
jitter = random.uniform(0, delay * 0.1) # Up to 10% jitter
reconnect_delay = delay + jitter
self._reconnect_count += 1
logger.info(f"Scheduling reconnect in {reconnect_delay:.2f}s (attempt {self._reconnect_count})")
def delayed_connect():
time.sleep(reconnect_delay)
self.connect()
# Re-establish subscriptions
if self._subscriptions:
self._resubscribe_all()
thread = threading.Thread(target=delayed_connect, daemon=True)
thread.start()
def close(self) -> None:
"""Gracefully close the connection."""
logger.info("Closing TickDB gateway connection")
self._running = False
if self._ws:
try:
self._ws.close()
except Exception as e:
logger.warning(f"Error during close: {e}")
# Example: Cross-market subscription
if __name__ == "__main__":
api_key = os.environ.get("TICKDB_API_KEY")
if not api_key:
raise RuntimeError("Please set TICKDB_API_KEY environment variable")
gateway = TickDBGateway(api_key)
try:
gateway.connect()
# Subscribe to cross-market data with a single connection
response = gateway.subscribe([
"kline:NASDAQ:AAPL:1m", # US equity kline
"depth:HKEX:0700:10", # HK stock depth (10 levels)
"trades:BINANCE:BTC-USDT", # Crypto trades
"ticker:NASDAQ:MSFT", # US equity ticker
])
logger.info(f"Subscription response: {response}")
# Keep the connection alive for data streaming
logger.info("Streaming cross-market data. Press Ctrl+C to exit.")
while True:
time.sleep(10)
except KeyboardInterrupt:
logger.info("Interrupted by user")
finally:
gateway.close()
Unified Data Model Specification
One of the key advantages of TickDB's gateway is the consistent data model across all markets. Here is the normalized schema for each message type:
Kline (OHLCV Candlestick)
{
"type": "kline",
"channel": "kline:NASDAQ:AAPL:1m",
"exchange": "NASDAQ",
"symbol": "AAPL",
"interval": "1m",
"data": {
"open_time": 1717100400000,
"close_time": 1717100460000,
"open": 212.50,
"high": 213.20,
"low": 212.30,
"close": 213.05,
"volume": 1245000,
"is_closed": true
}
}
Depth (Order Book)
{
"type": "depth",
"channel": "depth:HKEX:0700:10",
"exchange": "HKEX",
"symbol": "0700",
"data": {
"timestamp": 1717100450500,
"bids": [
{"price": 380.0, "quantity": 150000},
{"price": 379.8, "quantity": 220000}
],
"asks": [
{"price": 380.2, "quantity": 180000},
{"price": 380.5, "quantity": 95000}
]
}
}
Trades
{
"type": "trades",
"channel": "trades:BINANCE:BTC-USDT",
"exchange": "BINANCE",
"symbol": "BTC-USDT",
"data": {
"timestamp": 1717100450500,
"price": 67450.50,
"quantity": 0.0234,
"side": "buy",
"trade_id": "abc12345"
}
}
The consistency of this schema means your data processing pipeline does not need exchange-specific branches. A single kline handler works for US equities, HK stocks, and crypto because the field names and types are identical.
Cross-Market Deployment Configurations
Different deployment scenarios have different requirements. Here are the recommended configurations:
| Use Case | Recommended Plan | Subscription Strategy | Depth Levels |
|---|---|---|---|
| Individual quant research | Free / Starter | Selective subscriptions by symbol | L1 for US, L1–L5 for HK/Crypto |
| Active day trader | Professional | Real-time across 10–20 symbols | L5–L10 |
| Multi-strategy fund | Enterprise | Full market coverage with priority routing | L10–L50 |
| Backtesting (historical) | Professional / Enterprise | REST API for historical kline data | N/A |
For backtesting, note that the WebSocket gateway handles real-time streaming. Historical data retrieval uses the REST API (/v1/market/kline endpoint), which provides 10+ years of cleaned, aligned US equity OHLCV data. The two interfaces are complementary: REST for historical analysis, WebSocket for live execution.
Value Comparison: Unified Gateway vs. Direct Exchange Integration
For teams evaluating the build-versus-buy decision for cross-market data infrastructure:
| Capability | Direct Exchange Integration | TickDB Unified Gateway |
|---|---|---|
| Connections required | 1 per exchange | Single connection |
| Protocol adapters needed | N (one per exchange) | Zero (handled internally) |
| Symbol normalization | Custom mapping per exchange | Built-in EXCHANGE:SYMBOL format |
| Timezone handling | Custom calendar logic per market | Unified UTC + calendar metadata |
| Maintenance burden | O(N) — scales with exchanges | O(1) — single interface |
| Authentication | Unique per exchange (HMAC, tokens, etc.) | Single API key |
| Rate limit management | Per-exchange logic | Unified handling via gateway |
| Depth channel support | Varies by exchange | depth channel for US (L1), HK (L1–L10), Crypto (L1–L10) |
| Historical backtest data | Typically unavailable | 10+ years of US equity OHLCV |
Note: The trades endpoint does not cover US equities or A-shares. For tick-level trade data on US markets, alternative data sources are required.
Conclusion
The core challenge of cross-market market data is not just connectivity—it is the normalization layer that makes that data actionable. A single WebSocket connection to TickDB's unified gateway abstracts away protocol differences, symbol variations, and timezone complexity, leaving you with a consistent data model that works identically whether you are trading AAPL on Nasdaq, 0700 on HKEX, or BTC-USDT on Binance.
For quant traders, this means your strategy code remains clean and exchange-agnostic. For engineering teams, it means one adapter to maintain, one monitoring dashboard, and one incident response playbook. For backtesting pipelines, it means consistent historical data aligned to UTC across all markets.
Next Steps
If you are building a cross-market trading system and want to test the unified gateway:
- Sign up at tickdb.ai (free tier available, no credit card required)
- Generate an API key in the dashboard
- Set the
TICKDB_API_KEYenvironment variable and run the reference client above
If you need 10+ years of historical OHLCV data for strategy backtesting across US equities:
Reach out to enterprise@tickdb.ai for Professional and Enterprise plan details, including full historical data access and priority WebSocket routing.
If you are integrating with an AI coding assistant:
Search for and install the tickdb-market-data SKILL in your AI tool's marketplace to get context-aware TickDB API assistance.
This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results.