The problem with most Python trading systems isn't the strategy. It's the architecture.
I learned this the hard way at 3:47 AM on a Tuesday, staring at a frozen terminal while my mean-reversion strategy missed a 2.3% spread opportunity because the system was busy waiting for one WebSocket feed to reconnect. Three seconds of latency. Gone.
The irony? The same code had worked perfectly in backtesting. The problem wasn't the algorithm. It was that backtests run on static data. Live trading runs on a chaotic stream of market events that will expose every blocking call, every synchronous bottleneck, and every thread you forgot to clean up.
The solution isn't rewriting in C++ or Rust. It's asyncio — Python's built-in concurrency model that lets you handle thousands of market data streams without spawning a thread for each one.
This article dissects how asyncio works, why it matters for quantitative systems, and how to implement production-grade async patterns that won't leave you debugging at 4 AM.
The Synchronous Bottleneck: Why Your Python Trading System Stalls
Before we fix the problem, we need to understand why it exists.
Consider a typical synchronous market data pipeline:
import requests
import time
class SyncMarketData:
def __init__(self, symbols):
self.symbols = symbols
self.api_key = os.environ.get("TICKDB_API_KEY")
def fetch_all_prices(self):
results = []
for symbol in self.symbols: # Sequential, blocking
response = requests.get(
f"https://api.tickdb.ai/v1/market/kline/latest",
headers={"X-API-Key": self.api_key},
params={"symbol": symbol, "interval": "1m"}
)
results.append(response.json())
return results
With 50 symbols, each API call taking 80ms, you're looking at 4 seconds of sequential wait time. Your strategy is idle for 4 seconds while the network drips data one symbol at a time.
Synchronous I/O doesn't just waste time. It blocks the entire event loop of your strategy. While you're waiting for a WebSocket ping response, no other logic executes. No alerts fire. No positions adjust.
The Threading Trap
The obvious fix — spawning a thread per data stream — creates new problems:
| Problem | Impact |
|---|---|
| Memory overhead | Each thread consumes 8–10 MB of stack space |
| GIL contention | Python's Global Interpreter Lock serializes CPU-bound work |
| Context switching | Thousands of threads thrash the scheduler |
| Race conditions | Shared state between threads requires locks, deadlocks, and bugs |
A system handling 100 WebSocket connections with threads consumes 1 GB of RAM just for thread overhead. That's before you load any market data.
Asyncio solves this by running everything on a single thread. There's no GIL problem because there's no parallel CPU work. There's no context switching because there's only one task running at a time — but that task yields control when waiting for I/O, allowing other tasks to run in the meantime.
asyncio Fundamentals: Coroutines, Tasks, and the Event Loop
What asyncio Actually Is
asyncio is not threading. It's not multiprocessing. It's cooperative multitasking on a single thread.
The event loop is a central orchestrator that:
- Runs your coroutines
- Handles I/O waiting (network, disk)
- Switches between coroutines when they're blocked
A coroutine is a function that can pause execution and yield control back to the event loop:
import asyncio
async def fetch_price(symbol):
# When this line runs, asyncio yields to the event loop
# Other coroutines can execute while this one waits
response = await fetch_from_api(symbol)
return parse_price(response)
The await keyword is the critical piece. It tells asyncio: "I'm waiting for something. Go do other work until it's done."
The Event Loop in Practice
Here's how the event loop handles our 50-symbol problem:
import asyncio
async def fetch_price(symbol, api_key):
"""Fetch latest price for a single symbol."""
# In production, you'd use aiohttp here (covered later)
response = await asyncio.to_thread(requests_get, symbol, api_key)
return parse_price(response)
async def fetch_all_prices_async(symbols, api_key):
"""Fetch all prices concurrently — not sequentially."""
tasks = [
fetch_price(symbol, api_key)
for symbol in symbols
]
# asyncio.gather runs all tasks concurrently
results = await asyncio.gather(*tasks)
return dict(zip(symbols, results))
# Execution time comparison:
# Synchronous: 50 symbols × 80ms = 4,000ms
# Async: max(80ms, 80ms, ...) = 80ms (all requests in parallel)
asyncio.gather() is your best friend in async trading systems. It schedules all coroutines concurrently and waits for all of them to complete. The total time equals the slowest individual call, not the sum of all calls.
Tasks vs. Coroutines
A coroutine is a function object. A Task is a wrapper that schedules a coroutine on the event loop:
# Coroutine object — won't run until awaited
price_fetch = fetch_price("AAPL.US", api_key)
# Task — scheduled immediately on the event loop
task = asyncio.create_task(price_fetch)
# Now you can do other work
result = await task # Retrieve result when ready
In production quant systems, you'll wrap most data-fetching coroutines in Tasks so you can:
- Cancel them on shutdown
- Track their status
- Handle exceptions without losing the entire pipeline
Production-Grade Async WebSocket: Beyond the Hello World
Textbook asyncio tutorials end with a simple WebSocket example. Production trading systems need more.
The Full Reconnection Pattern
Network failures are not exceptional events — they're expected. A production WebSocket client must:
- Connect with exponential backoff
- Handle rate limits (TickDB returns
code: 3001) - Detect stale connections via heartbeat
- Gracefully shut down without leaking resources
Here's a complete implementation:
import asyncio
import json
import logging
import random
import time
import os
from typing import Optional, Callable, Dict, Any
import aiohttp
logger = logging.getLogger(__name__)
class AsyncTickDBClient:
"""
Production-grade async WebSocket client for TickDB market data.
Features:
- Exponential backoff with jitter on reconnect
- Heartbeat (ping/pong) for connection health
- Rate limit handling (code 3001)
- Graceful shutdown
- Symbol subscription management
"""
def __init__(
self,
api_key: Optional[str] = None,
base_url: str = "wss://api.tickdb.ai/v1/ws/market",
max_retries: int = 10,
base_delay: float = 1.0,
max_delay: float = 60.0,
heartbeat_interval: float = 30.0,
ping_timeout: float = 10.0,
):
self.api_key = api_key or os.environ.get("TICKDB_API_KEY")
if not self.api_key:
raise ValueError("API key required: set TICKDB_API_KEY environment variable")
self.base_url = f"{base_url}?api_key={self.api_key}"
self.max_retries = max_retries
self.base_delay = base_delay
self.max_delay = max_delay
self.heartbeat_interval = heartbeat_interval
self.ping_timeout = ping_timeout
self._session: Optional[aiohttp.ClientSession] = None
self._ws: Optional[aiohttp.ClientWebSocketResponse] = None
self._running = False
self._retry_count = 0
self._last_pong_time: float = 0
# Message handlers: symbol -> callback function
self._handlers: Dict[str, Callable] = {}
async def connect(self):
"""Establish WebSocket connection with retry logic."""
self._running = True
self._retry_count = 0
while self._running and self._retry_count < self.max_retries:
try:
# Create session if needed
if self._session is None or self._session.closed:
timeout = aiohttp.ClientTimeout(total=10, sock_read=5)
self._session = aiohttp.ClientSession(timeout=timeout)
logger.info(f"Connecting to TickDB WebSocket (attempt {self._retry_count + 1})")
self._ws = await self._session.ws_connect(self.base_url)
self._retry_count = 0 # Reset on successful connection
self._last_pong_time = time.time()
logger.info("Connected to TickDB WebSocket")
# Start background tasks
await asyncio.gather(
self._receive_loop(),
self._heartbeat_loop(),
)
except aiohttp.ClientConnectorError as e:
await self._handle_reconnect(f"Connection error: {e}")
except aiohttp.WSServerHandshakeError as e:
if e.status == 401:
raise ValueError("Invalid API key — check TICKDB_API_KEY")
await self._handle_reconnect(f"Handshake error: {e}")
except asyncio.CancelledError:
logger.info("WebSocket task cancelled")
raise
except Exception as e:
await self._handle_reconnect(f"Unexpected error: {e}")
async def _handle_reconnect(self, error_msg: str):
"""Exponential backoff with jitter before reconnect."""
self._retry_count += 1
# Exponential backoff: delay = min(base * 2^retry, max_delay)
delay = min(self.base_delay * (2 ** self._retry_count), self.max_delay)
# Jitter: add random 0-10% to prevent thundering herd
jitter = random.uniform(0, delay * 0.1)
delay += jitter
logger.warning(f"{error_msg}. Reconnecting in {delay:.1f}s...")
# Clean up stale connection
if self._ws and not self._ws.closed:
await self._ws.close()
await asyncio.sleep(delay)
async def _heartbeat_loop(self):
"""Send ping every heartbeat_interval seconds, verify pong response."""
while self._running and self._ws and not self._ws.closed:
await asyncio.sleep(self.heartbeat_interval)
if not self._running:
break
try:
await self._ws.send_json({"cmd": "ping"})
logger.debug("Ping sent")
# Wait for pong with timeout
try:
msg = await asyncio.wait_for(
self._ws.receive_json(),
timeout=self.ping_timeout
)
if msg.get("cmd") == "pong":
self._last_pong_time = time.time()
logger.debug(f"Pong received, latency OK")
except asyncio.TimeoutError:
logger.warning("Pong timeout — connection may be stale")
# Force reconnect by breaking the loop
break
except Exception as e:
logger.error(f"Heartbeat error: {e}")
break
async def _receive_loop(self):
"""Main message processing loop."""
while self._running and self._ws and not self._ws.closed:
try:
msg = await self._ws.receive_json()
await self._process_message(msg)
except asyncio.TimeoutError:
continue # No message, keep waiting
except aiohttp.ClientError as e:
logger.error(f"Receive error: {e}")
break
except Exception as e:
logger.error(f"Message processing error: {e}")
async def _process_message(self, msg: Dict[str, Any]):
"""Route incoming message to appropriate handler."""
msg_type = msg.get("type") or msg.get("cmd")
data = msg.get("data", {})
symbol = data.get("symbol", "")
# Handle rate limit responses
if msg.get("code") == 3001:
retry_after = int(msg.get("headers", {}).get("Retry-After", 5))
logger.warning(f"Rate limited. Waiting {retry_after}s")
await asyncio.sleep(retry_after)
return
# Route to symbol-specific handler
if symbol in self._handlers:
self._handlers[symbol](data)
elif "" in self._handlers: # Wildcard handler
self._handlers[""](msg)
def subscribe(self, symbols: list, handler: Callable[[Dict], None]):
"""Subscribe to real-time data for given symbols."""
if self._ws and not self._ws.closed:
subscribe_msg = {
"cmd": "subscribe",
"symbols": symbols,
"channels": ["depth", "kline_1m"] # Order book + 1-minute candles
}
asyncio.create_task(self._ws.send_json(subscribe_msg))
logger.info(f"Subscribed to {symbols}")
# Register handlers
for symbol in symbols:
self._handlers[symbol] = handler
async def unsubscribe(self, symbols: list):
"""Unsubscribe from symbols."""
if self._ws and not self._ws.closed:
unsubscribe_msg = {"cmd": "unsubscribe", "symbols": symbols}
await self._ws.send_json(unsubscribe_msg)
logger.info(f"Unsubscribed from {symbols}")
for symbol in symbols:
self._handlers.pop(symbol, None)
async def close(self):
"""Graceful shutdown — cancel all tasks and close connections."""
logger.info("Initiating graceful shutdown...")
self._running = False
if self._ws and not self._ws.closed:
await self._ws.close()
if self._session and not self._session.closed:
await self._session.close()
logger.info("Shutdown complete")
Using the Client
import asyncio
import logging
logging.basicConfig(level=logging.INFO)
async def handle_depth_update(data):
"""Process order book depth update."""
symbol = data.get("symbol")
bids = data.get("b", []) # Bid levels
asks = data.get("a", []) # Ask levels
# Calculate buy/sell pressure ratio
bid_volume = sum(float(level[1]) for level in bids[:5])
ask_volume = sum(float(level[1]) for level in asks[:5])
if ask_volume > 0:
pressure_ratio = bid_volume / ask_volume
print(f"{symbol}: pressure ratio = {pressure_ratio:.2f}")
# Alert on extreme imbalance
if pressure_ratio > 2.5:
print(f"⚠️ {symbol} buy pressure spike detected!")
async def main():
client = AsyncTickDBClient()
# Subscribe to multiple symbols concurrently
symbols = ["AAPL.US", "NVDA.US", "TSLA.US"]
client.subscribe(symbols, handle_depth_update)
try:
await client.connect()
except KeyboardInterrupt:
await client.close()
if __name__ == "__main__":
asyncio.run(main())
Concurrent Data Streams: Handling Multiple Markets
Real quant systems don't watch one market. They watch correlations across equities, options, futures, and crypto. asyncio makes this manageable.
The Aggregator Pattern
import asyncio
from dataclasses import dataclass
from typing import Dict, List
from datetime import datetime
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
@dataclass
class OrderBookSnapshot:
symbol: str
timestamp: datetime
bid_levels: List[tuple] # (price, size)
ask_levels: List[tuple]
@property
def mid_price(self) -> float:
if not self.bid_levels or not self.ask_levels:
return 0.0
return (float(self.bid_levels[0][0]) + float(self.ask_levels[0][0])) / 2
@property
def spread_bps(self) -> float:
if not self.bid_levels or not self.ask_levels:
return 0.0
bid = float(self.bid_levels[0][0])
ask = float(self.ask_levels[0][0])
return (ask - bid) / self.mid_price * 10000
@property
def pressure_ratio(self) -> float:
"""Top-5 bid volume / top-5 ask volume."""
bid_vol = sum(float(l[1]) for l in self.bid_levels[:5])
ask_vol = sum(float(l[1]) for l in self.ask_levels[:5])
return bid_vol / ask_vol if ask_vol > 0 else 0.0
class MultiMarketAggregator:
"""
Aggregates order book data across multiple markets with
cross-market imbalance detection.
"""
def __init__(self, symbols_by_market: Dict[str, List[str]]):
self.symbols_by_market = symbols_by_market
self.order_books: Dict[str, OrderBookSnapshot] = {}
self._lock = asyncio.Lock() # Protect shared state
async def update_book(self, symbol: str, bid_levels: List, ask_levels: List):
"""Thread-safe order book update."""
async with self._lock:
self.order_books[symbol] = OrderBookSnapshot(
symbol=symbol,
timestamp=datetime.now(),
bid_levels=bid_levels,
ask_levels=ask_levels
)
# Trigger cross-market analysis
await self._analyze_imbalances()
async def _analyze_imbalances(self):
"""Detect cross-market pressure divergences."""
if len(self.order_books) < 2:
return
# Calculate average pressure across all symbols
ratios = [ob.pressure_ratio for ob in self.order_books.values()]
avg_pressure = sum(ratios) / len(ratios)
# Flag outliers (>1.5 std deviations from mean)
variance = sum((r - avg_pressure) ** 2 for r in ratios) / len(ratios)
std_dev = variance ** 0.5
for symbol, ob in self.order_books.items():
if abs(ob.pressure_ratio - avg_pressure) > 1.5 * std_dev:
logger.warning(
f"Divergent pressure: {symbol} = {ob.pressure_ratio:.2f} "
f"(avg = {avg_pressure:.2f}, σ = {std_dev:.2f})"
)
def get_spread_report(self) -> str:
"""Generate human-readable spread report."""
lines = ["=== Spread Report ==="]
for symbol, ob in self.order_books.items():
lines.append(
f"{symbol:12} mid={ob.mid_price:8.2f} "
f"spread={ob.spread_bps:6.2f}bps pressure={ob.pressure_ratio:.2f}"
)
return "\n".join(lines)
# Example: Cross-market surveillance
async def run_multi_market():
aggregator = MultiMarketAggregator({
"us_equities": ["AAPL.US", "MSFT.US", "NVDA.US"],
"hk_equities": ["700.HK", "9988.HK"], # Hong Kong depth (up to L10)
})
# In production, you'd integrate with AsyncTickDBClient
# This demonstrates the aggregation logic
await aggregator.update_book(
"AAPL.US",
[("185.50", "10000"), ("185.48", "15000")],
[("185.52", "8000"), ("185.55", "12000")]
)
print(aggregator.get_spread_report())
if __name__ == "__main__":
asyncio.run(run_multi_market())
Task Groups for Structured Concurrency
Python 3.11 introduced TaskGroup, a cleaner way to manage concurrent tasks with automatic cancellation:
async def monitor_market(client: AsyncTickDBClient, symbols: List[str]):
"""Monitor multiple markets with structured concurrency."""
async with asyncio.TaskGroup() as tg:
# Each task is cancelled automatically if any fails
tasks = [
tg.create_task(monitor_single(client, symbol))
for symbol in symbols
]
# All tasks completed or one raised an exception
# Results are in tasks[i].result()
async def monitor_single(client: AsyncTickDBClient, symbol: str):
"""Monitor a single symbol."""
logger.info(f"Starting monitor for {symbol}")
try:
while True:
data = await client.get_latest_depth(symbol)
await process_depth(symbol, data)
await asyncio.sleep(0.1) # 100ms update interval
except asyncio.CancelledError:
logger.info(f"Stopping monitor for {symbol}")
raise
Performance Benchmarks: The 10x Claim Explained
The "10x faster" claim isn't marketing. It's arithmetic.
Sequential vs. Concurrent Request Times
| Scenario | Symbols | Latency per request | Total time |
|---|---|---|---|
| Synchronous sequential | 50 | 80ms | 4,000ms |
| Async concurrent (gather) | 50 | 80ms | 80ms |
| Async with rate limit wait | 50 | 80ms + backoff | ~150ms |
Speedup: 50x on request latency alone.
But the real gains come from WebSocket streaming vs. polling:
| Approach | Update frequency | CPU usage | Memory per symbol |
|---|---|---|---|
| Polling (1 req/sec) | 1 Hz | High (constant HTTP overhead) | ~50 KB |
| WebSocket (push) | 100+ Hz | Low (event-driven) | ~5 KB |
For a 100-symbol portfolio:
- Polling: 100 HTTP requests/second × 50 symbols = 5,000 req/sec
- WebSocket: 1 persistent connection × 100 messages/sec = 1 conn, 100 msg/sec
Where asyncio Doesn't Help
Async I/O accelerates I/O-bound workloads. It provides zero speedup for CPU-bound work:
# This won't speed up with asyncio — it's CPU-bound
async def calculate_indicator(data):
# Pure computation — asyncio can't help here
result = heavy_numpy_operations(data)
return result
# For CPU-bound work, use:
# - multiprocessing (separate processes, bypasses GIL)
# - numpy/vectorized operations (leverage C-level parallelism)
# - numba JIT compilation
The optimal architecture combines async I/O (for market data streaming) with vectorized computation (for indicator calculation).
Error Handling: The Production Reality
Production systems fail. The question is how gracefully they handle failure.
Timeout Patterns
import asyncio
async def fetch_with_timeout(client, symbol, timeout=5.0):
"""Fetch with explicit timeout — prevents indefinite hangs."""
try:
result = await asyncio.wait_for(
client.get_latest_depth(symbol),
timeout=timeout
)
return result
except asyncio.TimeoutError:
logger.error(f"Timeout fetching {symbol} after {timeout}s")
return None
except asyncio.CancelledError:
logger.warning(f"Cancelled fetching {symbol}")
raise
# Parallel fetch with individual timeouts
async def fetch_all_with_timeouts(client, symbols, timeout=5.0):
"""Fetch all symbols, but don't let any single symbol hang the batch."""
tasks = [
fetch_with_timeout(client, symbol, timeout)
for symbol in symbols
]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Handle exceptions gracefully
valid_results = []
for symbol, result in zip(symbols, results):
if isinstance(result, Exception):
logger.error(f"Failed to fetch {symbol}: {result}")
else:
valid_results.append((symbol, result))
return valid_results
Cancellation Safety
When your system shuts down (SIGTERM, Ctrl+C), you need to clean up gracefully:
async def run_with_shutdown(client: AsyncTickDBClient):
"""Run with proper cancellation handling."""
shutdown_event = asyncio.Event()
def signal_handler():
logger.info("Shutdown signal received")
shutdown_event.set()
# Register signal handlers
loop = asyncio.get_running_loop()
for sig in (signal.SIGTERM, signal.SIGINT):
loop.add_signal_handler(sig, signal_handler)
# Run until shutdown
try:
await asyncio.wait_for(
client.connect(),
timeout=None # Wait indefinitely until shutdown
)
except asyncio.CancelledError:
logger.info("Main task cancelled — cleaning up")
await client.close()
Integrating with TickDB: Complete Example
Here's a production-ready integration with TickDB's depth channel:
import asyncio
import logging
import os
from datetime import datetime
from tickdb_client import AsyncTickDBClient
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s"
)
logger = logging.getLogger(__name__)
class LiquidityMonitor:
"""
Real-time liquidity monitoring using TickDB depth channel.
Monitors order book pressure across multiple symbols and
alerts on liquidity anomalies (vacuum or flood).
"""
def __init__(self, symbols: list, pressure_threshold: float = 2.5):
self.symbols = symbols
self.pressure_threshold = pressure_threshold
self.client = AsyncTickDBClient()
self.last_alert_time = {} # Prevent alert spam
# Sliding window for pressure history
self.pressure_history: dict = {s: [] for s in symbols}
self.window_size = 20 # Keep last 20 readings
async def start(self):
"""Start monitoring."""
logger.info(f"Starting liquidity monitor for {self.symbols}")
# Subscribe to depth channel for all symbols
self.client.subscribe(self.symbols, self._on_depth_update)
# Run monitoring loop with periodic reporting
async with asyncio.TaskGroup() as tg:
tg.create_task(self.client.connect())
tg.create_task(self._report_loop())
def _on_depth_update(self, data: dict):
"""Process incoming depth update."""
symbol = data.get("symbol", "")
bids = data.get("b", [])
asks = data.get("a", [])
if not bids or not asks:
return
# Calculate pressure ratio (top-5 levels)
bid_vol = sum(float(l[1]) for l in bids[:5])
ask_vol = sum(float(l[1]) for l in asks[:5])
pressure = bid_vol / ask_vol if ask_vol > 0 else 1.0
# Update history
self.pressure_history[symbol].append(pressure)
if len(self.pressure_history[symbol]) > self.window_size:
self.pressure_history[symbol].pop(0)
# Check for anomaly
if pressure > self.pressure_threshold:
self._alert_buy_pressure(symbol, pressure)
elif pressure < (1 / self.pressure_threshold):
self._alert_sell_pressure(symbol, pressure)
def _alert_buy_pressure(self, symbol: str, pressure: float):
"""Alert on excessive buy-side pressure."""
now = datetime.now()
# Debounce: don't alert more than once per 10 seconds
if symbol in self.last_alert_time:
if (now - self.last_alert_time[symbol]).total_seconds() < 10:
return
self.last_alert_time[symbol] = now
logger.warning(
f"🚨 {symbol}: BUY PRESSURE ALERT — ratio = {pressure:.2f}"
)
def _alert_sell_pressure(self, symbol: str, pressure: float):
"""Alert on excessive sell-side pressure."""
now = datetime.now()
if symbol in self.last_alert_time:
if (now - self.last_alert_time[symbol]).total_seconds() < 10:
return
self.last_alert_time[symbol] = now
logger.warning(
f"🚨 {symbol}: SELL PRESSURE ALERT — ratio = {pressure:.2f}"
)
async def _report_loop(self):
"""Periodic liquidity report."""
while True:
await asyncio.sleep(60) # Report every minute
logger.info("=== Liquidity Report ===")
for symbol in self.symbols:
history = self.pressure_history[symbol]
if history:
avg = sum(history) / len(history)
logger.info(f" {symbol}: avg pressure = {avg:.2f}")
async def main():
# Watchlist: tech sector leaders
symbols = ["AAPL.US", "MSFT.US", "NVDA.US", "GOOGL.US", "META.US"]
monitor = LiquidityMonitor(
symbols=symbols,
pressure_threshold=2.5
)
try:
await monitor.start()
except KeyboardInterrupt:
logger.info("Shutting down...")
await monitor.client.close()
if __name__ == "__main__":
# Requires: pip install aiohttp
# Set: export TICKDB_API_KEY="your_key_here"
asyncio.run(main())
⚠️ Production note: For HFT workloads (>1,000 symbols or <10ms latency requirements), consider aiogrpc or raw socket implementations. asyncio + aiohttp adds ~1-2ms overhead per message, which matters at microsecond timeframes.
Architecture Patterns for Scale
The Fan-Out Pattern
For institutional systems monitoring thousands of symbols:
import asyncio
from typing import List
class FanOutMonitor:
"""
Distributes symbol monitoring across multiple worker tasks.
Prevents any single task from becoming a bottleneck.
"""
def __init__(self, all_symbols: List[str], workers: int = 4):
self.workers = workers
# Partition symbols across workers
chunk_size = (len(all_symbols) + workers - 1) // workers
self.symbol_chunks = [
all_symbols[i:i + chunk_size]
for i in range(0, len(all_symbols), chunk_size)
]
async def run(self):
"""Run all workers concurrently."""
logger.info(f"Starting {self.workers} worker tasks")
async with asyncio.TaskGroup() as tg:
for i, chunk in enumerate(self.symbol_chunks):
tg.create_task(self._worker(i, chunk))
async def _worker(self, worker_id: int, symbols: List[str]):
"""Each worker manages its own client connection."""
client = AsyncTickDBClient()
monitor = LiquidityMonitor(symbols)
try:
await monitor.start()
except asyncio.CancelledError:
logger.info(f"Worker {worker_id} cancelled")
await client.close()
Pipeline Pattern for Data Processing
async def order_book_pipeline(symbol: str, client: AsyncTickDBClient):
"""
Pipeline: fetch → normalize → calculate → alert.
Each stage runs concurrently via asyncio.Queue.
"""
fetch_queue = asyncio.Queue()
calc_queue = asyncio.Queue()
alert_queue = asyncio.Queue()
async def fetch():
async for data in client.depth_stream(symbol):
await fetch_queue.put(data)
async def calculate():
while True:
data = await fetch_queue.get()
normalized = normalize_depth(data)
indicators = calculate_indicators(normalized)
await calc_queue.put((symbol, indicators))
fetch_queue.task_done()
async def alert():
while True:
symbol, indicators = await calc_queue.get()
if indicators.pressure_ratio > 2.5:
logger.warning(f"ALERT: {symbol}")
calc_queue.task_done()
# Run pipeline stages concurrently
async with asyncio.TaskGroup() as tg:
tg.create_task(fetch())
tg.create_task(calculate())
tg.create_task(alert())
Common Pitfalls and How to Avoid Them
Pitfall 1: Blocking Calls in Async Functions
# ❌ WRONG — blocks the entire event loop
async def fetch_data():
time.sleep(5) # BLOCKS!
return requests.get(url)
# ✅ CORRECT — yields to event loop during wait
async def fetch_data():
await asyncio.sleep(5) # YIELDS!
return await async_http_get(url)
Pitfall 2: Forgetting to Await
# ❌ WRONG — task is created but never runs
async def main():
fetch_data() # Missing await!
# ✅ CORRECT
async def main():
await fetch_data()
Pitfall 3: Shared Mutable State
# ❌ WRONG — race condition
counter = 0
async def increment():
global counter
counter += 1 # Not atomic in async context
# ✅ CORRECT — use asyncio.Lock
counter = 0
counter_lock = asyncio.Lock()
async def increment():
global counter
async with counter_lock:
counter += 1
Pitfall 4: Not Handling Exceptions in gather()
# ❌ WRONG — one failure cancels all
results = await asyncio.gather(*tasks)
# ✅ CORRECT — return_exceptions prevents cascade failure
results = await asyncio.gather(*tasks, return_exceptions=True)
for symbol, result in zip(symbols, results):
if isinstance(result, Exception):
logger.error(f"Failed {symbol}: {result}")
Performance Tuning: squeezing Out More Throughput
Connection Pooling
async def setup_client():
"""Configure aiohttp with connection pooling for high throughput."""
connector = aiohttp.TCPConnector(
limit=100, # Max concurrent connections
limit_per_host=20, # Max connections per host
keepalive_timeout=30
)
session = aiohttp.ClientSession(connector=connector)
return session
Message Batching
If your data provider supports it, batch subscription requests:
# Instead of 50 individual subscribe messages:
for symbol in symbols:
await ws.send_json({"cmd": "subscribe", "symbol": symbol})
# Send one batched request:
await ws.send_json({
"cmd": "subscribe_batch",
"symbols": symbols,
"channels": ["depth", "kline_1m"]
})
Backpressure Management
async def process_with_backpressure(queue: asyncio.Queue, max_size=1000):
"""Prevent memory exhaustion when processing can't keep up."""
while True:
# If queue is full, pause fetching until we catch up
if queue.full():
logger.warning("Queue full — applying backpressure")
await asyncio.sleep(1) # Wait for consumer to catch up
item = await queue.get()
await process(item)
queue.task_done()
Conclusion
asyncio doesn't make your strategy smarter. It removes the architectural bottlenecks that prevent your strategy from executing at the speed the market demands.
The key takeaways:
Synchronous I/O is the enemy of latency — every blocking call is time your system isn't working.
WebSocket streaming beats polling — one persistent connection handles hundreds of symbols; polling spawns hundreds of HTTP requests.
Production patterns matter — exponential backoff, heartbeat, rate limit handling, and graceful shutdown aren't optional. They're the difference between a system that survives a network glitch and one that loses positions.
asyncio is not magic — it's cooperative multitasking. Your coroutines must
awaitproperly, handle exceptions, and avoid blocking calls.Combine async I/O with vectorized computation — asyncio handles the network; numpy handles the math.
The 3:47 AM debugging session that inspired this article? It never happened again after I migrated to async architecture. My systems reconnect automatically, handle rate limits gracefully, and process market data orders of magnitude faster.
Your strategy's edge lives in milliseconds. asyncio makes sure you don't waste them.
Next Steps
If you're building a production quant system:
- Sign up at tickdb.ai for a free API key (no credit card required)
- Install the
tickdb-market-dataSKILL in your AI coding assistant - Review the official TickDB WebSocket documentation for your specific asset class (US equities, HK equities, crypto)
If you need high-frequency data streams:
- US equities: TickDB depth channel provides L1 order book data
- HK equities: L1–L10 depth available for deeper book analysis
- Crypto: L1–L10 depth with 100+ Hz update rates
If you're migrating an existing Python system:
Start with the AsyncTickDBClient class in this article as your template. Replace synchronous requests calls with aiohttp, add reconnection logic, and wrap your data pipeline in asyncio.gather().
The event loop is waiting. What are you going to build?
This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Async patterns and performance characteristics described are based on typical Python environments and may vary based on system configuration and network conditions.