At 2:14 AM on a Thursday, a momentum strategy stopped working.
The trading system was still running. Orders were being generated. The execution layer was healthy. What nobody noticed until the morning review was that the market data feed had silently stalled — timestamps advancing, prices frozen, the bid-ask spread showing no movement for 47 minutes. By the time the quant team discovered the gap, the strategy had accumulated a 3.2% drawdown on stale positions.
This is not a hypothetical edge case. In systematic trading operations, data source failures rarely announce themselves with error messages. They manifest as phantom liquidity, frozen order books, and strategies quietly degrading while everything appears operational. Quantifying "the data source is lagging" and setting automated alerts for when latency crosses tolerable thresholds is not optional — it is foundational infrastructure.
This article provides a complete walkthrough for building a latency monitoring dashboard using Prometheus and Grafana. We will define measurable latency thresholds, instrument a Python data consumer with Prometheus metrics, visualize P50/P95/P99 latency distributions, and configure alerting rules that escalate before a stalled feed becomes a trading loss.
1. The Problem with "It Feels Slow"
Latency is not a single number. When traders say "the feed feels slow," they are describing an aggregate perception that encompasses at least four distinct latency components:
| Latency Component | Definition | Typical Range (US Equities) |
|---|---|---|
| Network latency | Time from exchange multicast to NIC receive buffer | 50–500 μs (co-location) to 5–50 ms (remote) |
| Ingestion latency | Time to decode, validate, and write to the local buffer | 100–500 μs per message |
| Processing latency | Time to apply business logic (normalization, enrichment) | 500 μs–5 ms |
| Delivery latency | Time from internal queue to subscriber callback | 1–10 ms |
A system that reports "1 ms latency" may be measuring only one of these components while ignoring the others. A monitoring strategy that tracks only the mean latency (P50) will miss the long-tail events — the P99 spikes — that cause order book snapshots to diverge from ground truth at the worst possible moments.
The discipline of latency monitoring requires three decisions upfront:
- Which component to measure? The full end-to-end path from exchange to application.
- Which percentile to alert on? P50 is optimistic; P95 is reasonable; P99 is where microstructural anomalies hide.
- What threshold constitutes a failure? This depends on your strategy's holding period and execution sensitivity.
2. Architecture Overview: Prometheus + Grafana Stack
The monitoring stack consists of three layers:
┌─────────────────────────────────────────────────────────────┐
│ Data Source Layer │
│ [TickDB WebSocket] ──► [Your Python Consumer] │
└──────────────────────────────┬──────────────────────────────┘
│ Emits metrics
▼
┌─────────────────────────────────────────────────────────────┐
│ Metrics Collection │
│ [prometheus_client Python library] │
│ Exposes: /metrics endpoint on port 8000 │
└──────────────────────────────┬──────────────────────────────┘
│ Scrapes every 15s
▼
┌─────────────────────────────────────────────────────────────┐
│ Time-Series Database │
│ [Prometheus Server] │
│ Stores time-series metrics with labels │
└──────────────────────────────┬──────────────────────────────┘
│ Query
▼
┌─────────────────────────────────────────────────────────────┐
│ Visualization & Alerting │
│ [Grafana] │
│ Dashboards: Latency heatmaps, P99 trends, alert status │
└─────────────────────────────────────────────────────────────┘
Prometheus is pull-based: the server scrapes /metrics endpoints on each target at a configurable interval (default 15 seconds). This eliminates the need for a push gateway in most architectures and ensures that the monitoring system itself does not become a bottleneck.
Grafana connects to Prometheus as a data source and renders the collected metrics into actionable visualizations. Its alerting engine evaluates PromQL queries against live data and fires notifications when thresholds are breached.
3. Instrumenting Your Data Consumer
3.1 Installing the Prometheus Client
pip install prometheus-client
3.2 Defining Metrics
The Prometheus client supports four metric types. For latency monitoring, three are relevant:
| Metric Type | Use Case | Example |
|---|---|---|
| Histogram | Latency distributions with configurable buckets | Message processing time |
| Gauge | Current state values | Active connections, queue depth |
| Counter | Monotonically increasing values | Total messages received, errors |
A histogram is the correct choice for latency because it automatically computes P50, P90, P95, and P99 (configurable via buckets). The following code sets up the metrics infrastructure:
from prometheus_client import Histogram, Gauge, Counter, CollectorRegistry, REGISTRY
import time
# Use a custom registry to avoid conflicts in multi-process deployments
registry = CollectorRegistry()
# Histogram: end-to-end latency from message timestamp to processing completion
LATENCY_HISTOGRAM = Histogram(
"tickdb_message_latency_seconds",
"Time from message timestamp to processing completion",
labelnames=["symbol", "data_type"],
buckets=(0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
registry=registry,
)
# Gauge: current WebSocket connection state (1=connected, 0=disconnected)
WS_CONNECTION_STATE = Gauge(
"tickdb_ws_connected",
"WebSocket connection state (1=connected, 0=disconnected)",
registry=registry,
)
# Gauge: timestamp of the last successfully processed message
LAST_MESSAGE_TIMESTAMP = Gauge(
"tickdb_last_message_ts",
"Unix timestamp of the last successfully processed message",
labelnames=["symbol"],
registry=registry,
)
# Counter: total messages received, processed, and errors
MESSAGES_RECEIVED = Counter(
"tickdb_messages_received_total",
"Total messages received from data source",
labelnames=["symbol", "data_type"],
registry=registry,
)
MESSAGES_ERRORS = Counter(
"tickdb_messages_errors_total",
"Total processing errors",
labelnames=["error_type"],
registry=registry,
)
# Counter: reconnections
RECONNECTION_COUNT = Counter(
"tickdb_reconnections_total",
"Total WebSocket reconnection attempts",
registry=registry,
)
3.3 Instrumenting the WebSocket Consumer
The following class wraps a WebSocket connection and emits Prometheus metrics at each step of the processing pipeline. It is designed to be dropped into an existing consumer with minimal changes to your application logic.
import os
import json
import time
import random
import websocket
import threading
from datetime import datetime, timezone
class InstrumentedTickDBConsumer:
"""
TickDB WebSocket consumer with Prometheus instrumentation.
⚠️ For production HFT workloads, consider aiohttp/asyncio for
non-blocking I/O. This implementation is suitable for strategy
monitoring and backfill validation use cases.
Environment variables required:
TICKDB_API_KEY: Your TickDB API key
TICKDB_WS_URL: WebSocket endpoint (default: wss://api.tickdb.ai/v1/stream)
"""
def __init__(self, symbols: list[str], data_types: list[str] = None):
self.api_key = os.environ.get("TICKDB_API_KEY")
if not self.api_key:
raise ValueError("TICKDB_API_KEY environment variable is not set")
self.ws_url = os.environ.get(
"TICKDB_WS_URL",
"wss://api.tickdb.ai/v1/stream"
)
self.symbols = symbols
self.data_types = data_types or ["kline", "depth"]
self.ws = None
self.running = False
self.retry_count = 0
self.max_retries = 10
self.base_delay = 1.0
self.max_delay = 60.0
def connect(self):
"""Establish WebSocket connection with authentication."""
query_params = f"?api_key={self.api_key}"
self.ws = websocket.WebSocketApp(
f"{self.ws_url}{query_params}",
on_open=self._on_open,
on_message=self._on_message,
on_error=self._on_error,
on_close=self._on_close,
)
self.running = True
# Run in a daemon thread; Ctrl-C will terminate the process
ws_thread = threading.Thread(target=self.ws.run_forever, daemon=True)
ws_thread.start()
def _on_open(self, ws):
"""Subscribe to symbols upon connection establishment."""
WS_CONNECTION_STATE.set(1)
self.retry_count = 0
print(f"[{datetime.now(timezone.utc).isoformat()}] WebSocket connected")
subscribe_payload = {
"cmd": "subscribe",
"params": {
"channels": self.data_types,
"symbols": self.symbols,
}
}
ws.send(json.dumps(subscribe_payload))
def _on_message(self, ws, raw_message):
"""
Process incoming messages and emit Prometheus metrics.
Latency is measured as: time_of_processing - message.server_timestamp
Note: Replace 'server_timestamp' with the actual timestamp field
from TickDB's message schema.
"""
try:
message = json.loads(raw_message)
# Extract message metadata
data_type = message.get("channel", "unknown")
symbol = message.get("symbol", "unknown")
# Measure end-to-end latency
# ⚠️ ADAPT THIS: Use the actual timestamp field from TickDB
# message_ts = message.get("server_timestamp") or message.get("ts")
message_ts = message.get("ts", time.time())
processing_time = time.time() - (message_ts / 1000) # ms to seconds
LATENCY_HISTOGRAM.labels(symbol=symbol, data_type=data_type).observe(
processing_time
)
LAST_MESSAGE_TIMESTAMP.labels(symbol=symbol).set(time.time())
MESSAGES_RECEIVED.labels(symbol=symbol, data_type=data_type).inc()
except json.JSONDecodeError as e:
MESSAGES_ERRORS.labels(error_type="json_decode").inc()
except KeyError as e:
MESSAGES_ERRORS.labels(error_type="missing_field").inc()
except Exception as e:
MESSAGES_ERRORS.labels(error_type="unhandled").inc()
def _on_error(self, ws, error):
"""Log errors and update connection state gauge."""
print(f"WebSocket error: {error}")
WS_CONNECTION_STATE.set(0)
MESSAGES_ERRORS.labels(error_type="websocket_error").inc()
def _on_close(self, ws, close_status_code, close_msg):
"""Handle disconnection with exponential backoff reconnection."""
WS_CONNECTION_STATE.set(0)
print(f"WebSocket closed: {close_status_code} - {close_msg}")
self._reconnect()
def _reconnect(self):
"""
Reconnect with exponential backoff and jitter.
Backoff formula: delay = min(base * 2^retry + jitter, max_delay)
Jitter prevents thundering herd if multiple instances reconnect simultaneously.
"""
if self.retry_count >= self.max_retries:
print(f"Max retries ({self.max_retries}) reached. Giving up.")
self.running = False
return
delay = min(self.base_delay * (2 ** self.retry_count), self.max_delay)
jitter = random.uniform(0, delay * 0.1) # 10% jitter
total_delay = delay + jitter
RECONNECTION_COUNT.inc()
self.retry_count += 1
print(f"Reconnecting in {total_delay:.2f} seconds (attempt {self.retry_count})")
time.sleep(total_delay)
if self.running:
self.connect()
def start(self):
"""Start the consumer."""
print(f"Starting TickDB consumer for symbols: {self.symbols}")
self.connect()
# Keep the main thread alive
while self.running:
time.sleep(1)
def stop(self):
"""Gracefully stop the consumer."""
self.running = False
if self.ws:
self.ws.close()
3.4 Exposing the /metrics Endpoint
To allow Prometheus to scrape your metrics, expose the /metrics endpoint. If you are using FastAPI, this is a single import:
from fastapi import FastAPI
from prometheus_client import make_asgi_app
app = FastAPI()
# Mount the Prometheus metrics endpoint at /metrics
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)
If you are not using a web framework, the start_http_server function provides a standalone server:
from prometheus_client import start_http_server
if __name__ == "__main__":
# Start the metrics server on port 8000
start_http_server(8000)
# Initialize and run the data consumer
consumer = InstrumentedTickDBConsumer(
symbols=["AAPL.US", "NVDA.US"],
data_types=["kline", "depth"]
)
consumer.start()
Prometheus will now scrape http://your-host:8000/metrics every 15 seconds.
4. Configuring Prometheus Scrape Targets
Add the following to your prometheus.yml configuration file:
global:
scrape_interval: 15s # How often to collect metrics
evaluation_interval: 15s # How often to evaluate alerting rules
scrape_configs:
- job_name: "tickdb-consumer"
static_configs:
- targets: ["your-consumer-host:8000"]
relabel_configs:
- source_labels: [__address__]
target_label: instance
regex: "([^:]+):.*"
replacement: "${1}"
Validate your configuration:
promtool check config prometheus.yml
5. Grafana Dashboard Design
5.1 Latency Distribution Panel
Use a heatmap visualization to show latency distribution over time. This reveals patterns that line charts miss — for example, whether latency spikes are isolated events or systemic degradation.
PromQL query for heatmap data:
sum by (le) (
rate(tickdb_message_latency_seconds_bucket{symbol="$symbol"}[$__rate_interval])
)
The le (less-than-or-equal) label buckets the histogram into latency ranges. Configure your heatmap with the same bucket boundaries as your histogram definition.
5.2 P99 Trend Panel
A time-series graph tracking P99 latency over the past 24 hours. P99 is computed from the histogram using the histogram_quantile function.
PromQL query:
histogram_quantile(0.99,
sum by (le) (
rate(tickdb_message_latency_seconds_bucket[$__rate_interval])
)
)
Overlay P50 and P95 on the same graph to show the full distribution shape:
histogram_quantile(0.50, sum by (le) (rate(tickdb_message_latency_seconds_bucket[$__rate_interval])))
histogram_quantile(0.95, sum by (le) (rate(tickdb_message_latency_seconds_bucket[$__rate_interval])))
histogram_quantile(0.99, sum by (le) (rate(tickdb_message_latency_seconds_bucket[$__rate_interval])))
5.3 Staleness Detection Panel
One of the most dangerous failure modes is a feed that appears healthy but has stopped updating. A gauge panel comparing the timestamp of the last received message against current time reveals this:
PromQL query:
time() - tickdb_last_message_ts{symbol="$symbol"}
Label this panel "Seconds Since Last Message." If this value exceeds your expected update frequency (e.g., 5 seconds for US equity level-1 data), the feed is stale.
5.4 Error Rate Panel
A stat panel showing the rate of errors per minute:
sum by (error_type) (rate(tickdb_messages_errors_total[$__rate_interval])) * 60
Break down errors by type to distinguish between transient network issues and systematic decoding failures.
6. Alerting Rules
6.1 Critical Alerts
The following alerting rules should page your on-call engineer immediately. These represent conditions that will corrupt strategy execution if not resolved within minutes.
groups:
- name: tickdb_latency_alerts
rules:
# Alert: P99 latency exceeds 500ms over a 5-minute window
- alert: TickDBP99LatencyHigh
expr: |
histogram_quantile(0.99,
sum by (le) (rate(tickdb_message_latency_seconds_bucket[5m]))
) > 0.5
for: 5m
labels:
severity: critical
annotations:
summary: "TickDB P99 latency exceeds 500ms"
description: "P99 latency is {{ $value | printf \"%.3f\" }}s on {{ $labels.instance }}. Data may be stale."
# Alert: No messages received in 30 seconds (staleness)
- alert: TickDBFeedStale
expr: |
time() - tickdb_last_message_ts > 30
for: 1m
labels:
severity: critical
annotations:
summary: "TickDB feed is stale"
description: "Last message received {{ $value | printf \"%.0f\" }}s ago. WebSocket may be disconnected."
# Alert: WebSocket disconnected
- alert: TickDBWSDisconnected
expr: tickdb_ws_connected == 0
for: 1m
labels:
severity: critical
annotations:
summary: "TickDB WebSocket disconnected"
description: "WebSocket connection to TickDB is down on {{ $labels.instance }}."
# Alert: Error rate exceeds 1% over 5 minutes
- alert: TickDBHighErrorRate
expr: |
sum(rate(tickdb_messages_errors_total[5m]))
/ sum(rate(tickdb_messages_received_total[5m])) > 0.01
for: 5m
labels:
severity: warning
annotations:
summary: "TickDB error rate exceeds 1%"
description: "Error rate is {{ $value | printf \"%.2f\" }}% on {{ $labels.instance }}."
6.2 Setting Thresholds
The thresholds above (500ms P99, 30-second staleness) are starting points. Calibrate them against your strategy's requirements:
| Strategy Type | Latency Sensitivity | Recommended P99 Threshold | Staleness Alert |
|---|---|---|---|
| High-frequency scalping | Extreme | 10–50 ms | 5 seconds |
| Intraday momentum | High | 100–500 ms | 15 seconds |
| Daily mean reversion | Moderate | 1–5 seconds | 60 seconds |
| Overnight position | Low | Not applicable | 5 minutes |
Track these thresholds in your Grafana dashboard as annotation markers on the latency graph. When you change a threshold, the annotation records the reason and the date — invaluable context for post-incident reviews.
7. Dashboard Layout
Organize your Grafana dashboard into four rows:
┌─────────────────────────────────────────────────────────────┐
│ HEADER: Symbol selector (AAPL.US, NVDA.US, BTC/USD) │
├─────────────────────────────────────────────────────────────┤
│ ROW 1: Connection Health │
│ [WS Connected: 0/1] [Reconnections: ###] [Uptime: HH:MM] │
├─────────────────────────────────────────────────────────────┤
│ ROW 2: Latency (time series) │
│ P50 / P95 / P99 latency — 24-hour rolling window │
├─────────────────────────────────────────────────────────────┤
│ ROW 3: Latency (heatmap) │
│ Histogram heatmap — reveals latency distribution patterns │
├─────────────────────────────────────────────────────────────┤
│ ROW 4: Data Quality │
│ [Last Message Age: ##s] [Messages/min] [Error Rate: ##%] │
├─────────────────────────────────────────────────────────────┤
│ ROW 5: Per-Symbol Breakdown │
│ Table: Symbol | P99 | Last Msg | Error Rate | Status │
└─────────────────────────────────────────────────────────────┘
Keep the connection health row at the top. A dashboard that shows latency numbers while the WebSocket is disconnected is worse than useless — it creates false confidence.
8. Operational Considerations
8.1 Cardinality Management
Each unique combination of labelvalues creates a new time series in Prometheus. If you monitor 500 symbols with 3 data types each, you have 1,500 unique series for latency alone. This is manageable, but if you add dynamic labels (e.g., including the exchange venue or order book level), cardinality can grow exponentially.
Rule: Keep label sets fixed and small. If you need per-symbol drill-down, use Grafana's panel repeats feature rather than adding symbol as a label.
8.2 Retention and Storage
Prometheus stores data locally with a configurable retention period (default 15 days). For quant trading use cases, retain at least 30 days of latency data to correlate performance regressions with specific market events.
For longer retention, consider Thanos or Cortex — open-source projects that extend Prometheus with object storage backends (S3, GCS) and global query aggregation across multiple Prometheus instances.
8.3 Multi-Instance Monitoring
If you run multiple consumers (for redundancy or geographic distribution), each instance should emit metrics with an instance label. Grafana's group by feature can then aggregate across instances:
# Average P99 across all instances for a given symbol
avg by (symbol) (
histogram_quantile(0.99, sum by (instance, le) (rate(tickdb_message_latency_seconds_bucket{symbol="AAPL.US"}[5m])))
)
9. Integrating TickDB with Your Monitoring Stack
The code examples above use a generic WebSocket consumer structure that maps directly to TickDB's API. For TickDB-specific integration details, the following endpoints and features are relevant:
| Capability | TickDB Endpoint / Feature | Notes |
|---|---|---|
| WebSocket streaming | wss://api.tickdb.ai/v1/stream?api_key= |
Auth via URL parameter |
| Kline (OHLCV) data | GET /v1/market/kline |
Historical backtesting |
| Order book depth | depth channel |
L1 for US equities |
| Connection health | Native ping/pong | Heartbeat handling in _heartbeat_loop |
| Rate limit handling | code: 3001 + Retry-After |
Exponential backoff |
The Prometheus instrumentation in this article is intentionally framework-agnostic. Drop the InstrumentedTickDBConsumer class into your existing data pipeline, ensure the /metrics endpoint is accessible to your Prometheus server, and your latency dashboard will begin populating within one scrape interval.
10. Closing
The difference between a strategy that degrades silently and one that alerts aggressively is a monitoring layer that treats latency as a first-class metric. A P99 latency that crosses your threshold is not merely a performance issue — it is a signal that your data may be diverging from the market, and every subsequent order is being placed on stale ground.
The monitoring stack described here — Prometheus histograms, Grafana heatmaps, and alert rules with explicit threshold calibration — transforms "the feed feels slow" into "P99 latency exceeded 500ms on AAPL.US for 5 consecutive minutes, triggering a page."
Build the dashboard before you need it. Calibrate thresholds against real market data. Review the alerts after every significant market event. A monitoring system that is not regularly exercised is a monitoring system that will fail when it matters most.
Next Steps
If you are building a data quality monitoring layer for the first time, start with the single-instance deployment described in this article. Add the Grafana dashboard JSON to your infrastructure-as-code repository and treat it as version-controlled code.
If you want to integrate TickDB as your data source: sign up at tickdb.ai to obtain a free API key. The WebSocket streaming endpoint supports the kline and depth channels used in the code examples above.
If you need institutional-grade historical data for backtesting: TickDB provides 10+ years of cleaned, aligned US equity OHLCV data via the /v1/market/kline endpoint, suitable for multi-year strategy validation across bull and bear cycles.
If you are using AI coding assistants for quant development: search for and install the tickdb-market-data SKILL in your AI tool's marketplace to access TickDB API documentation and code generation directly within your workflow.
This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Latency monitoring is a technical infrastructure practice; its effectiveness depends on accurate threshold calibration and disciplined alert response.