Keeping Your Quant Strategies Alive: Process Supervision with Supervisor and systemd

---

The 3 AM Wake-Up Call

At 3:47 AM on a Tuesday, your phone buzzes. A margin call. You fumble for your laptop, SSH into your server, and find the Python process running your pairs trading strategy dead — killed silently by an out-of-memory error triggered by a data spike that your exception handler never caught. By the time you reconnect and restart the strategy, the arbitrage window has closed. You lost $12,000 in six minutes of downtime.

This is not a hypothetical. It is the most common post-mortem in systematic trading: a process failure that goes unnoticed until the damage is already done. The irony is that preventing this class of failure costs less than one bad trade. This article covers two battle-tested tools for process supervision — supervisor and systemd — and shows you exactly how to configure both for a live quantitative trading system.

---

Why Quant Processes Are Fragile by Design

A production quant system sits at the intersection of unreliable inputs: network connections to market data feeds, exchange APIs with rate limits, broker connections with intermittent failures, and downstream databases that can lag or corrupt data. Each of these failure points can trigger an exception that your strategy does not handle gracefully.

Consider the typical failure taxonomy for a live quant system:

Failure mode	Frequency	Mean time to detection (typical)	Impact
Out-of-memory kill	Daily to weekly	Minutes to hours (if at all)	Strategy halts, positions unmanaged
Data feed disconnection	Hourly to daily	Seconds to minutes	Stale prices, wrong signals
Unhandled exception in strategy loop	Daily	Minutes to hours	Strategy stops or enters invalid state
Broker API timeout	Weekly	Minutes	Orders stuck, fills missed
Server reboot (intentional or crash)	Monthly	Minutes to hours	Complete outage until human intervenes

None of these failures require sophisticated tooling to prevent. They require process supervision: a watchdog that detects when a process dies, restarts it, and logs what happened. Both supervisor and systemd serve this purpose. They are not mutually exclusive — many production setups run supervisor inside a systemd unit — but they address different layers of the infrastructure.

---

Core Concepts: What Process Supervision Does

Before diving into configuration, establish a clear mental model. Process supervision performs four functions:

1. Process lifecycle management — Start, stop, and restart your strategy process.
2. Failure detection — Detect abnormal process termination (crash, OOM kill, signal termination).
3. Automatic restart — Pull the process back up on failure, optionally with a backoff delay.
4. Logging and stdout capture — Redirect process output to files so you can inspect what happened.

Both supervisor and systemd provide all four. The difference lies in the operational model: supervisor runs in user space as a regular daemon, while systemd is the init system — it starts at boot and manages processes with root-level privileges. This distinction drives the configuration trade-offs.

---

Supervisor: User-Space Process Supervision

When Supervisor Makes Sense

Supervisor is the better choice when you are running as a non-root user, want a simple configuration syntax, or need to manage multiple processes with a single tool. It is also the default choice when your server runs inside a container or VM where you do not have access to systemd.

Installation

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y supervisor

# Verify installation
supervisord --version

Configuration

Supervisor's configuration lives in /etc/supervisor/conf.d/. Each process gets its own configuration file. Here is a production-grade configuration for a pairs trading strategy:

; /etc/supervisor/conf.d/pairs-trading-strategy.conf

[program:pairs-trading-strategy]
; Command to execute. Using absolute paths is critical.
command=/home/quant/venv/bin/python /home/quant/strategies/pairs_trading.py

; Working directory — log files are written here by default
directory=/home/quant/strategies

; Run as a non-root user for security
user=quant

; Auto-restart policy
autorestart=true
; Treat unexpected exit codes as errors (exit code 0 is normal termination)
exitcodes=0

; Restart delay: start with 10s, increase up to 60s on repeated failures
startsecs=10
startretries=3
; Number of seconds between successive restarts before supervisor gives up
; 0 means "always restart"
; Setting a limit prevents infinite crash loops that could hammer the exchange
stopwaitsecs=60

; Environment variables — API keys should NEVER be in the command line
environment=TICKDB_API_KEY="%(ENV_TICKDB_API_KEY)s",LOG_LEVEL="INFO"

; Redirect stdout and stderr to log files
stdout_logfile=/var/log/supervisor/pairs-trading-stdout.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/var/log/supervisor/pairs-trading-stderr.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10

; Send SIGTERM (not SIGKILL) on stop — allows graceful shutdown
stopasgroup=true
killasgroup=true

; Priority in startup order (lower number = starts first)
priority=999

The Critical Configuration Details

Several options in this file are frequently misconfigured in quant setups:

• command: Always use an absolute path to the virtual environment's Python binary. Using python without a path resolves to whatever python is in $PATH, which may not be the version you tested against.
• environment: Storing API keys in environment variables rather than the command line prevents keys from appearing in process listings (ps aux) and log files.
• stopasgroup / killasgroup: Ensures that when supervisor stops the process, it terminates the entire process group. Without this, child processes spawned by your strategy (data fetchers, logging threads) can become orphaned zombies.
• stopwaitsecs: The default value of 10 seconds is too short for strategies that maintain open orders. Give the strategy time to cancel resting orders before it is killed.

Managing Supervisor

# Reload configuration after editing
sudo supervisorctl reread
sudo supervisorctl update

# Common operations
sudo supervisorctl status pairs-trading-strategy
sudo supervisorctl stop pairs-trading-strategy
sudo supervisorctl start pairs-trading-strategy
sudo supervisorctl restart pairs-trading-strategy

# View recent logs
sudo supervisorctl tail pairs-trading-strategy stderr

Health Check Script for Supervisor

Supervisor can use an external health check to determine if a process is healthy beyond just "is it running." This is essential for quant strategies, where a process may be alive but stuck in an infinite loop or producing garbage signals.

#!/bin/bash
# /home/quant/health_check.sh
# Health check for pairs trading strategy

# Check 1: Is the process running?
if ! supervisorctl status pairs-trading-strategy | grep -q "RUNNING"; then
    echo "Process not running"
    exit 1
fi

# Check 2: Is the strategy producing log output?
# If the last log entry is older than 5 minutes, the loop may be stuck
LAST_LOG=$(stat -c %Y /var/log/supervisor/pairs-trading-stdout.log 2>/dev/null)
NOW=$(date +%s)
LOG_AGE=$((NOW - LAST_LOG))

if [ "$LOG_AGE" -gt 300 ]; then
    echo "No log output in $LOG_AGE seconds — possible loop deadlock"
    exit 1
fi

# Check 3: Is the strategy connected to the data feed?
# Ping the internal health endpoint (assumes your strategy exposes one)
HEALTH_RESPONSE=$(curl -s --max-time 5 http://localhost:8080/health 2>/dev/null)

if [ "$HEALTH_RESPONSE" != "OK" ]; then
    echo "Health endpoint returned: $HEALTH_RESPONSE"
    exit 1
fi

echo "All checks passed"
exit 0

Integrate this with supervisor using a separate monitor process or with systemd as a timer.

---

systemd: Boot-Level Process Supervision

When systemd Makes Sense

systemd is the right tool when you need your strategy to start at boot without human intervention, when you want system-level resource limits (CPU, memory, file descriptors), or when you need to coordinate startup order between your strategy and its dependencies (Redis, a database, a reverse proxy).

systemd is also the only choice if your server runs on a major cloud provider's managed instances — AWS EC2, GCP Compute Engine, and Azure VMs all use systemd as the init system.

The systemd Unit File

; /etc/systemd/system/quant-pairs-trading.service

[Unit]
Description=Pairs Trading Strategy
After=network-online.target
Wants=network-online.target
; Start after dependencies — e.g., Redis for order state
After=redis.service
Wants=redis.service

[Service]
Type=simple
; Run as non-root user
User=quant
Group=quant

; Working directory
WorkingDirectory=/home/quant/strategies

; The actual command
ExecStart=/home/quant/venv/bin/python /home/quant/strategies/pairs_trading.py

; Environment
Environment="TICKDB_API_KEY=%(ENV_TICKDB_API_KEY)s"
Environment="LOG_LEVEL=INFO"

; Restart policy: always restart on non-zero exit, but cap restart frequency
Restart=on-failure
RestartSec=10
; Prevent restart storms: maximum 5 restarts in 60 seconds
StartLimitIntervalSec=60
StartLimitBurst=5

; Resource limits — critical for preventing OOM crashes
MemoryMax=2G
MemoryHigh=1.8G
LimitNOFILE=65536

; Logging: journald captures stdout/stderr automatically
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pairs-trading

; Graceful shutdown: give strategy 30 seconds to cancel orders and flush state
TimeoutStopSec=30

; Restart on filesystem changes (useful during deployments)
; WatchPath=/home/quant/strategies/
; ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

The Critical systemd Options for Trading Systems

Option	Purpose	Common mistake
Restart=on-failure	Restart on crash but not on clean exit	Using Restart=always — this causes a restart loop on strategies that intentionally exit
RestartSec=10	Wait 10 seconds before restarting	Restarting too fast, flooding the exchange with reconnect attempts
StartLimitBurst=5	After 5 restarts in 60s, stop trying	Not setting this — a crashing strategy can hammer an exchange's rate limits indefinitely
TimeoutStopSec=30	Allow 30 seconds for graceful shutdown	Setting this too low — a strategy with open orders needs time to cancel them before being killed
MemoryMax=2G	Hard cap on memory usage	Not setting this — an OOM kill is silent; MemoryMax triggers a controlled restart with a journal log
After=network-online.target	Wait for the network to be fully up before starting	Starting before the network is ready, then failing every data connection

Managing systemd

# Reload unit files after editing
sudo systemctl daemon-reload

# Enable to start at boot
sudo systemctl enable quant-pairs-trading.service

# Start / stop / restart
sudo systemctl start quant-pairs-trading.service
sudo systemctl stop quant-pairs-trading.service
sudo systemctl restart quant-pairs-trading.service

# Status with recent log entries
sudo systemctl status quant-pairs-trading.service
sudo journalctl -u quant-pairs-trading.service -n 50 --no-pager

# View restart history — tells you how many times the process has crashed
sudo systemctl show quant-pairs-trading.service -p NRestarts,LastExitStatus

Configuring a Watchdog Timer

systemd can be configured to watch for process stalls using its built-in watchdog mechanism. Your strategy must send a periodic "I am alive" signal to systemd. If the signal stops, systemd kills and restarts the process.

[Service]
# Send watchdog keep-alive ping every 30 seconds
WatchdogSec=30
NotifyAccess=main

In your strategy code, signal systemd:

import socket
import struct
import os

# Use sd_notify to tell systemd the process is alive
def watchdog_ping():
    """Send watchdog ping to systemd. Call this every 30 seconds."""
    notify_socket = os.environ.get("NOTIFY_SOCKET")
    if not notify_socket:
        return  # Not running under systemd

    # Send WATCHDOG=1 to indicate healthy state
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    try:
        sock.sendto(b"WATCHDOG=1\n", notify_socket)
    finally:
        sock.close()

# In your main loop:
# while True:
#     run_strategy_iteration()
#     watchdog_ping()
#     time.sleep(1)  # or your actual loop interval

---

Supervisor vs systemd: A Direct Comparison

Dimension	Supervisor	systemd
Scope	User-space daemon	Init system (kernel-adjacent)
Boot start	Manual (rc.local or cron)	Native, via enable
Configuration syntax	INI-style, human-readable	INI-style unit files
Resource limits	Via ulimit in wrapper script	Native (MemoryMax, LimitNOFILE)
Watchdog	DIY via health check script	Built-in (WatchdogSec)
Process groups	Requires stopasgroup	Automatic via cgroup management
Log rotation	Manual (logrotate)	Automatic via journald
Dependency management	Implicit ordering only	Full dependency graph with After=/Wants=
Cluster management	Not natively	No, but works with systemd-machined
Best for	Shared hosting, containers, non-root users	Dedicated servers, boot recovery, cloud VMs

For most quant trading deployments on cloud VMs, systemd is the primary choice and supervisor runs inside it as a supplementary layer for managing multi-process strategies (e.g., a data fetcher, strategy engine, and order router as separate supervisor-managed programs within a single systemd unit).

---

Deployment Guide by Scenario

Single Server, Non-Root User

Use supervisor as the sole process supervisor. Configure it to start on boot via cron:

# Add to crontab: @reboot /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
crontab -e

Dedicated Cloud VM (AWS EC2, GCP, Azure)

Use systemd as the primary supervisor. This ensures your strategy starts immediately after the operating system boots — even if you connect via SSM, serial console, or recovery mode.

# Deploy the unit file
sudo cp quant-pairs-trading.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable quant-pairs-trading.service
sudo systemctl start quant-pairs-trading.service

Docker Container Running a Strategy

Use supervisor inside the container (not systemd — containers typically run a single init process). The Dockerfile installs supervisor, and a supervisord.conf manages multiple processes:

# /etc/supervisor/conf.d/supervisord.conf (inside container)
[supervisord]
nodaemon=true
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
loglevel=info

[program:data-fetcher]
command=/app/data_fetcher.py
autorestart=true
stdout_logfile=/var/log/supervisor/data-fetcher.log

[program:strategy]
command=/app/strategy.py
autorestart=true
stdout_logfile=/var/log/supervisor/strategy.log
; Start after data fetcher is running
priority=100

Build the container with:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y supervisor
COPY --chown=quant:quant /app /app
USER quant
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"]

Monitoring the Watchdogs

Regardless of which tool you use, configure alerting on restart events. Supervisor logs restarts to /var/log/supervisor/supervisord.log; systemd logs them to journald. Set up a simple alert:

# Alert on restart — check for restart events in systemd journal
import subprocess
import requests

def check_restart_events(service_name="quant-pairs-trading"):
    result = subprocess.run(
        ["journalctl", "-u", service_name, "-p", "err", "--since", "1 hour", "--no-pager"],
        capture_output=True, text=True
    )

    if "StartLimitBurst" in result.stdout or "restart" in result.stdout.lower():
        message = f"⚠️ {service_name} restarted in the last hour.\n\n{result.stdout[-500:]}"
        # Send to Slack, PagerDuty, or email
        requests.post(
            "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
            json={"text": message},
            timeout=5
        )

---

The Orderly Shutdown Problem

Both supervisor and systemd send SIGTERM by default when stopping a process. For a quant strategy, SIGTERM creates a critical race condition: the strategy receives the termination signal, but it still has open orders in the market. If it exits before canceling those orders, you are left with unmanaged risk.

The correct pattern is a two-stage shutdown:

1. Signal the strategy to stop accepting new orders and begin canceling resting orders.
2. Wait for order state to reach zero (or a timeout), then exit.
3. If the timeout expires, force-kill with SIGKILL and alert.

import signal
import sys
import threading

cancel_in_progress = threading.Event()
orders_cleared = threading.Event()

def shutdown_handler(signum, frame):
    """Called on SIGTERM. Initiates graceful order cancellation."""
    print(f"Received signal {signum} — initiating graceful shutdown")
    cancel_in_progress.set()

    # Wait up to 25 seconds for orders to clear (TimeoutStopSec=30 in systemd)
    if not orders_cleared.wait(timeout=25):
        print("WARNING: Orders still open after timeout — forcing exit")
        sys.exit(1)

    print("All orders cleared — shutting down")
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown_handler)

# In your order management loop:
# def cancel_all_orders():
#     cancel_in_progress.set()
#     # ... cancel resting orders via broker API
#     orders_cleared.set()

This pattern is what TimeoutStopSec=30 in the systemd unit file is designed to accommodate. Without it, systemd sends SIGKILL 90 seconds after SIGTERM, which is far too long to wait before your strategy can restart.

---

Closing

A process that dies without a watchdog is not a system — it is a script. The difference between a reliable quant deployment and an expensive outage is three things: a supervisor that restarts the process, a resource cap that prevents OOM cascades, and a shutdown handler that closes positions before it exits.

Both supervisor and systemd give you these guarantees. Supervisor is simpler and portable. systemd is more powerful and boot-aware. The best production setups use both — systemd as the init system that survives server reboots, supervisor managing the individual strategy components inside a container or user session.

The configuration examples in this article are production-ready. Copy them, adapt them to your strategy, set your TICKDB_API_KEY environment variable, and deploy. The 3 AM wake-up call becomes far less likely.

---

Next Steps

If you're building your infrastructure from scratch, start with systemd on your server and the supervisor configuration for your development environment. The two systems are complementary, not competing.

If you want a data feed that survives the same failure modes as your strategy, TickDB provides a WebSocket-based market data API with native reconnection handling on the client side — compatible with any supervisor or systemd setup. Sign up at tickdb.ai (free tier, no credit card required).

If you're running multiple strategies and need a control plane, explore TickDB's batch data retrieval endpoint for historical backtesting — your development environment should mirror your production environment as closely as possible.

---

This article does not constitute investment advice. Trading strategies involve risk; process supervision mitigates operational risk but does not eliminate market risk. Past performance does not guarantee future results.