Keep Your Trading Strategies Alive: Process Supervision with supervisor and systemd

---

When your mean-reversion strategy crashes at 2:47 AM on a Saturday, the market does not wait. The bid-ask spread widens. Your unfilled orders sit orphaned in the exchange queue. Your competitors' algorithms harvest the liquidity dislocation you were supposed to capture.

This is not a hypothetical. In institutional quant shops, a crashed strategy is a quantifiable P&L event — measured in basis points lost per minute of downtime. The difference between a strategy that survives a segfault silently and one that alerts the on-call engineer at 3 AM lies entirely in process supervision architecture.

This article covers two production-grade solutions for keeping quantitative trading systems alive: supervisor, a user-space process manager ideal for development and small-scale deployments, and systemd, the init system that dominates production Linux servers. You will learn how to configure both, write health check scripts that actually catch failure modes specific to trading systems, and understand which tool belongs where in your infrastructure stack.

---

Why Trading Systems Need Dedicated Process Supervision

A Python script run with nohup python strategy.py & is not supervised. When that process dies — whether from an uncaught exception, a broken WebSocket connection, a memory leak, or a SIGKILL from the OOM killer — nobody knows. The process simply stops. Your strategy exits silently. Your capital sits idle.

Standard daemonization techniques (daemon = True in Python, fork() calls in C) solve the "run in background" problem but not the "survive crashes" problem. A properly supervised process enjoys three guarantees that no manual daemonization can provide:

1. Automatic restart on crash: The supervisor detects process death and re-spawns the child within seconds.
2. Restart limit enforcement: After N consecutive failures in a time window, the supervisor stops restarting and alerts you — preventing infinite crash loops that saturate CPU.
3. Log capture: All stdout and stderr from the supervised process are captured to disk, giving you a forensic record of what happened before the crash.

For quantitative systems specifically, process supervision also provides a natural integration point for health check scripts that verify not just "is the process alive" but "is the strategy actually trading."

---

Part 1: supervisor — The User-Space Process Manager

When to Use supervisor

supervisor is a Python-written process manager that runs entirely in user space. It does not require root access, it works on any Linux system, and its configuration is dead simple. Use supervisor when:

• You are running on a shared VPS or a personal trading server without root.
• You want quick iteration during development — supervisorctl reread and supervisorctl update take effect immediately.
• You need to manage multiple unrelated processes (your trading strategy, a data collection daemon, a notification service) under one tool.

supervisor is not the right choice for a production server that must start your strategies automatically after a server reboot. For that, you need systemd (covered in Part 2).

Installing supervisor

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install -y supervisor

# RHEL / CentOS / Amazon Linux
sudo yum install -y supervisor

# Verify installation
supervisord --version

supervisor consists of two components: supervisord, the daemon that manages child processes, and supervisorctl, the command-line client for controlling them.

Configuration Structure

supervisor reads configuration from /etc/supervisor/conf.d/*.conf on most systems. Each supervised program gets its own [program:x] section.

Minimal Production Configuration

; /etc/supervisor/conf.d/mean-reversion-strategy.conf

[program:mean-reversion-strategy]
; The command to run. Using an absolute path and virtual environment.
command=/home/trader/venv/bin/python /home/trader/strategies/mean_reversion.py
directory=/home/trader/strategies
user=trader
autostart=true
autorestart=true
; Startretries: number of restart attempts within 60 seconds before giving up.
startretries=5
; Exit codes in exitcodes are treated as "expected exits" — no restart.
exitcodes=0
; Stopsignal: signal used to stop the process gracefully.
stopsignal=TERM
stopwaitsecs=30
; Redirect stdout/stderr to log files
stdout_logfile=/var/log/supervisor/mean-reversion-stdout.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/var/log/supervisor/mean-reversion-stderr.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10
; Environment variables
environment=PYTHONUNBUFFERED="1",TICKDB_API_KEY="%(ENV_TICKDB_API_KEY)s"
; Priority: lower numbers start first, higher numbers stop first.
priority=999

Key configuration notes:

• The %(ENV_VAR)s syntax lets you reference environment variables set elsewhere — never hardcode API keys.
• autorestart=true means supervisor restarts the process after any non-zero exit code (unless the exit code is in exitcodes).
• stopwaitsecs=30 gives your strategy 30 seconds to shut down gracefully when you issue supervisorctl stop. If it does not exit, supervisor sends SIGKILL.

Managing supervisor

# Reload configuration after changes
sudo supervisorctl reread
sudo supervisorctl update

# Start, stop, restart a specific program
sudo supervisorctl start mean-reversion-strategy
sudo supervisorctl stop mean-reversion-strategy
sudo supervisorctl restart mean-reversion-strategy

# View status of all programs
sudo supervisorctl status

# View real-time tail of stdout log
sudo supervisorctl tail -f mean-reversion-strategy

Processing Trading-Specific Exit Codes

For quantitative strategies, some exit codes are expected and should not trigger a restart. For example, if your strategy exits deliberately when market conditions are unsuitable:

# mean_reversion.py — strategic exit codes
import sys

EXIT_MARKET_CLOSED = 10
EXIT_RISK_LIMIT_HIT = 11
EXIT_CONFIG_INVALID = 12

def check_market_conditions():
    if not is_trading_hours():
        print("Market closed — strategic exit.")
        sys.exit(EXIT_MARKET_CLOSED)
    if portfolio.risk_limit_exceeded():
        print("Risk limit exceeded — halting strategy.")
        sys.exit(EXIT_RISK_LIMIT_HIT)

Update your supervisor configuration to treat these as non-restart exits:

; Treat strategic exits as normal — do NOT restart
exitcodes=0;10;11;12

Log Rotation for Trading Logs

supervisor logs grow unbounded by default. In production, configure logrotate to manage them:

# /etc/logrotate.d/supervisor-trading
/var/log/supervisor/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 root root
    sharedscripts
    postrotate
        /usr/bin/supervisorctl signal HUP supervisor > /dev/null 2>&1 || true
    endscript
}

---

Part 2: systemd — The Production Standard

When to Use systemd

systemd is the init system on virtually every major Linux distribution (Ubuntu 20.04+, Debian 10+, RHEL 8+, Amazon Linux 2). If you have a production trading server, systemd is almost certainly already running. Use systemd when:

• You need your strategy to start automatically after a server reboot (this is systemd's core job).
• You need fine-grained control over resource limits (memory caps, CPU affinity, I/O priority).
• You are managing a multi-service trading infrastructure (data feed, strategy engine, risk manager, notification service).
• You need integration with system-level alerting (journald, sd_notify, alerting via systemd timers).

Why systemd Is Different from supervisor

supervisor runs alongside your application in user space. systemd replaces the old init system — it is the first process the kernel starts (PID 1), and it owns the lifecycle of everything on the machine.

This means systemd knows things supervisor never will:

• When the server boots (multi-user.target).
• When the power button is pressed.
• The state of the entire machine — not just one process.

For trading systems, this matters because your strategy should start after the network is up and before market open — systemd's dependency ordering makes this explicit and reliable.

Writing a systemd Unit File

Unit files live in /etc/systemd/system/ for system-wide services or ~/.config/systemd/user/ for user-space services. Here is a production-grade unit file for a trading strategy:

; /etc/systemd/system/mean-reversion-strategy.service

[Unit]
; Human-readable description
Description=Mean Reversion Trading Strategy
; Start AFTER networking is ready, file systems are mounted
After=network-online.target local-fs.target
; Stop BEFORE network goes down — allows graceful shutdown
Before=network-pre.target
; If this crashes repeatedly, alert via standard monitoring
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
; Run as the trading user, not root
User=trader
Group=trader

; Working directory
WorkingDirectory=/home/trader/strategies

; The executable — absolute path to venv Python
ExecStart=/home/trader/venv/bin/python /home/trader/strategies/mean_reversion.py

; Environment variables loaded from a secure source
EnvironmentFile=/etc/environment.trading
Environment=PYTHONUNBUFFERED=1

; Graceful shutdown: systemd sends SIGTERM, waits 60 seconds, then SIGKILL
TimeoutStopSec=60
KillMode=process

; Restart policy: always restart on non-zero exit (unless in exitcodes list)
Restart=on-failure
RestartSec=5

; Resource limits — critical for preventing runaway processes
; Memory limit prevents OOM killer from affecting other services
MemoryMax=2G
MemoryHigh=1.5G
; CPU time limit
CPUQuota=200%

; Logging: systemd's journald captures stdout/stderr automatically
; No need for separate log files — query with journalctl
StandardOutput=journal
StandardError=journal
SyslogIdentifier=mean-reversion

; Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/home/trader/strategies/logs /home/trader/strategies/state
PrivateTmp=true
; Deny network access if the strategy does not need it directly
; (If it connects via WebSocket/REST, set this appropriately)
RestrictNamespaces=true

[Install]
; Start automatically on boot after multi-user target
WantedBy=multi-user.target

Resource Limits for Trading Strategies

The MemoryMax and MemoryHigh directives in systemd deserve special attention for quantitative systems. A memory leak in a long-running strategy can consume gigabytes of RAM — without a hard limit, the Linux OOM killer will terminate your process at an unpredictable moment, potentially mid-trade. With MemoryHigh, systemd starts throttling memory allocation before hitting the hard MemoryMax limit, giving you time to investigate.

# Check memory consumption of your strategy
systemctl status mean-reversion-strategy

# View real-time logs
journalctl -u mean-reversion-strategy -f

# View logs since last restart
journalctl -u mean-reversion-strategy --since "1 hour ago"

Enabling and Starting the Service

# Reload systemd to pick up new unit files
sudo systemctl daemon-reload

# Enable: start automatically on boot
sudo systemctl enable mean-reversion-strategy

# Start now
sudo systemctl start mean-reversion-strategy

# Check status
sudo systemctl status mean-reversion-strategy

# Stop, restart
sudo systemctl stop mean-reversion-strategy
sudo systemctl restart mean-reversion-strategy

# View recent log entries
journalctl -u mean-reversion-strategy -n 50 --no-pager

Health Checks with systemd

systemd supports native health checks via the ExecStartPre and ExecStartPost directives. You can also use a watchdog mechanism for deeper liveness verification.

Basic Pre-Start Health Check

# Create a pre-start health check script
# /home/trader/scripts/pre-start-check.sh
#!/bin/bash
set -euo pipefail

# Check that the market data API is reachable
curl -sf --max-time 5 https://api.tickdb.ai/v1/symbols/available > /dev/null
if [ $? -ne 0 ]; then
    echo "ERROR: Cannot reach TickDB API — aborting strategy start"
    exit 1
fi

# Check that log directory is writable
if [ ! -w /home/trader/strategies/logs ]; then
    echo "ERROR: Log directory not writable"
    exit 1
fi

# Check available disk space (require at least 5GB)
AVAILABLE=$(df -BG /home/trader/strategies | awk 'NR==2 {print $4}' | tr -d 'G')
if [ "$AVAILABLE" -lt 5 ]; then
    echo "ERROR: Less than 5GB disk space available"
    exit 1
fi

echo "Health checks passed — starting strategy"
exit 0

Add this to your unit file:

ExecStartPre=/home/trader/scripts/pre-start-check.sh

Watchdog-Based Liveness Monitoring

systemd's built-in watchdog (WatchdogSec) sends a ping to your process and expects a pong back via sd_notify. If your process does not respond, systemd restarts it.

# mean_reversion.py — watchdog support
import socket
import os
import time

# Use sd_notify to communicate with systemd
def sd_notify(state, message=""):
    notify_socket = os.environ.get("NOTIFY_SOCKET")
    if not notify_socket:
        return
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    msg = f"STATUS={message}\n{state}"
    sock.sendto(msg.encode(), notify_socket)
    sock.close()

# Tell systemd we are ready
sd_notify("READY=1", "Strategy initialized, connecting to market data")

# Send watchdog ping every 30 seconds (must be less than WatchdogSec in unit file)
last_ping = time.time()
while True:
    try:
        run_strategy_iteration()
        # Check position health
        if not check_positions_healthy():
            sd_notify("STATUS=Critical: Position check failed", "WATCHDOG=1")
            raise RuntimeError("Position health check failed")
        
        # Send watchdog ping every 30 seconds
        if time.time() - last_ping > 25:
            sd_notify("STATUS=Running", "WATCHDOG=1")
            last_ping = time.time()
        
        time.sleep(1)
    except Exception as e:
        sd_notify("STATUS=Exception: " + str(e), "ERRNO=1")
        raise

Update the unit file to enable the watchdog:

; Send watchdog pings every 30 seconds; restart if no pong within 60 seconds
WatchdogSec=60

---

Part 3: Writing Effective Health Check Scripts

Health check scripts for trading systems must answer a deeper question than "is the process alive?" They must answer "is the strategy actually functioning?"

A process can be alive but broken — connected to the WebSocket, receiving ticks, but failing silently because a calculation error causes it to place orders with zero quantity.

Multi-Layer Health Check Architecture

#!/bin/bash
# /home/trader/scripts/health-check.sh
# Exit 0 = healthy, Exit 1 = unhealthy, Exit 2 = degraded (restart not needed)

set -o pipefail

STRATEGY_NAME="mean-reversion-strategy"
LOG_DIR="/home/trader/strategies/logs"
STATE_FILE="$LOG_DIR/strategy_state.json"

# Layer 1: Is the process running?
if ! systemctl is-active --quiet "$STRATEGY_NAME"; then
    echo "CRITICAL: Process not running"
    exit 1
fi

# Layer 2: Did it crash recently? Check journal for restarts
CRASH_COUNT=$(journalctl -u "$STRATEGY_NAME" --since "1 hour ago" \
    | grep -c "Strategy terminated" || true)
if [ "$CRASH_COUNT" -gt 3 ]; then
    echo "CRITICAL: $CRASH_COUNT restarts in the last hour"
    exit 1
fi

# Layer 3: Is the state file being updated?
if [ -f "$STATE_FILE" ]; then
    LAST_UPDATE=$(stat -c %Y "$STATE_FILE" 2>/dev/null)
    NOW=$(date +%s)
    AGE=$((NOW - LAST_UPDATE))
    if [ "$AGE" -gt 300 ]; then  # 5 minutes
        echo "WARNING: State file not updated in $AGE seconds"
        exit 2  # Degraded — do not restart, but alert
    fi
fi

# Layer 4: Check position consistency
if [ -f "$STATE_FILE" ]; then
    ORDERS_COUNT=$(jq -r '.open_orders // 0' "$STATE_FILE" 2>/dev/null)
    if [ "$ORDERS_COUNT" -gt 50 ]; then
        echo "WARNING: Unusually high open order count: $ORDERS_COUNT"
        exit 2
    fi
fi

# Layer 5: Log file anomaly detection
STDERR_LOG="/var/log/supervisor/mean-reversion-stderr.log"
if [ -f "$STDERR_LOG" ]; then
    # Count ERROR entries in the last 5 minutes
    ERROR_COUNT=$(tail -n 1000 "$STDERR_LOG" | \
        awk -v since="$(date -d '5 minutes ago' +%s)" \
        '/ERROR/ && $1 > since {count++} END {print count+0}' 2>/dev/null)
    if [ "$ERROR_COUNT" -gt 20 ]; then
        echo "WARNING: $ERROR_COUNT errors in recent logs"
        exit 2
    fi
fi

echo "OK: All health checks passed"
exit 0

Integrating Health Checks into Monitoring

For production systems, wire the health check into your alerting pipeline:

# check_health.py — called by cron or monitoring system
import subprocess
import requests
import os

def run_health_check():
    result = subprocess.run(
        ["/home/trader/scripts/health-check.sh"],
        capture_output=True, text=True
    )
    
    status = "HEALTHY" if result.returncode == 0 else \
             "DEGRADED" if result.returncode == 2 else "UNHEALTHY"
    
    # Send to monitoring (example: generic webhook)
    webhook_url = os.environ.get("ALERT_WEBHOOK_URL")
    if webhook_url and result.returncode != 0:
        message = f"[{status}] Mean Reversion Strategy\n{result.stdout}"
        requests.post(
            webhook_url,
            json={"text": message},
            timeout=5
        )
    
    # Auto-restart only for truly dead processes
    if result.returncode == 1:
        print("CRITICAL: Attempting automatic restart")
        subprocess.run(["sudo", "systemctl", "restart", "mean-reversion-strategy"])

if __name__ == "__main__":
    run_health_check()

Add to crontab for 5-minute health checks:

# /etc/cron.d/trading-health-checks
*/5 * * * * root /home/trader/venv/bin/python /home/trader/scripts/check_health.py >> /var/log/trading-health.log 2>&1

---

supervisor vs. systemd: Decision Guide

Criteria	supervisor	systemd
Auto-start on boot	No (by itself)	Yes (native)
User-space only (no root)	Yes	No (system services need root)
Configuration complexity	Low	Medium
Log management	Manual (redirect to files)	Native journald
Resource limits	Basic (ulimit in config)	Extensive (MemoryMax, CPUQuota, I/O priority)
Watchdog support	No (add-on only)	Native sd_notify
Dependency ordering	Manual (priority field)	Native After= / Requires=
Best for	Dev/staging, shared hosting	Production servers

The pragmatic answer: Use supervisor during development on your local machine or a cheap VPS. Use systemd on every production server. Many teams run both — supervisor for quick local iteration, systemd as the authoritative production supervisor.

---

A Note on Strategy-Specific Considerations

The configurations above assume a single-process strategy. If your trading infrastructure spans multiple processes — a data ingestion daemon, a risk engine, a strategy executor, and a GUI dashboard — systemd's native support for template units (services ending in @.service) handles dynamic instance management elegantly:

; Scale N strategy instances automatically
; Start with: systemctl start strategy@1 strategy@2 strategy@3
; Each instance gets its own config via %i
ExecStart=/home/trader/venv/bin/python /home/trader/strategies/strategy_runner.py %i

For multi-process architectures, also consider process groups in supervisor or systemd's Slice= directives to isolate resource consumption and apply security boundaries between components.

---

Closing

The difference between a quant strategy that crashes once and never recovers and one that bounces back within seconds is not a more robust programming language or a better exception handler. It is supervision architecture.

supervisor gives you a quick, portable way to keep your strategy alive during development. systemd gives you a production-grade solution that survives server reboots, enforces resource limits, and integrates with your system's alerting infrastructure.

Write the health check scripts first. A strategy that is monitored but not supervised can at least alert you before the damage compounds. A strategy that is supervised but not monitored is better — it fixes itself — but you still want to know when it is struggling.

This article does not constitute investment advice. Automated trading systems involve significant risk, including the risk of total capital loss. Ensure adequate testing and risk controls are in place before deploying any strategy to production.

---

Next Steps

If you are setting up a development environment, install supervisor, write your first unit file, and run your strategy with autorestart=true. Validate that it actually restarts after a kill -9.

If you are moving to production, migrate to systemd. Write the health check scripts in parallel. Configure WatchdogSec and sd_notify. Set MemoryMax to a value that gives you headroom but prevents runaway memory growth.

If you want to go deeper on monitoring, explore Prometheus node_exporter metrics for system-level visibility, Grafana dashboards for strategy-level health, and alert routing via PagerDuty or OpsGenie for 3 AM coverage.

If you use AI coding tools, search for the tickdb-market-data SKILL in your tool's marketplace to get production-ready code templates for connecting your supervised strategies to TickDB's market data infrastructure.