"At 3:17 AM on a Tuesday, a memory leak in the order execution module caused the Python process to exit silently. By the time the trader checked the dashboard at 7:00 AM, three hours of opportunity had evaporated. No alert fired. No process restarted. The strategy was simply... gone."

This scenario is not hypothetical. In systematic trading, the gap between strategy failure and human detection is measured in lost P&L. A single unmonitored crash during a volatile pre-market window can cost more than a week of development time. The solution is not a better exception handler inside your strategy code — it is an infrastructure layer that sits outside your process and treats it as a resource to be managed.

This article covers the two dominant process guardians on Linux systems: supervisor and systemd. You will learn how to configure each for quantitative trading workloads, write health-check scripts that detect the failure modes specific to trading systems, and design a restart policy that balances availability against the risk of hammering a broken system with restarts.

Why Your Strategy Needs an External Guardian

Your trading strategy runs as one or more processes: a data ingestion feed, a signal computation engine, an order management module, and a risk guard. Each can fail independently. Python's built-in exception handling catches errors inside the process, but it cannot recover from segfaults, OOM (out-of-memory) kills, or the entire process being terminated by the kernel's OOM killer.

An external process guardian solves three problems that in-process error handling cannot:

Problem What happens without a guardian What a guardian does
Crash Process exits, strategy stops Detects exit, restarts process immediately
OOM kill Kernel terminates process, no graceful shutdown Process reaped; guardian sees non-zero exit and restarts
Server reboot Strategy does not restart automatically Guardian starts strategy as a service on boot
Stuck process Process alive but unresponsive (deadlock, infinite loop) Health check script detects unresponsiveness, kills and restarts

The guardian does not know what your strategy does. It only manages the process lifecycle. This separation of concerns is intentional — it means the guardian works regardless of what language your strategy is written in or which APIs it calls.

Architecture Overview: Where the Guardian Fits

Before diving into configuration, establish the mental model. A production quant system typically has three layers:

┌─────────────────────────────────────────────┐
│  Layer 3: Monitoring & Alerting             │
│  (Grafana dashboards, Slack/PagerDuty)     │
├─────────────────────────────────────────────┤
│  Layer 2: Process Guardian                  │
│  (supervisor or systemd)                   │
│  - Restarts crashed processes               │
│  - Runs health-check scripts               │
│  - Manages logging rotation                │
│  - Handles server reboot recovery          │
├─────────────────────────────────────────────┤
│  Layer 1: Your Trading Strategy            │
│  (Data feed → Signal → Risk → Order)       │
│  (Python / C++ / Rust / Go)               │
└─────────────────────────────────────────────┘

The guardian watches Layer 1. Layer 3 alerts the human when Layer 2 needs attention.

Part 1: supervisor — The Simpler Guardian

supervisor is a client/server system that manages processes as subprocesses. It is written in Python and is well-suited for small to medium deployments where you need a lightweight process manager without the complexity of a full init system.

When to Choose supervisor

  • Single-server deployment, 1–10 trading processes
  • You need a web UI for process status (supervisorctl + web interface)
  • You want human-readable configuration files
  • Your team is comfortable with Python tooling

Installation

# Debian / Ubuntu
sudo apt-get update
sudo apt-get install supervisor

# RHEL / CentOS
sudo yum install epel-release
sudo yum install supervisor

# Verify installation
supervisord --version

After installation, supervisor is typically enabled and started automatically via its own init script. Configuration lives in /etc/supervisor/.

Configuration File Structure

supervisor uses a main configuration file (supervisord.conf) and supplemental configuration files from an include directory:

/etc/supervisor/
├── supervisord.conf          # Main configuration
└── conf.d/                  # Per-strategy configuration files
    ├── data-feed.conf
    ├── signal-engine.conf
    └── risk-guard.conf

Configuring a Trading Strategy Process

Here is a complete conf.d/signal-engine.conf for a Python-based signal computation strategy:

[program:signal-engine]
; Command to execute. Use absolute paths where possible.
command=/home/quant/strategies/signal_engine/main.py

; Working directory — critical for relative path resolution
directory=/home/quant/strategies/signal_engine

; Run as a non-root user (create the 'quant' user first)
user=quant

; Autostart and autorestart
autostart=true
autorestart=true

; Startretries: number of restart attempts before giving up
startretries=5

; Exit codes that are considered "expected" — don't restart on these
; 0 = clean exit, 42 = our custom "do not restart" exit code
exitcodes=0
; 2 = SIGHUP can be used to gracefully reload config without full restart
; 42 = custom "disable restarts" exit code from our strategy

; How long to wait before considering the process started (seconds)
startsecs=10

; stdout and stderr go to these log files; supervisor auto-rotates them
stdout_logfile=/var/log/supervisor/signal-engine-stdout.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/var/log/supervisor/signal-engine-stderr.log
stderr_logfile_maxbytes=50MB
stderr_logfile_backups=10

; Environment variables — put secrets here, NOT in the code
environment=TICKDB_API_KEY="%(ENV_TICKDB_API_KEY)s"

; Stop signal: use SIGTERM for clean shutdown, not SIGKILL
stopwaitsecs=30
stopsignal=TERM

⚠️ Engineering note: The environment directive allows you to pass secrets from the host environment into the supervised process. Never hardcode API keys or database credentials in command or command= arguments — they would be visible in process lists (ps aux).

The Critical autorestart Directive

supervisor's autorestart has three modes:

Value Behavior Best for
false Never auto-restart after exit Disabled processes during maintenance
true Restart after any unexpected exit Production strategies
unexpected Restart only if exit code is not in exitcodes= Strategies with intentional shutdown codes

For production trading systems, use autorestart=true. If your strategy has a graceful shutdown mode that exits with code 0, add other codes to exitcodes= to prevent unwanted restarts.

Health Check Integration

supervisor natively tracks process uptime and exit codes, but it does not detect hung processes. To handle hung processes, integrate a health-check script:

[program:signal-engine]
; ... existing configuration ...
command=/home/quant/strategies/signal_engine/main.py

; Run a health check via tail -f on a sentinel file
; The strategy must write a heartbeat to this file every N seconds
; If the file stops updating, supervisor treats it as a failure
; Note: This requires the 'supervisor' event listener system — see below

The native tail -f approach does not work reliably for hung-process detection. Instead, use supervisor's event listener system. Create an event listener configuration:

[eventlistener:signal-engine-health]
command=/usr/local/bin/health_check_signal_engine.sh
events=PROCESS_STATE_EXITED,PROCESS_STATE_FATAL
buffer_size=100
stdout_logfile=/var/log/supervisor/health-check-stdout.log
stderr_logfile=/var/log/supervisor/health-check-stderr.log
autorestart=true

And the accompanying health-check script:

#!/bin/bash
# /usr/local/bin/health_check_signal_engine.sh
# Monitors the heartbeat file. If it stops updating for > 60 seconds,
# kill the stuck process. Supervisor's event listener system handles restart.

HEARTBEAT_FILE="/tmp/signal_engine_heartbeat"
TIMEOUT=60

echo "READY - Health check listener started"

while read line; do
    if [ ! -f "$HEARTBEAT_FILE" ]; then
        echo "WARN: Heartbeat file missing: $HEARTBEAT_FILE"
        continue
    fi

    LAST_MODIFIED=$(stat -c %Y "$HEARTBEAT_FILE" 2>/dev/null)
    NOW=$(date +%s)
    ELAPSED=$((NOW - LAST_MODIFIED))

    if [ "$ELAPSED" -gt "$TIMEOUT" ]; then
        echo "CRITICAL: Heartbeat stale (${ELAPSED}s). Signaling process."
        # Write to supervisor's event listener stdin to trigger PROCESS_STATE_FATAL
        # The event listener framework writes EVENT event_type\nlen\npayload
        # This will cause supervisor to see a fatal event and restart the process
        kill -HUP 1  # Signal init to log the condition; process kill handled by a companion script
    fi
done

⚠️ Engineering note: The event listener approach is powerful but adds complexity. A simpler production pattern is to run the health check inside your strategy process itself — write a heartbeat file, and have a separate cron job or a watchdog script check it every minute. This is described in detail in Part 3.

Managing supervisor

# Reload configuration after changing .conf files
sudo supervisorctl reread
sudo supervisorctl update

# Start, stop, restart a specific process
sudo supervisorctl start signal-engine
sudo supervisorctl stop signal-engine
sudo supervisorctl restart signal-engine

# View status of all managed processes
sudo supervisorctl status

# View recent logs
sudo supervisorctl tail signal-engine
sudo supervisorctl tail -1000 signal-engine  # last 1000 lines

Restart Storms: The One Scenario supervisor Cannot Handle Alone

supervisor will restart a process that exits immediately, and then exit again, and then restart — a restart storm. If your strategy crashes because of a configuration error (e.g., an invalid API key), supervisor will restart it indefinitely within the startretries window.

The solution: startretries=5 limits restart attempts. After five failures, supervisor puts the process into a FATAL state and stops trying. Pair this with a monitoring alert on process state changes (see the event listener above) so you are notified before the fifth crash.

For most trading systems, this is sufficient. For systems where a brief connectivity outage causes repeated restarts, combine autorestart=true with startsecs=10 — if the process exits within 10 seconds, supervisor treats it as a startup failure rather than a successful run, which counts against startretries.

Part 2: systemd — The System-Level Guardian

systemd is the init system for most modern Linux distributions. It is more powerful and more complex than supervisor. Use systemd when you need:

  • Boot-time recovery (your strategy must start when the server boots)
  • Dependencies between services (e.g., data feed must start before signal engine)
  • Resource limits (CPU, memory, file descriptors)
  • Integration with the system journal (journalctl)
  • cgroup-based process tracking and isolation

When to Choose systemd Over supervisor

  • The server runs other services (web server, database, Redis) and you want a unified management interface
  • Your trading system has multiple processes with startup dependencies
  • You need per-process resource limits (critical for preventing an OOM kill on one process from taking down the whole box)
  • You are on a modern Linux distribution where systemd is the standard (Ubuntu 18.04+, RHEL 8+, Debian 10+)

Unit File for a Trading Strategy

Create a systemd unit file at /etc/systemd/system/signal-engine.service:

[Unit]
# Human-readable description
Description=Quant Signal Engine — Trend Following Strategy

# Start this service after the network is available
# For strategies that need external API access, 'network-online.target' is safer
After=network-online.target

# Optional: dependencies on other services
# Warns if data-feed.service fails, but does not stop this service
Wants=data-feed.service
# Requires data-feed.service to be running before starting this
Requires=data-feed.service

[Service]
# Execution type
Type=simple

# The command to run
# Use ExecStartPre for validation scripts (check config, API connectivity)
ExecStartPre=/usr/local/bin/validate_config.sh /home/quant/strategies/signal_engine/config.yaml
ExecStartPre=/usr/local/bin/check_api_key.sh

ExecStart=/usr/bin/python3 /home/quant/strategies/signal_engine/main.py

# Working directory
WorkingDirectory=/home/quant/strategies/signal_engine

# Run as non-root user
User=quant
Group=quant

# Restart policy — the most important directive
Restart=on-failure
RestartSec=5

# Environment file — load secrets from a file, not from the unit file itself
EnvironmentFile=/etc/quant/env/signal-engine.env

# Resource limits (prevents one process from consuming the entire server)
# These values depend on your strategy's memory profile — profile first
MemoryMax=2G
MemoryHigh=1.5G
CPUQuota=80%
LimitNOFILE=65536

# Logging — systemd's journal (viewable with journalctl)
StandardOutput=journal
StandardError=journal
SyslogIdentifier=signal-engine

# Graceful shutdown — give the process time to close positions and flush state
TimeoutStopSec=60

# Restart conditions — do NOT restart if these are the exit codes
# 0 = clean shutdown, 42 = intentional disable
RestartPreventExitStatus=0 42

[Install]
# Enable this service to start on boot
WantedBy=multi-user.target

Environment File for Secrets

Create /etc/quant/env/signal-engine.env with strict file permissions:

# /etc/quant/env/signal-engine.env
TICKDB_API_KEY="tk_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
DB_PASSWORD="your_secure_database_password"
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXX/YYYY/ZZZZ"

# Secure the file
sudo chmod 600 /etc/quant/env/signal-engine.env
sudo chown root:quant /etc/quant/env/signal-engine.env

⚠️ Engineering note: Never store API keys in the unit file itself. The unit file is world-readable on most systems. Use EnvironmentFile and restrict file permissions to 0600.

The Restart Policy Matrix

systemd's restart behavior is controlled by three directives:

Directive Values Effect
Restart= no, always, on-success, on-failure, on-abnormal, on-abort, on-watchdog When to attempt a restart
RestartSec= Integer (seconds) Delay between restart attempts
StartLimitBurst= Integer Max restarts within StartLimitIntervalSec
StartLimitIntervalSec= Integer (seconds) Time window for StartLimitBurst

For a production trading strategy:

Restart=on-failure
RestartSec=10
StartLimitBurst=5
StartLimitIntervalSec=300

This means: if the process crashes, wait 10 seconds and restart. But if it crashes 5 times within 5 minutes, stop trying and require a manual restart. This prevents restart storms from hammering a broken API endpoint.

To reset the restart counter after a successful run, systemd automatically resets the counter when the process runs for at least the time specified in RuntimeMaxSec= (if set) or successfully starts without immediately crashing.

Managing systemd Services

# Reload systemd after creating or editing a unit file
sudo systemctl daemon-reload

# Start, stop, restart
sudo systemctl start signal-engine
sudo systemctl stop signal-engine
sudo systemctl restart signal-engine

# Enable for boot-time start
sudo systemctl enable signal-engine

# View status (shows recent log lines inline)
sudo systemctl status signal-engine

# View logs with journalctl
sudo journalctl -u signal-engine              # all logs
sudo journalctl -u signal-engine -f          # follow (tail -f equivalent)
sudo journalctl -u signal-engine --since "1 hour ago"
sudo journalctl -u signal-engine -p err      # errors only

# Check if the service is enabled at boot
sudo systemctl is-enabled signal-engine
sudo systemctl is-active signal-engine

Multi-Process Dependency Chains

A realistic quant system has dependencies. The data feed must be running before the signal engine, and the risk guard must be running before order submission:

# /etc/systemd/system/data-feed.service
[Unit]
Description=Market Data Feed — TickDB WebSocket Ingestion
After=network-online.target
[Service]
ExecStart=/usr/bin/python3 /home/quant/data_feed/main.py
Restart=on-failure
RestartSec=5
User=quant

# /etc/systemd/system/signal-engine.service
[Unit]
Description=Quant Signal Engine
# Wait for data-feed to be active before starting
After=network-online.target data-feed.service
# Start data-feed if it is not running (but don't stop if it fails)
Wants=data-feed.service
[Service]
ExecStart=/usr/bin/python3 /home/quant/signal_engine/main.py
Restart=on-failure
RestartSec=5
User=quant

# /etc/systemd/system/risk-guard.service
[Unit]
Description=Risk Guard — Position and Loss Limits
# Must have both data feed and signal engine running
After=network-online.target data-feed.service signal-engine.service
Wants=data-feed.service signal-engine.service
[Service]
ExecStart=/usr/bin/python3 /home/quant/risk_guard/main.py
Restart=on-failure
RestartSec=5
User=quant

systemd starts services in dependency order. If data-feed.service fails and is not running, signal-engine.service and risk-guard.service will still start — this is by design (the Wants= directive). If you need hard dependency (stop if the dependency stops), use Requires= instead.

Part 3: Writing a Production-Grade Health Check Script

Both supervisor and systemd can restart crashed processes, but neither natively detects a process that is alive but unresponsive. A stuck process — one in an infinite loop, deadlock, or network call that never returns — consumes CPU and memory without producing output.

The solution is a health-check watchdog script. This script runs independently of your strategy and periodically verifies that the process is actually healthy.

The Heartbeat Protocol

The strategy process writes a heartbeat signal to a file (or a POSIX message queue) at a regular interval. The watchdog script checks the file's modification time. If the heartbeat is stale, the watchdog kills and restarts the process.

Here is a Python implementation of a heartbeat writer, designed to be run as a background thread inside your strategy:

# strategy_heartbeat.py
# Run as a daemon thread inside your strategy process
import os
import threading
import time
import signal
import sys

HEARTBEAT_FILE = "/tmp/signal_engine_heartbeat"
INTERVAL = 15  # seconds — write a heartbeat every 15 seconds


def write_heartbeat():
    """Write current timestamp to heartbeat file. Called by heartbeat thread."""
    pid = os.getpid()
    with open(HEARTBEAT_FILE, "w") as f:
        f.write(f"{int(time.time())}|{pid}|{threading.current_thread().name}\n")
    os.utime(HEARTBEAT_FILE, None)  # Update modification time


def heartbeat_loop():
    """Daemon thread: write heartbeat every INTERVAL seconds."""
    while True:
        try:
            write_heartbeat()
        except Exception as e:
            # If heartbeat writing fails, log it — but do not crash the strategy
            sys.stderr.write(f"[HEARTBEAT] Failed to write heartbeat: {e}\n")
        time.sleep(INTERVAL)


def start_heartbeat():
    """Call this from your strategy's main() function."""
    t = threading.Thread(target=heartbeat_loop, daemon=True, name="HeartbeatDaemon")
    t.start()
    write_heartbeat()  # Write immediately so the watchdog sees a fresh file


def stop_heartbeat():
    """Call this during graceful shutdown to clean up the heartbeat file."""
    try:
        if os.path.exists(HEARTBEAT_FILE):
            os.remove(HEARTBEAT_FILE)
    except OSError:
        pass

And the watchdog script that runs externally:

#!/usr/bin/env python3
# /usr/local/bin/watchdog_signal_engine.py
"""
External watchdog for signal-engine strategy.
Checks heartbeat file freshness. If stale, kills and notifies.

Run this as a separate systemd service:
  systemd service → /usr/local/bin/watchdog_signal_engine.py

The watchdog itself should be monitored — run it under systemd with Restart=always.
"""
import os
import sys
import time
import signal
import subprocess
import requests
import json
from datetime import datetime

HEARTBEAT_FILE = "/tmp/signal_engine_heartbeat"
PROCESS_NAME = "signal-engine"       # Used for supervisorctl
TIMEOUT_SECONDS = 60                  # Restart if no heartbeat for 60 seconds
CHECK_INTERVAL = 10                  # Check every 10 seconds
SLACK_WEBHOOK = os.environ.get("SLACK_WEBHOOK_URL")

LOG_FILE = "/var/log/quant/watchdog.log"


def log(msg):
    ts = datetime.now().isoformat()
    line = f"[{ts}] {msg}"
    print(line)
    try:
        os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
        with open(LOG_FILE, "a") as f:
            f.write(line + "\n")
    except OSError:
        pass


def send_alert(message, severity="WARNING"):
    """Send Slack alert if webhook is configured."""
    if not SLACK_WEBHOOK:
        return
    payload = {
        "text": f"[{severity}] Signal Engine Watchdog: {message}"
    }
    try:
        requests.post(
            SLACK_WEBHOOK,
            data=json.dumps(payload),
            headers={"Content-Type": "application/json"},
            timeout=5
        )
    except requests.RequestException as e:
        log(f"Failed to send Slack alert: {e}")


def get_heartbeat_age():
    """Return age of heartbeat file in seconds, or -1 if file missing."""
    if not os.path.exists(HEARTBEAT_FILE):
        return -1
    try:
        mtime = os.path.getmtime(HEARTBEAT_FILE)
        return time.time() - mtime
    except OSError:
        return -1


def is_process_running(pid):
    """Check if a process with given PID is actually running."""
    if pid <= 0:
        return False
    try:
        os.kill(pid, 0)
        return True
    except OSError:
        return False


def restart_process():
    """
    Restart the managed process. The mechanism depends on your init system.

    For systemd:
        subprocess.run(["systemctl", "restart", "signal-engine"], check=True)

    For supervisor:
        subprocess.run(["supervisorctl", "restart", PROCESS_NAME], check=True)
    """
    log("Restarting process...")
    try:
        # Detect which init system is managing the process
        systemd_check = subprocess.run(
            ["systemctl", "is-active", "--quiet", "signal-engine"],
            capture_output=True
        )
        if systemd_check.returncode == 0:
            subprocess.run(["systemctl", "restart", "signal-engine"], check=True)
            log("Restarted via systemd.")
        else:
            subprocess.run(["supervisorctl", "restart", PROCESS_NAME], check=True)
            log("Restarted via supervisor.")
    except subprocess.CalledProcessError as e:
        log(f"Restart failed: {e}")
        send_alert(f"Restart failed: {e}. Manual intervention required.", severity="CRITICAL")


def main():
    log("Watchdog started.")
    consecutive_failures = 0

    while True:
        age = get_heartbeat_age()

        if age < 0:
            log("Heartbeat file not found — process may not have started yet.")
        elif age > TIMEOUT_SECONDS:
            consecutive_failures += 1
            log(f"CRITICAL: Heartbeat stale ({age:.1f}s old). "
                f"Consecutive failures: {consecutive_failures}")

            # Read PID from heartbeat file to identify the process
            pid = None
            if os.path.exists(HEARTBEAT_FILE):
                try:
                    with open(HEARTBEAT_FILE) as f:
                        parts = f.read().strip().split("|")
                        if len(parts) >= 2:
                            pid = int(parts[1])
                except (ValueError, IOError):
                    pass

            if pid and is_process_running(pid):
                log(f"Killing stuck process (PID {pid}).")
                try:
                    os.kill(pid, signal.SIGTERM)
                    time.sleep(5)  # Give it time to terminate gracefully
                    os.kill(pid, signal.SIGKILL)
                except OSError:
                    pass

            send_alert(
                f"Heartbeat stale ({age:.0f}s). Process killed and restart triggered."
            )
            restart_process()
            consecutive_failures = 0
        else:
            if consecutive_failures > 0:
                log(f"Heartbeat recovered ({age:.1f}s). Monitoring continues.")
                consecutive_failures = 0
            elif age > 30:
                log(f"Heartbeat OK ({age:.1f}s old).")

        time.sleep(CHECK_INTERVAL)


if __name__ == "__main__":
    # Make the watchdog itself restart on crash
    while True:
        try:
            main()
        except Exception as e:
            log(f"Watchdog crashed: {e}. Restarting in 10 seconds.")
            send_alert(f"Watchdog itself crashed: {e}. Restarting.", severity="CRITICAL")
            time.sleep(10)

systemd Service for the Watchdog

The watchdog itself should be managed by systemd with Restart=always:

# /etc/systemd/system/signal-engine-watchdog.service
[Unit]
Description=Signal Engine Watchdog
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 /usr/local/bin/watchdog_signal_engine.py
User=quant
Restart=always
RestartSec=10
EnvironmentFile=/etc/quant/env/signal-engine.env

[Install]
WantedBy=multi-user.target

⚠️ Engineering note: A common mistake is placing the watchdog on the same machine as the strategy without resource limits. If the server is under memory pressure, the watchdog itself may be OOM-killed before it can restart the strategy. Assign the watchdog process a reserved memory allocation (MemoryMin=128M in the unit file) or run it on a separate, lighter server.

Part 4: Boot Recovery — Ensuring Your Strategy Starts After a Server Restart

This is the scenario that kills live portfolios: a planned or unplanned server reboot. Without explicit boot-time recovery, your strategy does not restart automatically. The server comes back online, but your strategy stays down.

For systemd: Enable the Service

This is one command:

sudo systemctl enable signal-engine
sudo systemctl enable data-feed
sudo systemctl enable risk-guard
sudo systemctl enable signal-engine-watchdog

systemctl enable creates the symbolic links needed for the service to start on boot. Verify:

sudo systemctl is-enabled signal-engine
# Output: linked → it is enabled

For supervisor: Use the init script

supervisor ships with an init script that starts on boot. Ensure it is enabled:

# On SysVinit-based systems
sudo update-rc.d supervisor defaults

# On systems with chkconfig
sudo chkconfig --add supervisor
sudo chkconfig supervisor on

# Verify supervisor starts on boot
sudo systemctl is-enabled supervisor

Boot-Time Verification Checklist

After any server restart (planned or unplanned), run this checklist before the market opens:

# 1. Check all processes are running
sudo systemctl status signal-engine data-feed risk-guard signal-engine-watchdog
# All should show "active (running)"

# 2. Verify heartbeat files exist and are fresh
find /tmp -name "*_heartbeat" -mmin -5
# Should list heartbeat files modified in the last 5 minutes

# 3. Check recent logs for errors
sudo journalctl -u signal-engine --since "1 hour ago" -p err --no-pager

# 4. Verify network connectivity to data sources
curl -s --max-time 5 https://api.tickdb.ai/v1/market/kline/latest \
  -H "X-API-Key: $TICKDB_API_KEY" | python3 -c "import sys,json; d=json.load(sys.stdin); print('TickDB OK:', d.get('code'))"

# 5. Verify strategy process is not in a hung state
# Check CPU usage over 60 seconds — a strategy stuck in a loop will show 100% CPU
mpstat 1 60 | grep -A 1 "Average" || true

Part 5: supervisor vs. systemd — Decision Guide

Criteria supervisor systemd
Boot recovery Requires init script setup, less standardized Native, one systemctl enable command
Process dependencies Manual ordering via priority=NNN First-class Requires=, After=, Wants=
Resource limits Basic via ulimit in the command wrapper Native MemoryMax, CPUQuota, LimitNOFILE
Logging Rotated log files System journal (journalctl) with structured metadata
Web UI Built-in web interface on port 9001 No native web UI
Startup time Slightly slower (Python startup) Near-instant (C binary)
Complexity Lower — human-readable INI files Higher — INI-style with many directives
Container compatibility Works in Docker (with --privileged and volume mounts) Not recommended inside containers (use Docker's restart policy instead)
Best for Single-server dev/staging, small teams Production servers, multi-process systems, institutional deployments

The Hybrid Approach

Many production systems use both: supervisor for quick iteration and a web UI during development, with systemd as the authoritative boot-time manager. In this setup, supervisor manages the processes, but a systemd service manages supervisor itself:

# /etc/systemd/system/supervisor.service
[Unit]
Description=Supervisor Process Manager
After=network-online.target

[Service]
Type=forking
ExecStart=/usr/bin/supervisord -c /etc/supervisor/supervisord.conf
ExecStop=/usr/bin/supervisorctl shutdown
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target

This way, supervisor processes are treated as a unit by systemd. A server reboot triggers the supervisor service, which then starts all managed trading processes in the correct order.

Closing

The code you write for a trading strategy is only as reliable as the infrastructure that surrounds it. A process guardian — whether supervisor, systemd, or a combination of both — is not an optional add-on for production systems. It is the layer that ensures your strategy survives the inevitable: crashes, memory exhaustion, network outages, and server reboots.

The key principles to remember:

  1. Restart policies prevent restart storms. Set startretries in supervisor and StartLimitBurst in systemd. Allow a finite number of restarts before requiring manual intervention.
  2. Health checks catch hung processes. Both supervisor and systemd detect crashes, but neither detects a process that is alive but unresponsive. A heartbeat-based watchdog is the standard solution.
  3. Secrets belong in environment files, not in configuration files. The unit file and supervisor config are world-readable on most systems.
  4. Boot-time recovery is verified, not assumed. Run the post-reboot checklist before every market open.

Next Steps

If you are setting up your first production environment, start with supervisor for its simplicity and web UI. Migrate to systemd when you need boot recovery, resource limits, or multi-process dependency management.

If you are already using systemd, write a watchdog service following the pattern in this article. Verify your boot-time recovery with a test reboot during market hours — not during a live session.

If you want to integrate data monitoring into your process guardian, explore how TickDB's WebSocket depth channel can feed into your health-check logic — anomalous order book data can signal upstream problems before they cause a process crash.

If you need a complete backtesting history to validate that your strategy's restart behavior does not introduce systematic bias (e.g., the strategy systematically misses volatile opening windows because it restarts too slowly), reach out to enterprise@tickdb.ai for institutional data plans.

This article does not constitute investment advice. Trading strategies involve substantial risk of loss. Automated trading systems require thorough testing in paper-trading or sandbox environments before deployment with real capital.