From Solo to Squad: Building Production-Grade Data Infrastructure for Small Quantitative Teams

---

The Moment Everything Changes

You built a strategy. It works. You were alone.

Then you brought in a partner — a former colleague who writes cleaner code than you and has ideas of their own. Suddenly, your main.py that lives on your laptop becomes a coordination problem. Whose version is live? Who pushed the broken backtest? Where is the clean dataset that Chen spent three hours cleaning last week?

Three-person quantitative teams face a specific kind of infrastructure crisis: they are complex enough that coordination breaks, but not large enough to have a dedicated DevOps engineer. The solution is not more people. The solution is a data infrastructure that scales with the work, not the headcount.

This article walks through the production-grade infrastructure stack a three-person quant team needs — covering shared data access, version-controlled code repositories, API key management, and permission controls. Every recommendation is concrete, implements in under two hours, and survives contact with a live trading system.

---

The Three Collaboration Failure Modes

Before designing the system, it helps to name the failure modes that kill small quant teams.

Failure Mode 1: Data Silos

Each team member maintains their own copy of market data. The US equity OHLCV dataset lives on your workstation. Chen has a copy on a NAS drive. Marcus pulls fresh data every time he runs a backtest. Results diverge not because the strategy changed, but because the data did.

Failure Mode 2: Code Drift

You refactored signal_engine.py last Tuesday. Chen is still running the version from two weeks ago — the one with the bug you thought you fixed. Backtest results do not match. Nobody can reproduce the live system's behavior.

Failure Mode 3: Credential Leakage

Marcus needs to call the market data API from his research script. You share the API key over Slack. Three months later, the key is in a GitHub repository that was accidentally made public. The credential is burned. You rotate it. Chen's scheduled job breaks. Nobody knows why until market open.

These are not hypothetical. They are the three most common infrastructure failures in sub-five-person quant teams. The fix for each is architectural, not disciplinary.

---

Infrastructure Architecture: The Four Pillars

A resilient shared infrastructure for a small quant team rests on four pillars. Each pillar has a specific purpose and a recommended implementation.

Pillar	Purpose	Recommended Tool	Cost
Shared data storage	Single source of truth for market data	S3-compatible storage + TickDB for live feeds	$20–$80/month
Version control	Reproducible code and backtest history	GitHub private repository	Free (free tier)
Secrets management	Secure storage and rotation of API keys	HashiCorp Vault or AWS Secrets Manager	$0–$50/month
Access control	Per-user permissions and audit logging	GitHub Teams + Cloud IAM	Included in above

The architecture ties together as follows: TickDB provides the live and historical market data feeds. Raw and processed datasets are stored in S3-compatible storage. All strategy code lives in a private GitHub repository with branch protection. API keys and database credentials are stored in a secrets manager, never in code or Slack messages. Access is governed by GitHub Teams and cloud IAM roles.

---

Pillar 1: Shared Data Storage with TickDB

The Problem with Local Data Copies

The most dangerous assumption a quant team makes is that a dataset is static. It is not. Exchange APIs evolve. Data vendors update their normalization logic. A dataset that was "good enough" last quarter may have subtle changes that invalidate a six-month backtest run today.

TickDB solves this by providing a unified, timestamped feed of historical and live market data through a consistent REST and WebSocket API. When your team pulls from a single endpoint, every backtest references the same canonical dataset.

Setting Up Shared Access

For a three-person team, the recommended approach is a shared TickDB account with individual API keys per user. This preserves auditability — you can see which team member's code consumed how much quota — while enabling centralized billing.

Each team member generates their own API key from the TickDB dashboard. The keys are not shared. They are stored in the secrets manager (covered in Pillar 3).

Fetching Historical Data for Shared Storage

The following Python script demonstrates a production-grade data fetcher that pulls historical OHLCV data from TickDB, stores it in a local S3-compatible path, and includes retry logic with exponential backoff.

import os
import json
import time
import hashlib
import logging
from datetime import datetime, timedelta
from pathlib import Path

import requests

# ⚠️ API key loaded from environment — never hardcoded
TICKDB_API_KEY = os.environ.get("TICKDB_API_KEY")
if not TICKDB_API_KEY:
    raise ValueError("TICKDB_API_KEY environment variable is not set")

BASE_URL = "https://api.tickdb.ai/v1"
HEADERS = {"X-API-Key": TICKDB_API_KEY}
DATA_ROOT = Path(os.environ.get("TICKDB_DATA_ROOT", "./data"))
LOG_FILE = Path(os.environ.get("LOG_FILE", "./fetch.log"))

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
    handlers=[logging.FileHandler(LOG_FILE), logging.StreamHandler()],
)
logger = logging.getLogger(__name__)


def fetch_kline(symbol: str, interval: str, start_time: int, end_time: int, max_retries: int = 3):
    """
    Fetch historical kline (OHLCV) data from TickDB.
    
    Args:
        symbol: Exchange symbol, e.g. "AAPL.US"
        interval: Candle interval, e.g. "1h", "1d"
        start_time: Unix timestamp (ms) for range start
        end_time: Unix timestamp (ms) for range end
        max_retries: Maximum retry attempts on failure
    
    Returns:
        List of kline records
    """
    url = f"{BASE_URL}/market/kline"
    params = {
        "symbol": symbol,
        "interval": interval,
        "start_time": start_time,
        "end_time": end_time,
        "limit": 1000,  # Maximum allowed per request
    }
    
    all_records = []
    retry_count = 0
    base_delay = 2.0
    
    while retry_count <= max_retries:
        try:
            response = requests.get(
                url,
                headers=HEADERS,
                params=params,
                timeout=(3.05, 15),  # (connect timeout, read timeout)
            )
            
            if response.status_code == 429:
                retry_after = int(response.headers.get("Retry-After", base_delay))
                logger.warning(f"Rate limited. Retrying after {retry_after}s")
                time.sleep(retry_after)
                continue
                
            response.raise_for_status()
            data = response.json()
            
            if data.get("code") != 0:
                error_code = data.get("code")
                error_msg = data.get("message", "Unknown error")
                
                # Handle rate limit within response body
                if error_code == 3001:
                    retry_after = int(data.get("retry_after", base_delay))
                    logger.warning(f"Rate limited (code 3001). Retrying after {retry_after}s")
                    time.sleep(retry_after)
                    continue
                    
                raise RuntimeError(f"API error {error_code}: {error_msg}")
            
            records = data.get("data", {}).get("klines", [])
            all_records.extend(records)
            logger.info(f"Fetched {len(records)} records for {symbol} @ {interval}")
            return all_records
            
        except requests.exceptions.Timeout:
            retry_count += 1
            delay = base_delay * (2 ** retry_count) + time.uniform(0, 0.5)
            logger.warning(f"Request timed out. Retry {retry_count}/{max_retries} in {delay:.1f}s")
            time.sleep(delay)
            
        except requests.exceptions.RequestException as e:
            retry_count += 1
            delay = base_delay * (2 ** retry_count) + time.uniform(0, 0.5)
            logger.error(f"Request failed: {e}. Retry {retry_count}/{max_retries} in {delay:.1f}s")
            time.sleep(delay)
    
    raise RuntimeError(f"Failed to fetch data after {max_retries} retries")


def store_data_locally(symbol: str, interval: str, records: list, data_root: Path):
    """
    Store fetched kline records in a structured local directory.
    File naming: {symbol}_{interval}_{YYYYMMDD}.json
    """
    if not records:
        logger.warning(f"No records to store for {symbol} @ {interval}")
        return
    
    first_ts = records[0].get("t", 0)
    date_str = datetime.fromtimestamp(first_ts / 1000).strftime("%Y%m%d")
    filename = f"{symbol.replace('/', '_')}_{interval}_{date_str}.json"
    
    filepath = data_root / symbol.replace("/", "_") / interval / filename
    filepath.parent.mkdir(parents=True, exist_ok=True)
    
    with open(filepath, "w") as f:
        json.dump({
            "symbol": symbol,
            "interval": interval,
            "fetched_at": datetime.utcnow().isoformat(),
            "record_count": len(records),
            "records": records,
        }, f, indent=2)
    
    logger.info(f"Stored {len(records)} records to {filepath}")


def compute_data_hash(records: list) -> str:
    """Compute SHA-256 hash of records for integrity verification."""
    content = json.dumps(records, sort_keys=True)
    return hashlib.sha256(content.encode()).hexdigest()[:16]


def fetch_and_store_symbol(symbol: str, interval: str, days_back: int = 365):
    """
    Main workflow: fetch historical data and store to local path.
    """
    end_time_ms = int(datetime.utcnow().timestamp() * 1000)
    start_time_ms = int((datetime.utcnow() - timedelta(days=days_back)).timestamp() * 1000)
    
    logger.info(f"Fetching {symbol} @ {interval} from {days_back} days ago")
    
    records = fetch_kline(symbol, interval, start_time_ms, end_time_ms)
    
    if records:
        store_data_locally(symbol, interval, records, DATA_ROOT)
        record_hash = compute_data_hash(records)
        logger.info(f"Data integrity hash (first 16 chars): {record_hash}")
    
    return records


if __name__ == "__main__":
    # Example: fetch AAPL daily data for the past year
    symbols = ["AAPL.US", "NVDA.US", "SPY.US"]
    interval = "1d"
    
    for symbol in symbols:
        try:
            fetch_and_store_symbol(symbol, interval, days_back=365)
        except Exception as e:
            logger.error(f"Failed to fetch {symbol}: {e}")

Why This Code Is Production-Grade

The fetcher above is not a toy script. It handles the failure modes that appear in real deployment:

• Timeout handling: Every request has a (3.05, 15) tuple timeout — 3.05 seconds to establish connection, 15 seconds to read the response. The 3.05 second connect timeout is intentional — it exceeds the 3-second mark that load balancers use to drop slow connections.
• Exponential backoff with jitter: On failure, the script doubles the wait time between retries and adds random jitter (0–0.5 seconds) to prevent thundering herd behavior when the API recovers.
• Rate limit handling: The script checks both HTTP 429 status codes and TickDB's internal code: 3001 response field. It reads the Retry-After header and pauses accordingly.
• Integrity hashing: A SHA-256 hash of each dataset is logged. When two team members compare backtest results, they can verify they are running against identical data.

---

Pillar 2: Git Workflow for Quantitative Code

The Branching Strategy

For a three-person quant team, a simple branching strategy beats a complex one. Use a trunk-based development model with short-lived feature branches.

main (protected)
  └── feat/signal-refactor
  └── feat/new-factor
  └── hotfix/data-pipeline-fix

• main is protected. No direct pushes. All changes go through pull requests with at least one required reviewer.
• Feature branches live for 1–3 days. Longer-lived branches diverge and create merge nightmares.
• Commits are atomic. Each commit should represent one logical change — a new factor, a bug fix, a data pipeline update.

Commit Message Convention

Adopt a conventional commit format that makes git log readable and enables automated changelog generation.

feat: add mean-reversion factor based on z-score crossover
fix: correct timestamp alignment in futures data pipeline
data: update AAPL dividend adjustments for Q1 2025
backtest: add transaction cost model with 0.5 bps per trade
infra: configure GitHub Actions for nightly backtest run

Backtest Reproducibility via Git Tags

Tag every backtest run with a commit hash. This is non-negotiable for team reproducibility.

# Run the backtest
python backtest.py --config configs/earnings_reversion.yaml

# Tag the results with the current commit
git add results/backtest_2025_02_15.json
git commit -m "backtest: earnings reversion run for Feb 2025 earnings season"
git tag -a backtest/earnings_2025q1 -m "Full backtest of earnings reversion strategy, Q1 2025"
git push origin main --tags

When Marcus asks why his results differ from Chen's, the answer is one git log command away.

GitHub Actions for Automated Backtesting

Set up a nightly pipeline that runs the canonical backtest suite and publishes results as a GitHub Actions artifact.

# .github/workflows/nightly_backtest.yml
name: Nightly Backtest

on:
  schedule:
    - cron: "0 6 * * *"  # Run at 6 AM UTC daily
  workflow_dispatch:  # Allow manual trigger

jobs:
  backtest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
      
      - name: Run backtest suite
        run: python -m pytest tests/backtest_suite.py --junitxml=results/junit.xml
        env:
          TICKDB_API_KEY: ${{ secrets.TICKDB_API_KEY }}
      
      - name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: backtest-results-${{ github.run_number }}
          path: results/

---

Pillar 3: API Key and Secrets Management

The Vault Architecture

Never store API keys in environment files, code, or chat messages. For a three-person team, HashiCorp Vault in dev mode is the right balance of security and operational simplicity.

Vault runs locally on each developer's machine (or on a shared lab server) and provides:

• Centralized secrets storage: All API keys, database passwords, and cloud credentials live in one place.
• Dynamic credentials: Vault can generate temporary AWS or database credentials that expire automatically.
• Audit logging: Every secret access is logged with the requesting user and timestamp.

Setting Up Vault (Dev Mode)

# Start Vault in dev mode (not for production use — dev only)
vault server -dev -dev-root-token-id="dev-token"

# In another terminal, set up a path for team secrets
export VAULT_ADDR="http://127.0.0.1:8200"
export VAULT_TOKEN="dev-token"

# Store a TickDB API key
vault kv put secret/tickdb/main \
  api_key="tk_live_xxxxxxxxxxxxxxxxxxxx" \
  owner="chen@example.com" \
  created_at="$(date -u +%Y-%m-%d)"

# Store a second key for Marcus
vault kv put secret/tickdb/marcus \
  api_key="tk_live_yyyyyyyyyyyyyyyyyyyyy" \
  owner="marcus@example.com" \
  created_at="$(date -u +%Y-%m-%d)"

# Verify
vault kv get secret/tickdb/main

Application Integration

Your Python code reads keys from Vault at startup, never from a hardcoded string.

import hvac
import os

def get_tickdb_credentials():
    """
    Retrieve TickDB API key from HashiCorp Vault.
    Falls back to environment variable for CI/CD pipelines.
    """
    vault_addr = os.environ.get("VAULT_ADDR")
    vault_token = os.environ.get("VAULT_TOKEN")
    
    if vault_addr and vault_token:
        client = hvac.Client(url=vault_addr, token=vault_token)
        secret = client.secrets.kv.v2.read_secret_version(
            path="secret/tickdb/main",
            mount_point="secret",
        )
        return secret["data"]["data"]["api_key"]
    
    # CI/CD fallback: use CI environment variable
    api_key = os.environ.get("TICKDB_API_KEY")
    if not api_key:
        raise ValueError("No TickDB credentials found. Set VAULT_ADDR/VAULT_TOKEN or TICKDB_API_KEY.")
    return api_key

API Key Rotation Protocol

When a key is compromised, rotation must be fast and auditable.

1. Generate a new key from the TickDB dashboard.
2. Update the new key in Vault: vault kv patch secret/tickdb/main api_key="tk_live_NEWKEY..."
3. Trigger a restart of any running processes that read from Vault. Dynamic secret providers like Vault automatically cache credentials for a configured TTL (time-to-live). Set the TTL to 5 minutes maximum for API keys.
4. Revoke the old key from the TickDB dashboard immediately after confirming the new key is operational.

---

Pillar 4: Permission Control

GitHub Teams Structure

For a three-person team, three GitHub Teams map cleanly to roles:

Team	Members	Permissions
@quant-team/admin	Team lead	Full repository access, can merge PRs, manage settings
@quant-team/researchers	Chen, Marcus	Read/write on main repository, cannot modify branch protection
@quant-team/observers	External advisor	Read-only access, can comment on PRs but not push

Data Access Permissions

Not every team member needs access to every dataset. If your team works across multiple strategies, assign data access based on the strategy a user is actively contributing to.

TickDB's API key system supports this naturally: generate per-user keys and configure separate rate limit pools. An API key assigned to "earnings strategy" has no additional privilege over a key assigned to "futures arbitrage" — but the audit trail tells you exactly which team's code consumed which quota.

Secrets Manager Permissions

In Vault, use policies to restrict access:

# researcher-policy.hcl
path "secret/data/tickdb/*" {
  capabilities = ["read"]
}

path "secret/data/strategies/*" {
  capabilities = ["read"]
}

path "secret/data/infrastructure/*" {
  capabilities = []
}

This policy gives a researcher read access to data keys but no access to infrastructure credentials (cloud keys, database passwords). Apply the policy to Chen's and Marcus's Vault tokens.

---

Deployment Guide by Team Stage

The infrastructure described above adapts as the team grows. Here is a deployment roadmap by team size and complexity.

Stage	Team size	Key additions	Remove or upgrade
Stage 1: Foundation	1–2	GitHub private repo, Vault dev mode, shared S3 bucket	None
Stage 2: Collaboration	3–4	GitHub Teams with branch protection, separate API keys per user, scheduled GitHub Actions backtests	Local Vault → Vault server
Stage 3: Production	5–10	Vault server with HA, IAM roles for cloud resources, separate strategy repositories, on-call rotation	Shared repo → per-strategy repos

For your three-person team, Stage 2 is the target. Do not prematurely adopt Stage 3 complexity.

---

Putting It Together: The Shared Workflow

The four pillars work together as a single workflow that every team member follows.

1. Pull latest code
   git checkout main && git pull

2. Fetch latest data to shared storage
   python scripts/fetch_data.py --symbol AAPL.US --interval 1d

3. Run backtest
   python backtest.py --config configs/earnings_reversion.yaml

4. Commit results with tag
   git add results/ && git commit -m "backtest: reversion strategy, $(date +%Y%m%d)"
   git tag backtest/reversion_$(date +%Y%m%d)

5. Push and create PR
   git push origin main --tags
   # Open PR in GitHub — requires at least one reviewer's approval

If a team member's backtest produces different results, the diagnostic steps are:

1. Confirm both are on the same commit: git log --oneline
2. Confirm both are using the same dataset hash: check the logged hash from the fetcher
3. Confirm both are using the same environment variables: diff <(env | sort) <(other_user_env | sort)

---

What Not to Build

A three-person team should resist the urge to over-engineer. The following are premature optimizations for a team of your size:

• Custom data pipelines: Use TickDB's REST and WebSocket APIs directly. Building a custom ingestion layer adds maintenance burden with zero benefit at this scale.
• Multi-cloud architecture: Pick one cloud provider. Running AWS for some strategies and GCP for others creates data alignment nightmares.
• Microservices: Keep the codebase in one monorepo. Microservice communication overhead is not worth it for three people managing a handful of strategies.
• Real-time trading infrastructure: Until you have a strategy with a validated backtest and positive live track record over at least 60 days, paper trading with shared dashboards is sufficient.

---

Next Steps

If you're evaluating market data infrastructure for your team, visit tickdb.ai to explore the API documentation, generate a free API key with no credit card required, and review the coverage matrix for your target asset classes.

If you're building a shared data pipeline, the code examples in this article are directly usable as starting points. Clone the GitHub repository, configure your Vault secrets, and replace the symbol list with your target universe.

If you need institutional-grade historical data for cross-cycle backtesting, reach out to enterprise@tickdb.ai for plans that include 10+ years of cleaned US equity OHLCV data, dedicated rate limit pools, and SLA-backed uptime guarantees.

If you're an AI-assisted developer, search for the tickdb-market-data SKILL in your AI tool's marketplace. It provides pre-built function calls for kline, depth, and trades endpoints — reducing the time to integrate TickDB into your research workflow.

---

This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results. Infrastructure decisions should align with your team's regulatory obligations and operational capacity.