At 10:00:02 on a volatile trading morning, the bid sits at $100.00 and the ask sits at $100.01. A market order for 1,000 shares appears to have an obvious destination: the visible order book.

It does not.

That order may execute against a posted quote on a lit exchange, rest internally at the broker-dealer, match with a wholesale market maker, or find liquidity in an alternative trading system that does not continuously display its full book. The final transaction report may show only an off-exchange identifier rather than the economic counterparty or exact venue economics.

This distinction matters because displayed quotes represent only part of available liquidity. Price discovery, order routing, and execution quality cannot be understood from the top of the visible book alone.

For retail and institutional traders alike, the practical lesson is that a quote is an interface, not the market itself. This article explains how dark pools, wholesale market makers, and other off-exchange venues fit into US equity execution; how “40% off-exchange” should—and should not—be interpreted; and how price improvement, routing transparency, and order-flow measurement affect trading decisions.

1. The “40%” Figure: What It Measures

The often-cited claim that approximately 40% of US equity volume occurs in dark pools is directionally useful but technically imprecise. Industry estimates commonly place total off-exchange activity near 40%, depending on the period, venue classification, and measurement methodology.

Off-exchange trading is broader than dark-pool trading.

Trading segment Typical order-flow behavior Visibility Main analytical concern
Lit exchange Publishes quotes and trades; automated matching High Queue position, displayed depth, adverse selection
Alternative trading system May display some or all quotes and trades Partial to high, depending on the ATS Internalization, routing priority, counterparty concentration
Wholesale market maker Holds inventory and internalizes customer flow Minimal public depth Price improvement, inventory risk, conflict management
Broker-dealer internalization Executes an order within the firm Venue-specific reporting Best execution, routing incentives, conflicts
Other OTC activity Covers a wider set of non-exchange transactions Low Reporting overlap, benchmark attribution

The aggregate off-exchange share can include ATS activity, wholesaler internalization, broker-dealer transactions, and volume reported through the FINRA Trade Reporting Facility, or TRF. The TRF is a reporting facility, not the venue where every underlying trade necessarily occurred.

This distinction prevents a common analytical error:

Forty percent off-exchange does not mean forty percent in identically structured dark pools.

The number also varies over time. It can rise during volatile periods, large institutional rebalance windows, and periods when wholesale market makers internalize more aggressively. It can fall when participants demand more transparent liquidity or when execution algorithms route more flow to displayed venues.

Why Retail Traders Often Miss This Flow

Retail order-routing disclosures show where orders were sent or how they were handled, but they do not provide a complete venue-by-venue attribution of individual fills.

The missing information usually includes:

  • The exact depth available at another venue when the broker routed the order.
  • Whether an ATS matched the order against an internal or external liquidity source.
  • The wholesale market maker’s inventory position at execution.
  • The quote state at every relevant trading venue at the exact routing decision time.
  • Whether an internalization delivered measurable price improvement.

Consequently, a retail trader can observe a good execution without knowing whether it came from superior routing or from a transiently favorable internalization decision. That uncertainty makes execution logs more useful than a simple venue label.

2. How Dark Pools and Market Makers Work

2.1 Visible and Hidden Liquidity

A lit exchange maintains a public limit order book. The National Best Bid and Offer, or NBBO, represents the best displayed bid and offer across protected trading venues.

A dark pool or ATS may operate under one of several models:

  1. Pre-trade opacity: Orders enter without continuous display, but trades may be reported afterward.
  2. Post-trade opacity: Quotes are visible, but individual executions are not continuously disclosed.
  3. Internalization with an external offset: A broker-dealer or wholesaler matches a customer order and offsets the resulting exposure elsewhere.

The amount of darkness therefore varies by venue and trading model. “Dark” does not mean unregulated or invisible in every sense. Registered ATSs operate under SEC and FINRA rules, including transparency, reporting, and market-access obligations.

2.2 The Role of the Market Maker

A market maker continuously quotes bids and offers and manages the inventory created when clients trade against those quotes.

The market maker earns the spread when it buys at the bid and sells at the ask. It also faces several risks:

  • Adverse selection: Trading against an informed order.
  • Inventory risk: Holding a position that moves against the firm.
  • Timing risk: Being filled on one side just before a price move.
  • Execution risk: Paying transaction costs or hedging at a worse price.
  • Routing risk: Sending a marketable order to a venue without obtaining a better outcome.

The wholesale market maker is therefore not simply a passive counterparty. Its quoting behavior determines how much liquidity becomes available to retail flow and how that liquidity is priced.

2.3 What “Price Improvement” Means

Price improvement occurs when a customer receives a better execution price than the applicable reference price.

For a marketable buy order, the simple calculation is:

[
\text{Price Improvement} = \text{Reference Price} - \text{Execution Price}
]

For a marketable sell order:

[
\text{Price Improvement} = \text{Execution Price} - \text{Reference Price}
]

Suppose the arrival NBBO is $100.00 × $100.01 and a buy order executes at $99.99.

Measure Value
Arrival bid $100.00
Arrival ask $100.01
Execution price $99.99
Improvement per share $0.01
Improvement in basis points 1.0 bp

The execution appears better than the displayed ask, but “one basis point of price improvement” does not automatically mean the broker violated its best-execution obligations. Conversely, executing exactly at the NBBO does not prove the order received the best available execution.

Best execution is a process obligation. It requires a broker-dealer to seek the most favorable terms reasonably available under the circumstances. Venue choice is only one component. Order size, urgency, market conditions, expected toxicity, and execution certainty can all affect the analysis.

2.4 Regulatory Boundaries

Several rules define the operating environment:

  • Regulation NMS Rule 610 requires reasonable access to quotes.
  • Rule 611 generally prohibits trading through protected quotes, subject to defined exceptions.
  • SEC Rule 300 regulates alternative trading systems.
  • FINRA Rule 613 governs ATS transparency.
  • FINRA Rule 606 requires disclosures about customer order routing and execution.
  • FINRA Rule 5310 requires best-execution review as part of a broker-dealer’s supervisory framework.

These rules do not create one universal execution algorithm. They establish boundaries around protected quotes, reporting, routing disclosures, and best-execution obligations.

3. The Order-Routing Lifecycle

The hidden half of the market becomes easier to analyze as a sequence of decisions rather than a collection of venue labels.

┌─────────────────────────────────────────────────────────────┐
│  1. ORDER ARRIVAL                                           │
│  Price, size, side, limit constraints, urgency, order type │
└────────────────────────────┬────────────────────────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────┐
│  2. REFERENCE MARKET STATE                                  │
│  NBBO, displayed depth, recent prints, venue state, volatility│
└────────────────────────────┬────────────────────────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────┐
│  3. ROUTING DECISION                                         │
│  Exchange | ATS | wholesaler | internalization | smart route │
└────────────────────────────┬────────────────────────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────┐
│  4. EXECUTION                                               │
│  Fill price, size, venue class, latency, implied cost       │
└────────────────────────────┬────────────────────────────────┘
                             ▼
┌─────────────────────────────────────────────────────────────┐
│  5. POST-TRADE QUALIFICATION                                │
│  Price improvement, effective spread, adverse selection    │
└─────────────────────────────────────────────────────────────┘

Pre-Trade

The broker or execution algorithm captures the NBBO and other reference information before deciding where to route the order. The quality of this snapshot matters because the comparison price may change between arrival and execution.

For large orders, the algorithm may divide the order into child orders and route them according to price, venue availability, fill probability, or a proprietary liquidity model.

During the Order

A routing decision can occur within microseconds. The algorithm may check displayed venues, internal liquidity sources, ATSs, and wholesalers according to its configured priority.

This stage creates a subtle data problem. The best quote at the routing decision may already be stale when the order reaches the selected venue. A fill can therefore look inferior relative to the original arrival quote even if the execution was reasonable relative to the market at execution time.

Post-Trade

Post-trade analysis should distinguish:

  • Quoted spread: Best ask minus best bid.
  • Effective spread: Twice the difference between the reference midpoint and the execution price for a round-trip trade.
  • Price improvement: Difference between the execution price and the relevant reference price.
  • Implementation shortfall: Difference between the decision price and the final execution price, including fees and market impact.
  • Markout: Price movement after execution, used to distinguish favorable fills from adverse selection.

A fill below the reference midpoint may look attractive immediately but perform poorly later if the market moved because the counterparty was better informed. Markout therefore provides a necessary second dimension.

4. Production-Grade Execution Data Collector

The following reference collector connects to an execution feed that exposes venue classification, arrival reference price, side, and fill price. It is intentionally vendor-neutral because the analysis requires execution-level fields that aggregated OHLCV data cannot provide.

The feed contract assumes JSON messages similar to:

{
  "type": "execution",
  "venue_class": "ats",
  "venue": "EXAMPLE",
  "side": "buy",
  "price": 99.99,
  "arrival_mid": 100.00,
  "timestamp": "2026-04-20T14:00:00.123Z"
}

The venue_class field must come from the feed’s venue taxonomy. A venue name alone does not prove that a fill represents a dark pool, ATS, wholesaler, or internalization.

import asyncio
import json
import os
import random
import time
from urllib.parse import quote

import websockets
from websockets.exceptions import InvalidStatus

BASE_RETRY_SECONDS = 1.0
MAX_RETRY_SECONDS = 60.0
HEARTBEAT_SECONDS = 15.0
PONG_TIMEOUT_SECONDS = 45.0

# Required environment variables:
# EXEC_FEED_WS_URL=wss://your-feed.example/ws
# EXEC_FEED_API_KEY=your_api_key
#
# The API key is transmitted as a query parameter because this example
# follows the TickDB WebSocket authentication convention.
#
# ⚠️ Confirm that the upstream provider uses the same parameter name.
# Never print the authenticated URL or expose the key in logs.

WEBSOCKET_URL = os.environ.get("EXEC_FEED_WS_URL")
API_KEY = os.environ.get("EXEC_FEED_API_KEY")
SUBSCRIBE_MESSAGE = os.environ.get(
    "EXEC_FEED_SUBSCRIBE",
    json.dumps({"type": "subscribe", "channels": ["executions"]}),
)

if not WEBSOCKET_URL or not API_KEY:
    raise RuntimeError(
        "EXEC_FEED_WS_URL and EXEC_FEED_API_KEY are required"
    )


def authenticated_url(base_url: str, api_key: str) -> str:
    separator = "&" if "?" in base_url else "?"
    return f"{base_url}{separator}api_key={quote(api_key, safe='')}"


def retry_after_seconds(message: dict) -> int:
    """Read Retry-After from a feed payload or nested headers."""
    value = message.get("retry_after")
    headers = message.get("headers") or {}

    if value is None:
        value = headers.get("Retry-After") or headers.get("retry-after")

    try:
        return max(1, int(value))
    except (TypeError, ValueError):
        return 5


def classify_venue(message: dict) -> str:
    """Preserve the provider's venue classification without guessing."""
    return str(
        message.get("venue_class")
        or message.get("execution_venue_type")
        or "unknown"
    ).lower()


def calculate_improvement(message: dict) -> tuple[float | None, float | None]:
    """Return dollar and basis-point improvement when fields are valid."""
    try:
        reference = float(message["arrival_mid"])
        execution = float(message["price"])
        side = str(message["side"]).lower()
    except (KeyError, TypeError, ValueError):
        return None, None

    if reference <= 0 or side not in {"buy", "sell"}:
        return None, None

    signed_improvement = (
        reference - execution if side == "buy" else execution - reference
    )

    return signed_improvement, signed_improvement / reference * 10_000


def write_observation(message: dict) -> None:
    """Append normalized execution observations to JSON Lines."""
    improvement, improvement_bps = calculate_improvement(message)
    observation = {
        "timestamp": message.get("timestamp"),
        "venue": message.get("venue"),
        "venue_class": classify_venue(message),
        "side": message.get("side"),
        "price": message.get("price"),
        "arrival_mid": message.get("arrival_mid"),
        "price_improvement": improvement,
        "price_improvement_bps": improvement_bps,
    }

    with open("executions.jsonl", "a", encoding="utf-8") as output:
        output.write(json.dumps(observation, separators=(",", ":")) + "\n")


async def consume_feed() -> None:
    url = authenticated_url(WEBSOCKET_URL, API_KEY)
    last_pong = time.monotonic()

    async with websockets.connect(
        url,
        ping_interval=20,
        ping_timeout=10,
        close_timeout=5,
        max_size=2**20,
    ) as websocket:
        await websocket.send(SUBSCRIBE_MESSAGE)

        async def receive_messages() -> None:
            nonlocal last_pong

            async for raw_message in websocket:
                if isinstance(raw_message, bytes):
                    raw_message = raw_message.decode("utf-8")

                message = json.loads(raw_message)

                if message.get("type") == "pong" or message.get("cmd") == "pong":
                    last_pong = time.monotonic()
                    continue

                if message.get("code") == 3001:
                    delay = retry_after_seconds(message)
                    print(f"Rate limit reached; waiting {delay} seconds")
                    await asyncio.sleep(delay)
                    continue

                if message.get("type") == "execution":
                    write_observation(message)

        receiver = asyncio.create_task(receive_messages())

        try:
            while True:
                await asyncio.sleep(HEARTBEAT_SECONDS)
                await websocket.send(json.dumps({"cmd": "ping"}))

                if time.monotonic() - last_pong > PONG_TIMEOUT_SECONDS:
                    raise TimeoutError("WebSocket heartbeat timed out")

            # The receiver task exits only when the socket closes or raises.
        finally:
            receiver.cancel()


async def main() -> None:
    attempt = 0

    while True:
        try:
            attempt = 0
            await consume_feed()
        except InvalidStatus as error:
            status = getattr(error.response, "status_code", None)
            headers = getattr(error.response, "headers", {}) or {}
            retry_after = headers.get("Retry-After") or headers.get("retry-after")

            if status == 429:
                delay = int(retry_after or 5)
                print(f"Handshake rate limit; waiting {delay} seconds")
                await asyncio.sleep(delay)
                continue

            if status in {401, 403}:
                raise RuntimeError(
                    "Authentication failed; verify EXEC_FEED_API_KEY"
                ) from error

            raise
        except asyncio.CancelledError:
            raise
        except Exception as error:
            attempt += 1
            base_delay = min(
                BASE_RETRY_SECONDS * (2 ** (attempt - 1)),
                MAX_RETRY_SECONDS,
            )
            jitter = random.uniform(0, base_delay * 0.1)
            delay = base_delay + jitter

            print(f"Feed connection failed: {error}")
            print(f"Reconnecting in {delay:.2f} seconds")
            await asyncio.sleep(delay)


if __name__ == "__main__":
    # ⚠️ For high-frequency production workloads, use aiohttp/asyncio
    # with bounded queues, persistent state, and a process supervisor.
    asyncio.run(main())

Engineering Notes

The collector stores observations in append-only JSON Lines format, which simplifies replay and downstream aggregation. A production system should add:

  • A bounded queue between ingestion and persistence.
  • Clock synchronization for venue and strategy timestamps.
  • Disk rotation and schema versioning.
  • Idempotent processing keyed by execution identifier.
  • Monitoring for disconnect duration, message latency, and missing heartbeats.
  • Separate handling for authentication failures and transient network failures.
  • Venue taxonomy validation to prevent an “ATS” label from being applied to unrelated activity.

The collector also does not equate low price improvement with poor execution. It records only the available fields. Effective spread, volatility, order toxicity, and post-fill markout must be added before making a routing-quality assessment.

5. The Core Measurement Algorithm

The central analytical metric is the signed price improvement:

[
\text{Signed Improvement}_t =
\begin{cases}
M_t - P_t, & \text{if the side is buy} \
P_t - M_t, & \text{if the side is sell}
\end{cases}
]

where (M_t) is the reference midpoint or benchmark at the routing decision and (P_t) is the execution price.

The metric should be normalized because a one-cent improvement means something different for a $10 stock than for a $1,000 stock:

[
\text{Improvement}_{bps,t} =
\frac{\text{Signed Improvement}_t}{M_t} \times 10{,}000
]

Recommended Segmentation

Do not calculate execution quality as one aggregate average. Segment the data by:

Dimension Examples Why it matters
Venue class Lit exchange, ATS, wholesaler, internalization Different quoting and inventory models
Order type Market, marketable limit, non-marketable limit Different sources of price improvement
Time of day Open, close, macro-event window Wider spreads and faster information arrival
Symbol Mega-cap, mid-cap, small-cap Different liquidity and tick-size regimes
Volatility Normal, elevated, extreme Changes fill probability and markout
Size bucket Odd lot, institutional, block Larger orders interact differently with hidden liquidity
Reference price Arrival NBBO, arrival midpoint, decision-time quote Prevents benchmark ambiguity
Urgency Patient, normal, aggressive Changes the opportunity cost of liquidity

A useful production summary is:

[
\bar{I}{v} = \frac{1}{N_v}
\sum{t \in v} I_t
]

where (I_t) is signed improvement and (v) identifies a venue, symbol, or other segment.

Measuring Adverse Selection

Price improvement alone is incomplete. For each fill, calculate a post-trade markout:

[
\text{Markout}{h} =
\begin{cases}
P{t+h} - P_t, & \text{buy} \
P_t - P_{t+h}, & \text{sell}
\end{cases}
]

A positive markout indicates that the market moved favorably after execution. A negative markout suggests adverse selection, although it can also reflect an unrelated market shock.

The combination of improvement and markout produces four useful categories:

Price improvement Later markout Possible interpretation
Positive Positive Genuine favorable execution
Positive Negative Immediate benefit with adverse selection or reversal
Negative Positive Poor immediate price followed by favorable movement
Negative Negative Likely routing, latency, or market-impact problem

These categories are diagnostic rather than definitive. Order purpose and market context remain necessary.

6. Venue Comparison: What Each Model Optimizes

Capability Lit exchange ATS or dark pool Wholesale market maker Broker internalization
Public quote continuity High Varies Low Venue-specific
Depth transparency Often high May be limited Internal inventory hidden Internal inventory hidden
Price discovery Direct Contributes through trades Contributes through quotes and fills Contributes when inventory changes
Typical execution Displayed matching Internal or external matching Principal or inventory-based fill Customer order handled internally
Retail price-improvement opportunity Limited Possible but venue-dependent Central Often material
Main execution risk Queue position Venue opacity Adverse selection and inventory risk Conflicts and routing quality
Best analytical requirement Full depth and queue data Venue and fill classification Quote and markout timestamps Detailed broker execution records

No model is universally superior. A wholesale market maker may provide immediate price improvement but conceal adverse selection. A lit exchange may expose more information but leave the order exposed to queue competition and information leakage. A dark pool may reduce displayed information leakage while increasing uncertainty about who provides the liquidity.

The key question is not “Which venue is dark?” but “Which liquidity source provides the best execution under the stated market and order constraints?”

7. What This Means for Different Traders

Individual Investors

An individual investor should not interpret a dark-pool execution as a sign that the broker found a secret price. It may reflect a routing algorithm, internalization policy, ATS match, or wholesaler quote evaluated against the prevailing market.

The most useful retail-level checks are:

  1. Compare the execution price with the NBBO available at the time of order entry.
  2. Check whether the order was marketable or limit-based.
  3. Review whether improvement varied by time of day or order size.
  4. Avoid interpreting a single fill as proof of a persistent routing advantage.
  5. Compare displayed-price improvement with later markout where execution timestamps permit.

A broker that repeatedly executes at the NBBO may offer adequate best execution without creating visible price improvement. Conversely, one favorable internalization fill does not establish that the broker is systematically better than alternatives.

Individual Quant Developers

Individual developers should preserve the decision-time quote, execution timestamp, venue class, and reference benchmark with every simulated fill.

A backtest that applies the closing price or a daily VWAP to an intraday strategy can conceal routing differences that would dominate live implementation. The minimum useful fields are:

  • Order arrival and decision timestamps.
  • Arrival NBBO or midpoint.
  • Fill price and size.
  • Venue classification.
  • Venue and strategy timestamps.
  • Fees, rebates, and exchange fees.
  • Post-fill price path for markout.

Institutional Teams

Institutional teams should evaluate routing at the policy and order-type level rather than through one venue-level average. The review should separate displayed execution, internalization, wholesaler flow, and ATS flow.

A useful execution-quality dashboard includes:

Metric Displayed ATS Wholesaler Internalization
Filled notional Required Required Required Required
Signed improvement in bps Required Required Required Required
Effective spread Required Required Required Required
Post-fill markout Required Required Required Required
Adverse-selection rate Required Required Required Required
Fill rate Required Required Required Required
Quote staleness at routing Recommended Recommended Recommended Recommended
Partial-fill rate Recommended Recommended Recommended Recommended
Order rejection rate Recommended Recommended Recommended Recommended

The review should also examine execution performance conditional on volatility and market impact. A venue may appear expensive during a stressed period because it is absorbing risk that another venue passes to the client.

Strategy-Specific Implications

Dark-pool activity changes several common assumptions.

Quote fades become less reliable. A displayed quote may disappear because an informed trader interacted with visible liquidity, while hidden liquidity continues trading elsewhere. A strategy that treats disappearing quotes as the only signal misses that structure.

Headline volume overstates displayed depth. Session volume includes executions that never appeared in the public book. The visible book is therefore a partial representation of overall liquidity.

Low-volume periods can contain meaningful hidden volume. This is particularly relevant around corporate events, index rebalances, and closing auctions.

Execution quality cannot be inferred from fills alone. Without the decision-time quote and post-fill path, a favorable print can merely reflect a favorable market move.

Trade classification requires care. A trade reported through an OTC facility does not necessarily identify the underlying execution venue. Any strategy that relies on venue labels must document the provider’s taxonomy and reporting model.

Data Capability Boundaries

Historical US equity OHLCV supports session-level analysis, volatility research, and strategy backtesting, but aggregate bars cannot reveal whether a fill occurred in a dark pool. Even L1 depth data cannot reconstruct hidden depth, identify the complete venue universe, or separate wholesale internalization from other off-exchange execution.

TickDB provides more than 10 years of cleaned and aligned US equity OHLCV through its kline endpoint, which is suitable for historical strategy and market-session analysis. Its US equity depth capability is limited to L1; the depth channel is not a source of full US order-book history or dark-venue attribution. The platform therefore should not be used to infer off-exchange execution from aggregated candles.

For backtesting, REST authentication uses the X-API-Key header. For WebSocket connections, the API key is passed as a URL parameter. The historical kline endpoint should be used for completed periods, while the latest-candle endpoint serves live dashboard use cases.

8. Execution Checklist

Before trusting a dark-pool or execution-quality conclusion, verify the following:

Data Integrity

  • Venue classifications come from documented provider metadata.
  • Arrival, decision, and execution timestamps are synchronized.
  • The reference quote is captured at the correct decision point.
  • Partial fills remain linked to the parent order.
  • Corporate actions and symbol changes do not distort historical prices.
  • OTC reporting overlap is not mistaken for unique underlying volume.

Strategy Evaluation

  • Price improvement is calculated with the correct order side.
  • Results are reported in both dollars and basis points.
  • Immediate improvement is paired with post-fill markout.
  • Venue statistics are segmented by volatility and time of day.
  • Fees, rebates, slippage, and latency are included.
  • The benchmark is fixed before evaluation.

Operational Risk

  • WebSocket heartbeats detect silent failures.
  • Reconnection uses exponential backoff and jitter.
  • Rate limits respect Retry-After.
  • Authentication remains in environment variables.
  • Ingestion and persistence operate independently.
  • Logs exclude API keys and sensitive order details.

9. Conclusion

The visible bid and ask remain the trader’s first reference point, but they describe only one layer of liquidity. The other layer includes ATSs, wholesale market makers, broker-dealer internalization, and off-exchange reporting structures that influence both execution price and information leakage.

The hidden half of the market does not invalidate displayed quotes. It changes their interpretation. A good execution requires more than a favorable print: the trader must know which reference price applied, where the order was routed, whether the fill represented genuine improvement, and how the market moved after execution.

For retail investors, this means execution quality deserves the same attention as order-entry speed. For quantitative teams, it means venue and markout data belong in the research pipeline rather than in a separate compliance report.

Next Steps

If you are an investor seeking deeper market-structure analysis, subscribe to the TickDB newsletter for weekly research on liquidity, execution, and market behavior.

If you want to build and test market-data workflows, sign up at tickdb.ai for a free API key, set TICKDB_API_KEY as an environment variable, and use the authenticated kline endpoint for completed historical US equity periods.

If you require institutional-scale historical OHLCV coverage, review TickDB’s Professional and Enterprise options for research and strategy-backtesting workflows.

If you use AI coding assistants, search for and install the tickdb-market-data SKILL in your tool’s marketplace.

This article does not constitute investment advice. Markets involve risk; past performance does not guarantee future results.