The primary matching engine for CME Group’s futures and options markets is housed in CyrusOne Aurora I, a purpose-built data center for the CME Globex electronic trading platform. The facility is located at: 2905 Diehl Road, Aurora, IL 60502.

Some vendors market “Chicago-based access” to CME without specifying the exact facility. Because facilities outside Aurora are significantly cheaper, this is sometimes used to obscure material performance tradeoffs when the vendor is not situated in the Aurora primary colocation facility.

When evaluating your vendor and colocation options, it’s important to confirm the specific data center and network path.

CyrusOne Aurora I (CHI1)

Pros

• Primary colocation site.
• Fastest possible access to CME matching engine, highest determinism and timestamp accuracy.
• Direct cross-connects to most institutional vendors offering CME data and execution services, such as Databento.

Cons

• High cost (rack pricing often in the five-figure per month range).
• CyrusOne typically requires larger footprints (e.g., multiple cabinets) for a a direct colocation agreement, so resellers and MSPs are common for firms needing only a few servers or a single rack.
• Fewer, more expensive, WAN network carrier options.

CyrusOne Aurora II and III (CHI2 and CHI3)

Pros

• Lower cost than Aurora I.
• Next highest determinism and timestamp accuracy after Aurora I.
• Next lowest latency after Aurora I (on same campus).

Cons

• Similar downsides as Aurora I.

350 E Cermak (Equinix CH1/4/5)

Pros

• Major Internet and cloud interconnection hub.
• Easier carrier access.
• Relatively diverse selection of vendors.
• Proximity to public cloud (e.g., AWS, Google) on-ramps.

Cons

• Higher latency (at least 287 microseconds one-way on fiber) and less timestamp determinism than AUR1.

Other regional sites

Some examples include CoreSite CH1, Databank ORD1, Databank ORD2, Digital Realty ORD11, and Equinix CH7 Westmont.

Pros

• Lower cost.

Cons

• Not appropriate for latency-critical trading.
• Increased jitter and latency.
• Poor options for connecting to other services.

Changes in market microstructure, like updates to trading rules, matching engine behavior, or the wire protocol, can also affect your strategy’s performance.

These changes are especially relevant to intraday strategies and are often overlooked. It’s also easy to lose track of when these changes happened you’re backtesting over many years.

To help, we’ve put together a cheatsheet of major CME Globex microstructure changes to guide your use of Databento’s historical futures tick data — which spans back to 2010.

About Databento

Databento is a licensed provider of market data and APIs for over 50 trading venues, including major futures venues like CME, CBOT, NYMEX, COMEX, ICE, EEX, and Eurex. We help firms of all sizes access market data more easily. Our coverage includes high-precision market data at tick-by-tick, L2, and L3 granularity going back to 2010.

This tutorial shows you how to build a real-time stock screener (or scanner) that constantly analyzes a streaming real-time market data feed for changes on all U.S. equities—over 9,000 NMS stocks and ETFs.

At the end of this tutorial, you’ll be able to get output timestamped to the nanosecond like this:

[2025-04-23T04:00:01.688717194-04:00] NVDA moved by 4.40% (current: 98.1900, previous: 102.7100)
[2025-04-23T04:00:02.121450448-04:00] BABA moved by 10.34% (current: 131.2750, previous: 118.9700)
[2025-04-23T06:36:23.507827237-04:00] AMZN moved by 7.36% (current: 193.8850, previous: 180.6000)

You can also skip ahead to get the entire code on GitHub here.

What’s wrong with existing stock screener tools?

A major challenge with implementing a stock scanner is that it needs to listen to all tickers on the market. Very few market data feeds are able to stream real-time prices continuously for all ~9,000 U.S. equity tickers, especially as a WAN-shaped feed over internet.

Many retail companies provide this as a commercial service, but they usually focus on retail trading use cases and don’t give you much programmatic customization over the alerts to trigger on. They also don’t provide very strong guarantees on delivery or timestamping accuracy.

Another problem is that most retail tools are bad at handling pre-market prices before the 9:30 AM open.

Stock screener example

Our stock screener should:

• Handle the entire U.S. equities universe of ~9,000 symbols efficiently.
• Have median sub-ms feed delay to NY4/5 (WAN-shaped).
• Take less than 5 seconds to fetch historical data, subscribe, and start scanning.
• Monitor all US stocks and ETFs for price movements exceeding a configurable threshold.
• Compares current prices against previous day’s closing prices.
• Displays alerts when significant moves are detected.

State management

First, we’ll write a class to hold all of the state for this stock screener.

We need a few constants:

• PX_SCALE is used to express prices in decimal units. By default, Databento uses a fixed integer representation of decimal prices to avoid loss of precision, and each unit represents 1 “nanodollar” or 1e-9 of a dollar.
• PX_NULL is used to handle null prices. This happens when one side of the book is empty—which happens more frequently than you might expect during pre-market hours or on illiquid stocks that you’ll encounter when consuming the entire universe of US stocks and ETFs.
• PCT_MOVE_THRESHOLD will be our arbitrary threshold for triggering an alert. In this case, we’ll write a simple alert on whenever a stock moves 3% or more in either direction.

Also, we’ll create 3 dictionaries so we can look up each symbol:

• self.symbol_directory will track the numeric instrument ID of the symbol, which raw exchange feeds like Nasdaq Basic change daily.
• self.last_day_lookup will look up the last closing price of the ticker.
• self.is_signal_lit helps us indicate if an alert has already fired for the ticker, so we only print once when the threshold is first exceeded.

from datetime import datetime, timedelta
from typing import Dict, Any

import databento as db
import pandas as pd
import pytz


class PriceMovementScanner:
    """Scanner for detecting large price movements in all US equities."""

    # Constants
    PX_SCALE: float = 1e-9
    PX_NULL: int = 2**63 - 1
    PCT_MOVE_THRESHOLD: float = 0.03

    def __init__(self, pct_threshold: float = None, today: str = TODAY) -> None:
        """Initialize scanner with configurable threshold and date."""
        self.pct_threshold = pct_threshold or self.PCT_MOVE_THRESHOLD
        self.today = today
        self.today_midnight_ns = int(pd.Timestamp(today).timestamp() * 1e9)
        self.symbol_directory: Dict[int, str] = {}
        self.last_day_lookup: Dict[str, float] = self.get_last_day_lookup()
        self.is_signal_lit: Dict[str, bool] = {symbol: False for symbol in self.last_day_lookup}

Getting previous day’s stock and ETF prices

We’ll add a class method for populating self.last_day_lookup using Databento’s historical client. Let’s understand what this is doing:

• dataset="EQUS.SUMMARY" — This uses the Databento US Equities Summary feed, which gives consolidated end-of-day prices (OHLCVs) across all NMS exchange and ATSs. This is designed to maximize CTA/UTP SIP coverage without license fees and provides official prices and volumes that are suited for daily EOD use.
• schema="ohlcv-1d" — This fetches daily OHLCV aggregates.
• symbols="ALL_SYMBOLS" — This fetches all ticker symbols on the dataset.

    def get_last_day_lookup(self) -> Dict[str, float]:
        """Get yesterday's closing prices for all symbols."""
        client = db.Historical()

        now = pd.Timestamp(self.today).date()
        yesterday = (pd.Timestamp(self.today) - timedelta(days=1)).date()

        # Get yesterday's closing prices
        data = client.timeseries.get_range(
            dataset="EQUS.SUMMARY",
            schema="ohlcv-1d",
            symbols="ALL_SYMBOLS",
            start=yesterday,
            end=now,
        )

        # Request symbology: This is required for ALL_SYMBOLS requests
        # which don't automatically map instrument ID to raw ticker symbol
        symbology_json = data.request_symbology(client)
        data.insert_symbology_json(symbology_json, clear_existing=True)

        df = data.to_df()
        # TODO: Adjust for overnight splits here, e.g., using Databento corporate actions API

        return dict(zip(df["symbol"], df["close"]))

As seen in the inline comment, a more complete implementation will use corporate actions data to adjust the closing prices for overnight splits. This method is flexible for you to integrate other major events or fundamentals like dividends, market cap, EPS and P/E ratio updates.

You may also want to start screening only after the book has stabilized after the pre-market, either by using trade prices instead of mid-price, requiring the spread to reach a certain number of bps, or after a certain fixed time.

Adding scanning and alerting

First, this populates the symbol directory for symbol mapping messages, which are received initially when you subscribe to a real-time feed on Databento.

Then, we compute the midprice on each MBP-1 message (top of book update), and compare that against the closing price of the previous day.

    def scan(self, event: Any) -> None:
        """
        Scan for large price movements in market data events.
        """
        if isinstance(event, db.SymbolMappingMsg):
            self.symbol_directory[event.hd.instrument_id] = event.stype_out_symbol
            return

        if not isinstance(event, db.MBP1Msg):
            return
            
        # Skip if event is from replay before today using `.subscribe(..., start=...)` parameter
        #if event.hd.ts_event < self.today_midnight_ns:
        #    return

        symbol = self.symbol_directory[event.instrument_id]
        bid = event.levels[0].bid_px
        ask = event.levels[0].ask_px

        if bid == self.PX_NULL or ask == self.PX_NULL:
            # Handle when one side of book is empty
            return

        mid = (event.levels[0].bid_px + event.levels[0].ask_px) * self.PX_SCALE * 0.5
        last = self.last_day_lookup[symbol]
        abs_r = abs(mid - last) / last

        if abs_r > self.pct_threshold and not self.is_signal_lit[symbol]:
            ts = pd.Timestamp(event.hd.ts_event, unit='ns').tz_localize('UTC').tz_convert('US/Eastern')
            print(
                f"[{ts.isoformat()}] {symbol} moved by {abs_r * 100:.2f}% "
                f"(current: {mid:.4f}, previous: {last:.4f})"
            )
            self.is_signal_lit[symbol] = True

Running your stock screener

Now, we just need to run the code. Observe our use of these parameters:

• dataset="EQUS.MINI" — This uses the Databento Equities Mini feed, a cost-effective consolidated feed. You can also modify this code to scan all options or futures by swapping this dataset ID.
• schema="mbp-1" — MBP-1 provides every order book event that updates the best bid or offer (BBO), and includes every trade and changes to book depth, alongside total size and order count at BBO.
• symbols="ALL_SYMBOLS" — This instructs the client to subscribe to every symbol on the feed. If you only have specific tickers on your watchlist, you can also specify them here, like ["AMZN", "AAPL", "GOOG", "MSFT", "NVDA"] for Amazon, Apple, Google, Microsoft, and NVIDIA. By default, Databento uses Nasdaq symbology for U.S. stocks.
• start=0 — Intraday replay is helpful for debugging this example outside of market hours.

def main() -> None:
    scanner = PriceMovementScanner()
    live = db.Live()

    live.subscribe(
        dataset="EQUS.MINI",
        schema="mbp-1",
        symbols="ALL_SYMBOLS",
        start=0,
    )

    live.add_callback(scanner.scan)
    live.start()
    live.block_for_close()


if __name__ == "__main__":
    main()

Results

[2025-04-24T04:00:00.007704938-04:00] TSLA moved by 5.43% (current: 245.4300, previous: 259.5100)
[2025-04-24T04:00:00.008339140-04:00] TQQQ moved by 11.78% (current: 46.0000, previous: 52.1400)
[2025-04-24T04:00:00.009881258-04:00] DBC moved by 5.84% (current: 22.5650, previous: 21.3200)
[2025-04-24T04:00:00.011338853-04:00] DBA moved by 4.81% (current: 25.9400, previous: 27.2500)
[2025-04-24T04:00:00.019717308-04:00] WEAT moved by 3.35% (current: 4.4650, previous: 4.6200)
[2025-04-24T04:00:00.048734686-04:00] RSST moved by 4.37% (current: 19.5950, previous: 20.4900)
[2025-04-24T04:00:00.052540506-04:00] BCI moved by 17.21% (current: 24.3800, previous: 20.8000)
[2025-04-24T04:00:00.111035469-04:00] SOYB moved by 4.26% (current: 22.8100, previous: 21.8777)
[2025-04-24T04:00:00.158550097-04:00] FTGC moved by 3.49% (current: 25.6250, previous: 24.7600)
[2025-04-24T04:00:00.158736420-04:00] CMDY moved by 3.03% (current: 51.7200, previous: 50.2000)

You can see that our screener was able to pick up the pre-market move on Tesla, Inc. (TSLA) exactly 7.7 milliseconds after 4 AM EDT, much before the first piece published on Bloomberg News on that day at 5:30 AM EDT.

Full code

You can get the full script for this example on GitHub here.

If you’re interested instead in scanning stocks on historical data only, see our docs for an example on finding the top pre-market movers.

Or learn more about Databento U.S. equities coverage, which spans over 20,000+ stocks and ETFs since inception across all U.S. exchanges and ATSs. We’re an official distributor for all major proprietary feeds like Nasdaq TotalView, NYSE Integrated, NYSE Arca Integrated, and more.

Since US equities are fungible across exchanges, does it really matter where you trade? Our latest cheatsheet explores some of the key differences between the NMS exchanges, including:

• Primary listing venues. These are important for IPOs.
• Venues with auction imbalance data. These are critical for strategies like index rebalancing.
• Maker-taker vs. taker-maker pricing models. These offer different ways to achieve effective execution prices that bypass the sub-penny rule.
• Retail price improvement programs. These are designed to attract retail order flow.
• Extended pre-market and post-market trading sessions.
• Primary colocation site. These determine costs and infrastructure requirements for connecting to the venue.

Correction: Cboe EDGA switched to a maker-taker fee model in November 2024. This will be fixed in future updates of this cheatsheet.

About Databento

Interested to learn more? Check out Databento’s microstructure guide and blog.

Databento provides next-gen market data APIs built for speed and scale, helping firms of all sizes power their trading strategies and analytics. As an official distributor for 70+ exchanges, we deliver high-precision futures, options, and equities data directly from exchange colocation sites — offering both flexible pay-as-you-go and subscription pricing.

In the latest API tutorial added to Databento’s docs, we show how you can track the exact queue position of an order with MBO data. This is compared against the estimated queue position from L2 (MBP) data only.

Estimating queue position from L2 data

Without MBO data, a common approach to estimate queue position is to define a parametric model for cancellations and update your estimated position each time the aggregate market depth at the price is reduced.

Suppose p(𝑥) denotes the probability of cancel in front of you when your order is positioned 0 ≤ 𝑥 ≤ 1 through the queue, then there are two trivial cases:

• If you just joined the queue, any cancellation or down-modification at the level must come from in front, i.e. 𝑝(𝑥=1) = 1.
• If you are at the front of the queue, 𝑝(𝑥=0) = 0, all cancels and modifies must be from behind.

This becomes a matter of fitting a monotonic function between those two points. Say, a naive model 𝑝(𝑥) = 𝑥 assumes cancels arrive uniformly through the queue. But with some market intuition, you’d expect cancels to be biased towards the back of the queue where orders have poor fill rate and hence lower value.

Comparison against exact queue position from MBO data

How strong is this bias? Probably much stronger than you’d think.

In this tutorial, we parameterize this bias -1 < k ≤ 0 where k = 0 (no bias) corresponds to the naive model where cancels are uniformly distributed in the queue, and this bias is stronger as it becomes more negative.

Then, we show a few instances of estimated queue position vs. the exact queue position (ground truth) obtained from MBO data:

It’s evident that cancels are mostly pulled from the back of the queue.

Downsides of estimating queue position from L2 data

The most common mistake and downside of this approach is underestimating the bias and being overly optimistic your good fill rates: There are many open source and commercial trading platforms that support backtesting with passive orders using L2 data; this feature is sometimes touted as “market replay”. All of these platforms we’ve come across have been overly optimistic. (They also underestimate your adverse fill rates, which is a separate topic.)

Another downside of this approach is that it only works on price levels that are “in scope” given the number of visible levels in your L2 data. For example, if you have MBP-10 data (ten levels of market depth), and if the BBO moves away from your order until it’s outside of the top ten levels, then further changes to the price level become censored, making it much harder to estimate your queue position.

Despite these shortcomings, modeling queue position from L2 data is still an important capability since some trading venues don’t provide MBO data.

Pairs trading is a form of statistical arbitrage that takes advantage of mean reversion or convergence in the prices of two instruments.

The simplest variation of this strategy involves taking long and short positions simultaneously on a pair of cointegrated instruments. But more generally, you can construct a spread from any linear combination of the two instruments with the approach in this tutorial — even going long simultaneously on two instruments.

Setup and dependencies

We’ll make use of the following external libraries:

• databento for market data.
• pandas for data manipulation.
• matplotlib for plotting.
• statsmodels for the Engle-Granger two-step cointegration test.
• scikit-learn to fit the linear regression between the two price series.

import databento as db
import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.stattools import coint
from sklearn.linear_model import LinearRegression

CME WTI (CL) vs. ICE Brent crude oil (BRN)

Most existing literature shows the application of this strategy to equities, however the strategy can be applied to any asset class.

We’ll demonstrate the pairs trading strategy on the two most liquid crude oil products traded on CME and ICE, the WTI crude oil futures (CL) and Brent crude oil futures (BRN) contracts respectively.

This also showcases the value of Databento’s supplemental receive timestamps, ts_recv, which are used to construct our OHLCV aggregates. Most academic literature avoids cross-venue arbitrage like this because timestamps from two different exchanges are based on the different reference clocks that cannot be synced easily. ts_recv mitigates this problem by providing a common reference time with sub-microsecond accuracy between geographically-distant venues.

Note: Databento timestamps CME crude and ICE Brent data on the same Aurora I gateway. If you’re using a different data source, you may have to apply a delay offset for the time between Aurora I and 350 E Cermak where their matching engines are situated respectively.

Configuration

Exchange and clearing fees can be found on CME and ICE’s websites. Tick sizes and notional tick values can be obtained from instrument definitions.

# Instrument-specific configuration
SYMBOL_X = "CLM4"
SYMBOL_Y = "BRN FMN0024!"
TICK_SIZE_X = 0.01
TICK_SIZE_Y = 0.01
TICK_VALUE_X = 10.0
TICK_VALUE_Y = 10.0
FEES_PER_SIDE_X = 0.70      # NYMEX corporate member fees
FEES_PER_SIDE_Y = 0.88

# Global configuration
COMMISSIONS_PER_SIDE = 0.08

# Backtest configuration
START = "2024-04-01"
END = "2024-05-15"

Data extraction

We’ll fetch 1-minute OHLCV aggregates for ease of demonstration, but you may specify schema="mbp-1" to use MBP-1 instead:

def fetch_data():

    client = db.Historical()

    data = client.timeseries.get_range(
        dataset="GLBX.MDP3",
        schema="ohlcv-1m",
        stype_in="raw_symbol",
        symbols=[SYMBOL_X],
        start=START,
        end=END,
    )

    df_x = data.to_df()["close"].to_frame(name="x")

    data = client.timeseries.get_range(
        dataset="IFEU.IMPACT",
        schema="ohlcv-1m",
        stype_in="raw_symbol",
        symbols=[SYMBOL_Y],
        start=START,
        end=END,
    )

    df_y = data.to_df()["close"].to_frame(name="y")

    return df_x.join(df_y, how="outer").ffill()

Since OHLCV aggregates only print when there’s a trade in the interval, you’ll need to use the .ffill() method to synchronize the two time series in cases where one leg doesn't print.

Tip: Although CL and BRN are liquid futures contracts, there’s the possibility that both sides don’t print in a given minute interval. A more complete implementation should resample during every minute of the trading session.

Engle-Granger cointegration test

The basis of this strategy is to check that the residuals of a linear regression between the prices of the two instruments exhibits stationarity, also called the Engle-Granger two-step cointegration test. This implies that the two instruments’ price series are cointegrated, and that the spread constructed based on this linear association of the two instruments’ prices is stationary. In turn, this implies that the spread is mean-reverting.

Cointegration vs. correlation

Cointegration is preferred over correlation when constructing a pairs trade, since correlation doesn’t apply to non-stationary time series, and financial instrument prices are typically non-stationary and trending. High correlation between such series can occur even without a meaningful relationship.

For instance, two price series which are trending in the same direction but diverging in the long run may exhibit a spurious, positive correlation.

On the other hand, cointegrated prices series may exhibit positive, negative, or even zero correlation over any period due to short-term fluctuations, but tend to track closely over the long run.

Implementation

The strategy is simple: First, over the lookback period (last 100 records), we run the cointegration test. If the two series are found to be cointegrated, then the coefficient of linear regression between the two series provides the ratio of the two instruments needed to construct the spread, also called the hedge ratio. We’ll also compute the sample mean and standard deviation over this period.

Then, over the forward window (also 100 records), we compute the spread (residual) and normalize it by taking its z-score.

df = fetch_data()

# Model and monetization parameters
LOOKBACK = 100
ENTRY_THRESHOLD = 1.5
EXIT_THRESHOLD = 0.5
P_THRESHOLD = 0.05

# Initialization
df["cointegrated"] = 0
df["residual"] = 0.0
df["zscore"] = 0.0
df["position_x"] = 0
df["position_y"] = 0
is_cointegrated = False
lr = LinearRegression()

for i in range(LOOKBACK, len(df), LOOKBACK):

    x = df["x"].iloc[i-LOOKBACK:i].values[:,None]
    y = df["y"].iloc[i-LOOKBACK:i].values[:,None]

    if is_cointegrated:
        # Compute and normalize signal on forward window
        x_new = df["x"].iloc[i:i+LOOKBACK].values[:,None]
        y_new = df["y"].iloc[i:i+LOOKBACK].values[:,None]
        spread_back = y - lr.coef_ * x
        spread_forward = y_new - lr.coef_ * x_new
        zscore = (spread_forward - spread_back.mean()) / spread_back.std()
        df.iloc[i:i+LOOKBACK, df.columns.get_loc("cointegrated")] = 1
        df.iloc[i:i+LOOKBACK, df.columns.get_loc("residual")] = spread_forward
        df.iloc[i:i+LOOKBACK, df.columns.get_loc("zscore")] = zscore

    _, p, _ = coint(x,y)
    is_cointegrated = p < P_THRESHOLD
    lr.fit(x,y)

If the normalized residual is negative, it means the instrument associated with y is underpriced, i.e. we’ll buy y and sell x to monetize the prediction that their prices will converge in the future. Conversely, if the normalized residual is positive, we’ll sell y and buy x.

Since the two products have a similar notional value and futures have large minimum order size, we’ll trade a static hedge ratio of 1. If you’re modifying this strategy to trade stocks or ETFs or products that have significant differences in volatility or notional value, you should consider using the dynamic hedge ratio lr.coef_ to scale your position.

Tip: The hedge ratio lr.coef_ may be negative. This regularly occurs, for instance, if one leg of the pair is an inverse ETF.

# Standardized residual is negative => y is underpriced => sell x, buy y
# Standardized residual is positive => y is overpriced => buy x, sell y
df.loc[df.zscore < -ENTRY_THRESHOLD, 'position_y'] = 1
df.loc[df.zscore > ENTRY_THRESHOLD, 'position_y'] = -1
df['position_x'] = -df['position_y']

It’s common to specify a score-based or time-based exit condition. For example, we can exit the position when the spread converges to within +/-0.5 standard deviations. This has the effect of longer holding periods and lower transaction costs.

# Exit positions when z-score crosses +/-0.5
hold_y_long = df["zscore"].apply(lambda z: 1 if z <= -EXIT_THRESHOLD else 0)
hold_y_short = df["zscore"].apply(lambda z: 1 if z >= EXIT_THRESHOLD else 0)

# Carry forward positions until exit condition is met
df["position_y"] = df["position_y"].mask((df["position_y"].shift() == -1) & (hold_y_short == 1), -1)
df["position_y"] = df["position_y"].mask((df["position_y"].shift() == 1) & (hold_y_long == 1), 1)
df["position_x"] = -df["position_y"]

Here’s a visualization of the strategy. The outer dashed lines show the entry conditions while the inner dashed lines show the exit conditions:

Estimating slippage

Exact instantaneous slippage from crossing the spread can be calculated using MBP-1 data and execution timestamps from the backtest. In the interest of time, you can gauge the viability of the strategy by applying a constant spread cost per contract side. This is typically assumed to be half a tick or one tick for liquid futures, but we can also estimate it quickly from a small slice of MBP-1 data:

def estimate_slippage():

    import datetime

    client = db.Historical()

    data = client.timeseries.get_range(
        dataset="GLBX.MDP3",
        schema="mbp-1",
        stype_in="raw_symbol",
        symbols=[SYMBOL_X],
        start=START,
    )

    df = data.to_df()
    avg_spread_x = (df["ask_px_00"] - df["bid_px_00"]).mean()

    data = client.timeseries.get_range(
        dataset="IFEU.IMPACT",
        schema="mbp-1",
        stype_in="raw_symbol",
        symbols=[SYMBOL_Y],
        start=START,
    )

    df = data.to_df()
    avg_spread_y = (df["ask_px_00"] - df["bid_px_00"]).mean()

    return avg_spread_x, avg_spread_y

avg_spread_x, avg_spread_y = estimate_slippage()

Backtest

We can now compute the gross PnL and transaction costs:

# Compute backtest results
df["pnl_x"] = 0.0
df["pnl_y"] = 0.0
df.iloc[:-1, df.columns.get_loc("pnl_x")] = df['position_x'].iloc[:-1].values * df["x"].diff()[1:].values / TICK_SIZE_X * TICK_VALUE_X
df.iloc[:-1, df.columns.get_loc("pnl_y")] = df['position_y'].iloc[:-1].values * df["y"].diff()[1:].values / TICK_SIZE_Y * TICK_VALUE_Y

df["volume_x"] = df["position_x"].diff().abs()
df["volume_y"] = df["position_y"].diff().abs()
df["tcost"] = df["volume_x"] * FEES_PER_SIDE_X + df["volume_y"] * FEES_PER_SIDE_Y + \
              (df["volume_x"] + df["volume_y"]) * COMMISSIONS_PER_SIDE + \
              (df["volume_x"] * avg_spread_x / TICK_SIZE_X / 2 * TICK_VALUE_X) + \
              (df["volume_y"] * avg_spread_y / TICK_SIZE_Y / 2 * TICK_VALUE_Y).cumsum()
df["gross_pnl"] = (df["pnl_x"] + df["pnl_y"]).cumsum()
df["net_pnl"] = df["gross_pnl"] - df["tcost"]

Results

# Plot and print results
daily_pnl = df["net_pnl"].groupby(df.index.date).sum()
sr = daily_pnl.mean() / daily_pnl.std() * 252**.5
print(f"Sharpe ratio: {sr:.2f}")

plt.plot(df["gross_pnl"], label="Gross PnL")
plt.plot(df["net_pnl"], label="Net PnL")
plt.plot(df["tcost"], label="t-cost")
plt.xlabel("Date")
plt.ylabel("PnL (USD)")
plt.legend()
plt.show()

Sharpe ratio: 8.57

Full code

The full code example can be found on our GitHub here.

Practical considerations and improvements

This tutorial is kept intentionally short using a simple vectorized implementation. Ideally, you should use MBP-1 data with an event-driven backtest loop that incorporates the delays between your system and the exchange gateways. This can be done using the market replay method.

This strategy is very sensitive to slippage and the exact expiration month that’s traded; non-lead month contracts will have wider spreads which can make the strategy infeasible.

We assumed exchange fees and commissions that are realistic for a medium-sized trading firm or a team within a larger firm that’s already a CME member firm. In practice, achieving similar rates requires a NYMEX 106.J corporate membership and thousands of contract sides executed per day.

Scaling up to larger trade sizes will significantly expand the strategy’s state machine, since you’ll have to handle partial executions and different sizes on each leg of the spread to achieve optimal hedge ratio. MBP-1 data can also inform trade sizing.

We introduced a few free model and monetization parameters, namely LOOKBACK, ENTRY_THRESHOLD, EXIT_THRESHOLD. These can be optimized using any hyperparameter tuning method (e.g. grid search) and cross-validated.

There are also a few common variations of this strategy to consider:

• Basket trading by constructing combinations of more than two instruments, using another procedure such as the Johansen test.
• Calibrating an Ornstein-Uhlenbeck process, which gives a parametric model for time-based exit.
• Using total least squares instead of ordinary least squares to determine the hedge ratio.
• Scanning a wide universe systematically for cointegrated pairs, to discover unintuitive relationships. However, this runs the risk of poor generalization if too many pairs are tested. Dimensionality reduction methods like PCA can improve the search and generalization error.

Further reading

Alexander, C. (2008). Market Risk Analysis: Practical Financial Econometrics (Vol. 2). Wiley.

Tsay, R. S. (2010). Analysis of Financial Time Series (3rd edition). Wiley.

Engle, R. F., & Granger, C. W. J. (1987). Co-integration and error correction: Representation, estimation, and testing. Econometrica, 55(2), 251–276.

Overview

This guide demonstrates how to display Databento’s market data with TradingView’s popular Lightweight Charts library, entirely in Python. It also shows how to display indicators in the charts.

TradingView is a popular charting platform and Databento is a provider of high-quality financial market data. This guide makes use of the lightweight-charts-python library, which provides a user-friendly, Pythonic interface to a simplified version of TradingView — so you don’t need to learn or use JavaScript to take advantage of TradingView’s interactive features.

If you want to explore advanced features of Lightweight Charts, you can find the documentation here.

We will make use of data from Databento’s ohlcv-1m schema. First, install databento and the lightweight-charts-python library:

pip install databento
pip install lightweight-charts

Import both libraries and pandas:

import pandas as pd
import databento as db 
from lightweight_charts import Chart

Declare a main function and set your Databento API key:

if __name__ = "__main__": 
    key = "YOUR_DB_API_KEY_HERE"
    key = "YOUR_DB_API_KEY_HERE"
    client = Historical(key=key)

Getting Data from Databento

Fetch the OHLCV data using Databento’s simple API. Databento has a simple usage-based pricing plan for new users that charges per data request. Depending on your needs, it might be a good idea to choose a subscription plan for flat-rate access or store the data locally to reduce costs that may be incurred on repeat requests.

data = client.timeseries.get_range(
        dataset="XNAS.ITCH",
        symbols=["NVDA"],
        schema="ohlcv-1m",
        start="2024-05-13T08:00",
        end="2024-05-13T23:59",
        limit=1000,
    ) 

# Optional: Save the data to file 
# path = "instrument_data.dbn"
# data.to_file(path=path)

Displaying Data with Lightweight Charts

Convert the data to a dataframe and and initialize lightweight_charts. If you saved the data to file in the previous step, you can load the data with the DBNStore helper class:

# Optional: Load data from file
# from Databento import DBNStore
# data = DBNStore.from_file(path="instrument_data.dbn")

df = data.to_df() 

# Setting toolbox to True allows you to make drawings on the chart
chart = Chart(toolbox=True)
chart.set(df)

# Set block to True, otherwise the chart window will automatically close
chart.show(block = True)

If everything goes alright, you get a basic chart as depicted below:

Plotting Lines

There are various plot types that can be used to overlay derived data and analytics on the chart. Using an example from another Databento article, we will plot the VWAP of NVDA during the session. Note that additional code from this point onwards will be included before calling the chart.set() and chart.show() methods.

# Calculate vwap and put the result in a new dataframe
df["pvt"] = df["close"] * df["volume"]
df["pvt"] = df["close"] * df["volume"]
df["vwap"] = df["pvt"].cumsum() / df["volume"].cumsum()
ohlcv_vwap_df = pd.DataFrame({
"vwap": df["vwap"]
}, index=df.index)

# Create a line bound to the chart, and set data to the line
line = chart.create_line(name="vwap", color="blue", width=3)
line.set(ohlcv_vwap_df)
    
# Important: Put the code above before these two lines
chart.set(df)
chart.show(block=True)

Applying Custom Styles

The lightweight-charts-python library supports customization of chart elements like font, colors, watermark, and gridlines. Below is an example using Databento branding:

chart.layout(background_color='#F2F2FF', text_color='#090008', font_size=13,
                 font_family='Helvetica')
chart.candle_style(up_color='#A09EFA', down_color='#474BD7',
                       border_up_color='#6266E5', border_down_color='#222262',
                       wick_up_color='#6266E5', wick_down_color='#222262')
chart.volume_config(up_color='#777777', down_color='#cccccc')
chart.watermark('NVDA 1 min', color='#dddddd')
chart.crosshair(mode='normal', vert_color='#222222', vert_style='dotted',
                    horz_color='#222222', horz_style='dotted')
chart.legend(visible=True, font_size=14, color='#222222')
chart.grid(vert_enabled=False, horz_enabled=False)

The lightweight-charts library also supports Jupyter and Streamlit. Simply follow the code below depending on your use case and replace accordingly:

# Jupyter
from lightweight_charts import JupyterChart
from lightweight_charts import JupyterChart

chart = JupyterChart()
# ... set data to the chart
chart.load()

# Streamlit
from lightweight_charts.widgets import StreamlitChart

chart = StreamlitChart()
# Set data to the chart
chart.load()

By combining Databento’s market data with the lightweight-charts-python library, you can create highly customizable and interactive TradingView-style visualizations with minimal effort. This guide covered foundational concepts, and future articles will explore advanced topics such as live data streaming and visualization of order book features.

Credits

The following article was written by Idris Dosunmu, Databento user and Lead Product Manager at Mintara Digital adapted for Medium. Mintara Digital is a digital product consultancy for fintechs, hedge funds, and family offices. The original post can be found here.

On behalf of Databento, we’d like to thank Idris for generously sharing his knowledge via this tutorial. You can find Idris on LinkedIn here.

For suggestions or requests for specific visualization techniques, please reach out to Idris via idris@mintaragroup.com.

Today we’re launching a major upgrade to Databento’s packet capture (PCAP) service , with the introduction of intraday delivery.

What’s new

Previously, customers with a daily PCAP subscription needed to wait up to 24 hours to receive the previous day’s PCAP files. With our upgraded infrastructure, customers can receive PCAP files within 20 minutes of the original market data message in most cases.

Delivery is done through a standard AWS S3 bucket and is usually set up within 1 or 2 business days of signing an order form.

What are PCAPs and how are they used for market data?

Packet captures contain raw market data, in their original wire format and sequence.

The most valuable benefit of getting market data in PCAP format is its losslessness. PCAPs allow a user to replay the market precisely, incorporating effects like out-of-order packet arrival and microstructural features of the original market data messages that may not be found even in normalized MBO (L3) data. Many of our enterprise customers also rely on our PCAP files and normalized data simultaneously to audit our data integrity.

Databento’s PCAPs also include our PTP receive timestamps, which allow users to measure timestamp deltas that are necessary for accurate backtesting in sub-millisecond environments like TCA and smart order routing.

Since the PCAPs preserve the native feed protocol, latency-sensitive users can use PCAPs to test their parser and feed handler implementations and benchmark their trading systems through packet replay.

How are intraday PCAP files useful?

This solution was developed for several of our users who had time-sensitive pipelines for intraday or post-trade analytics.

While a few market data vendors provide PCAPs, none of them were able to provide deterministic guarantees on delivery time that were better than end-of-day (T+1) either due to licensing or technological limitations.

PCAPs files can be several GBs each after compression and can be slow to rotate and transfer out of capture servers at the colocation site. It’s possible to deliver PCAPs arbitrarily fast, but going to the extreme, a real-time stream of PCAPs will simply resemble the raw multicast feed and is inconvenient for a variety of data pipeline and ETL workflows that are designed around file or S3 access. Rotating the files too frequently, e.g. every minute, can be cumbersome on your ops team and file metadata.

With Databento’s extensive real-time distribution licenses and WAN bandwidth, we were able to create a fast path for these users to receive PCAP files intraday.

Intraday PCAP file deliveries are also ideal for customers who want to avoid real-time access fees, as delayed data license fees are usually much more affordable.

What’s the real-world performance of your intraday PCAP delivery?

Since going to production, we’ve been able to deliver the median PCAP file within 20 minutes of the first packet in the file. 99th percentile delivery time is within 43 minutes of the first packet in the file. We segment our PCAP files in 10-minute intervals, i.e. the file is delivered within 33 minutes of the last packet in the file at the 99th percentile.

Who uses Databento’s market data PCAPs?

Our existing users include major proprietary trading firms, quant hedge funds, leading ISVs, market data redistributors, and developers of low latency execution and feed handling solutions.

What other advantages set Databento’s PCAP service apart?

Our team started Databento after using every commercial PCAP service at our previous roles in tier 1 trading firms. Some of us even architected the existing capture setups that are in use at other data vendors today. So we’ve built our PCAP service to improve on the many limitations of legacy market data providers. If you’re looking for a PCAP provider, just ask: can your vendor do these?

• Lossless, full OPRA A/B tapes up to 100 gigabit line rate.
• Same data used and tested in our normalized data solutions, ensuring application-level checks (like book building) for data integrity on every ticker.
• Precise truncation of packets at the file timestamp boundaries.
• No out-of-order or missing file uploads unless there’s a data gap.
• Authorized distributor of ICE and NYSE market data and pcaps.
• Full order book and depth of market feeds from smaller US equity exchanges.
• Accurate annotation of network topology and RX deltas in capture system.
• FPGA-based hardware timestamps.

What is a depth of market (DoM) price ladder?

The DoM price ladder is a common way to visualize a limit order book in market-by-price (L2) form.

In the past, most commercial implementations of the DoM have been on native desktop applications. The origins of this type of visualization can be traced to desktop charting apps like that of Trading Technologies (which held a patent for the design since over 20 years ago, that has since expired). Market depth data is large and difficult to process, and it tended to be easier to implement the DoM visualization with the necessary performance on a native desktop application.

These days, with the speed of modern web browsers and client-side rendering techniques, it’s possible to implement a fast, snappy DoM as a web-based UI. This approach has seen many iterations on different trading platforms, especially seeing a widespread adoption on crypto exchanges like Coinbase.

This post highlights the typical stack you’d use to implement the DoM as a web-based browser application, similar to the one found on Coinbase.

Recommended design and tools

• React.js. For the web frontend with client-side rendering.
• Databento. For simple, fast access to market depth (L2) or full order book (L3) data. This type of data is typically very expensive, due to its size, licensing costs, and scarcity—and Databento’s usage-based pricing plans make it easier to get around these cost constraints.
• Rust. A performant, memory-safe programming language. Databento provides an official Rust client library, which makes it easy to implement the backend in Rust.
• WebSocket, SSE or WebTransport over HTTP/3. To communicate between the backend and frontend. At the time of this writing (Sep 2024), Databento has a raw TCP API for its real-time data and doesn’t expose a WebSocket API. Most web browsers don’t allow you to open raw TCP sockets due to security restrictions, so your frontend won’t be able to get data directly from Databento. But this is not an issue as most browser applications should have backend service layer that fetches the data; the backend service can expose real-time data over WebSocket, SSE, or any protocol that supports streaming.

Code and live demo

The code and implementation for this guide is contributed by Geoff Young (gty3 on GitHub).

See GitHub project for the code: https://github.com/gty3/dom-replay

See live demo: https://orderflowreplay.com/

These are key terms that you’ll find useful in navigating colocation and server hosting for a trading system, and also in describing how your system interacts with a trading venue’s matching engine.

Primary colocation. The site with the lowest latency to the matching engine. Usually the data center where the matching engine is housed, but this is not always the case—some venues only allow connectivity outside of the data center where their matching engine resides. For example, NYSE’s primary colocation site is in NYSE’s Mahwah data center.

Proximity hosting. Any non-primary colocation site offering a latency advantage due to proximity to the matching engine, typically within the same city or nearby. For example, any hosting location in New York or New Jersey could be considered a proximity hosting site for Nasdaq, whose primary colocation is in Carteret.

Point of presence (PoP). Any location offering direct connections to a trading venue without intermediaries other than the primary colocation site. For example, EBS has PoPs in Secaucus and Slough.

Latency equalization. Ensuring equal latency to the matching engine. Typically achieved by spooling optical fiber and ensuring equal length of cross-connects to the matching engine. Also called fiber equalization.

Handoff. The point where traffic passes between the matching engine’s and the trading participant’s networks. A cross-connect to a matching engine’s extranet.

Boundary switch. A network switch on which a trading venue’s handoff terminates.

Load balancer. A typical part of a matching engine. Distributes messaging traffic across servers or gateways that compose the matching engine.

Wire protocol. Specifies how to communicate with the matching engine. In financial trading environments, this often refers interchangeably to any communication protocol, and not only the physical and data link layers (OSI layers 1 and 2) as the term “wire” seems to imply. Includes application-level protocols (like ITCH, OUCH), which describe message structure for data and order entry, and session-level protocols (like MoldUDP64).

Direct market access (DMA). The ability to send orders directly to a matching engine without intermediaries.

Matching algorithm. The rules that determine order priority and how trades are matched. Examples include pure FIFO, pro rata, and price-broker-time.

Gateways. The servers that receive order messages or publish market data.