This is a small writeup I have made after spending a week and a half trying to derive the black scholes formula in hopes to easily understand how it works for future option strategies. This writeup covers the following topics:

• Binomial trees convergence to Black Scholes
• Brownian motion basics, (dW²) == dt
• Geometric brownian motion, log normal returns
• Itos lemma
• Randomness hedging to Black Scholes PDE

Keep in mind it does not cover out of scope advanced topics such as the solution of PDE via heat, Greeks, IV smile and so on. However the core concepts covered here should be sufficient as prerequisites on further understanding of said advanced topics.

Binomial trees converge to Black Scholes

Say I have a stock price at time t, Sₜ = 100. This stock at t + 1 can move up or down by the same factor 0.2: S_up = 120, S_down = 80. Cash grows by 1.05 per period and is denoted by B (bonds).

Suppose there is a call option out there with strike price K = 100, expiring in 1 time period, and the payoff of such option is 20 in the up state and 0 in the down state. How much should the option be priced to avoid arbitrage against a structured portfolio with the same payout?

Suppose such a portfolio exists. How would it be structured? The portfolio holds Δ shares of the stock and B in bonds, where Δ is the number of shares and B is the amount invested in bonds today. Its value today (time t) is:

Vₜ = ΔSₜ + B

And its value one period later (time t + 1), after the bond grows by 1.05, is:

Vₜ₊₁ = ΔSₜ₊₁ + 1.05B

So at t, the portfolio value is:

Vₜ = 100Δ + B

So for the following time period, matching the payout requirements based on the states, the portfolio returns would be:

120Δ + 1.05B = 20   → up state
 80Δ + 1.05B = 0    → down state

With 2 unknowns, Δ and B, we can solve them directly:

(120 − 80)Δ = 20 − 0
        40Δ = 20
          Δ = 0.5

80(0.5) + 1.05B = 0
     40 + 1.05B = 0
              B = −40 / 1.05
              B = −38.0952

We can substitute the unknowns into the original Vₜ formula to figure out the portfolio value at that time, which is the price of the call option:

Vₜ = 0.5(100) + (−38.0952) ≈ 11.90

C = 11.90

The probability of stock price movement is not used in the calculation. Since we matched the payoff of every state, this replication method will work regardless of the likelihood of the outcomes.

The option’s price is determined by:

• Current stock price
• Possible future stock prices
• Option payoff (match up/down payoff difference)
• Risk free rate (adjust payoff to correct level)

which is the foundation of the binomial option pricing model.

A related concept that uses these values is the risk neutral probability:

q = ((1 + r) − d) / (u − d)

q = risk neutral probability of upward movement
r = risk free interest rate per period (1.05 − 1 = 0.05)
u = up factor (1.2)
d = down factor (0.8)

In our case, q = (1.05 − 0.8) / (1.2 − 0.8) = 0.625.

The call option price using binomial model pricing would be:

C = (q·C_u + (1 − q)·C_d) / (1 + r)
  = (0.625(20) + 0.375(0)) / 1.05
  = 11.90

C_u = up state payoff (single step)
C_d = down state payoff (single step)

Now given a second time period with the same u, d and R, where the final stock prices are 144, 96 and 64, with matured call payoffs of 44, 0 and 0 respectively, we can construct a binomial tree like so:

Since the formulas we have used so far only work for 1 step, to get the option price C at t = 0 we have to work backward from t = 1. Using the binomial option pricing model, calculate q and C for both t = 1 nodes.

The 120 node:

q = ((1 + 0.05) − 0.8) / (1.2 − 0.8) = 0.625

C = (0.625(44) + 0.375(0)) / 1.05 = 26.19   → option value for t = 1, node #1

The 80 node:

q = 0.625 (same)

C = (0 + 0) / 1.05 = 0   → option value for t = 1, node #2

From that, we can calculate the 100 node at t = 0:

q = 0.625 (same)

C = (0.625(26.19) + 0.375(0)) / 1.05 = 15.59   → option value for t = 0

The general binomial sum formula that does all that is:

C₀ = (1 / Rᴺ) · Σⱼ₌₀ᴺ C(N, j) · qʲ · (1 − q)ᴺ⁻ʲ · max(S₀·uʲ·dᴺ⁻ʲ − K, 0)

Where N is the number of periods and C(N, j) is the binomial coefficient “N choose j”. This formula assumes u, d, R and q are constant across all N periods. This creates a recombining symmetric binomial tree.

In a more practical sense, the binomial model has a limit. Over one time period, besides the up and down factors being constant, the model does not tie u and d to the volatility of the underlying stock.

In the naive tree, u and d are inputs. So the volatility of price movement depends on the step size (how deep the tree is). In the CRR tree (Cox Ross Rubinstein), the input is the volatility, and u and d are then adjusted according to the step size. CRR replaces:

u → u = e^(σ√Δt)
d → d = 1/u

Δt = T/N

T = expiry in years
N = interval count

Since CRR scales the up and down step size based on N, assuming T is fixed, it demonstrates a Brownian motion property where the variance converges as N grows; the naive tree’s variance explodes as N grows.

The Brownian motion property is more apparent at high N, which allows the CRR option price to be close to the Black Scholes option price (which also has a Brownian motion assumption). Both models have similar inputs as well. Because of this, Black Scholes is viewed as the continuous version of CRR with a different modeling approach.

Brownian motion basics

A standard Brownian motion series W(t) has 4 properties:

1. Starts at zero: W(0) = 0
2. Normally distributed increments: for s < t, the change W(t) − W(s) follows a normal distribution with mean 0 and variance t − s, i.e. W(t) − W(s) ∼ N(0, t − s). Example: if t − s = 4, the increment W(10) − W(6) will have mean 0 and variance 4.
3. Independent increments
4. The path is continuous.

For rule 2, it is also implicitly stated that the variance of increments should ONLY scale with the time difference between observations, not with how frequent the observations are (discrete), which is why naive binomial trees are not considered Brownian motion.

To fulfill rule 2, the step size needs to be scaled with the number of steps for the whole interval. Referring to Δt = T/N, the step size of a standard Brownian motion needs to be √Δt. If we were to use, let’s say, Δt as the step size, the variance of one step would be:

Var(ΔW) = (Δt)²

After time t, we take t/Δt steps, and since the steps are independent, the total variance adds up:

Var(W(t)) = (t/Δt)·(Δt)²
          = t·Δt

As Δt → 0, the variance approaches zero. And since t·Δt ≠ t, the rule is violated. However, if we use √Δt as the step size, the variance of one step would be:

Var(ΔW) = (√Δt)²
        = Δt

And the total variance would be:

Var(W(t)) = (t/Δt)·Δt
          = t

Which is exactly the variance required by Brownian motion.

Brownian motion is random and it is not differentiable like ordinary calculus / smooth functions, which is important for Black Scholes modeling later.

The main difference between Brownian motion calculus and ordinary calculus is the identity rule dW² = dt. dW here represents a small change in the value of the Brownian motion and dt represents a small change in time. The equation

dW² = dt

expresses Brownian motion’s quadratic variation, [W]ₜ = t. Quadratic variation measures the total accumulated squared changes over time. For the quadratic variation of a normal smooth function like a parabola, for example

f(x) = x²  from x = 0 to x = 1

and we calculate its quadratic variation by dividing the function into 4 smaller parts:

We then calculate the function values:

x       f(x) = x²    Difference
0.00    0.0000       —
0.25    0.0625       0.0625
0.50    0.2500       0.1875
0.75    0.5625       0.3125
1.00    1.0000       0.4375

The changes in f(x) between consecutive points are also displayed. With 4 intervals, the approximate quadratic variation is the sum of squared changes:

QV = Σᵢ₌₁ⁿ (Δfᵢ)²

For the example: QV = 0.328125. As we divide the interval into more parts, each individual change gets smaller, so the sum of squared changes approaches zero.

[f]ₜ = 0

For Brownian motion, we can evaluate its quadratic variation over similar steps; start by dividing into smaller intervals:

Δt = T/n

Each Brownian increment satisfies:

ΔWᵢ ∼ N(0, Δt)

Now square each increment and sum them together:

Qₙ = Σᵢ₌₁ⁿ (ΔWᵢ)²

To evaluate that term, we can use the expected value of ΔWᵢ². For any random variable X:

Var(X) = E[X²] − (E[X])²
 E[X²] = Var(X) + (E[X])²

For a Brownian increment, E[ΔWᵢ] (mean) = 0 and Var(ΔWᵢ) = Δt. Thus:

E[ΔWᵢ²] = Δt + 0 = Δt

This means the expected square of each increment equals the length of the time interval. With this result, we can calculate E[Qₙ]:

E[Qₙ] = E[Σᵢ₌₁ⁿ (ΔWᵢ)²]
      = nΔt = T

The interpretation is that no matter how finely you divide the interval, the expected sum of squared increments is always T. With the expected value done, we need the variance of Qₙ as n → ∞ to complete the quadratic variation.

For a normal random variable X ∼ N(0, σ²), its fourth moment is:

E[X⁴] = 3σ⁴

Using the variance formula:

Var(X²) = E[X⁴] − (E[X²])²
        = 3σ⁴ − (σ²)² = 2σ⁴

For a Brownian increment, σ² = Δt. So:

Var((ΔWᵢ)²) = 2(Δt)²

Since increments are independent, the variances of the squares add:

Var(Qₙ) = Σᵢ₌₁ⁿ Var((ΔWᵢ)²) = n · 2(Δt)²

Replace Δt = T/n:

Var(Qₙ) = 2n(T/n)²
        = 2T²/n

Var(Qₙ) = 2T²/n → 0  as n → ∞

As we make the time intervals smaller, the variance of the total sum of squared increments approaches zero. Given two facts:

E[Qₙ] = T,   Var(Qₙ) = 2T²/n → 0

The mean remains T, while the variance approaches zero. Therefore, the sum converges to T in mean square. In probability terms:

Σᵢ₌₁ⁿ (ΔWᵢ)² → T

Which means:

• each individual increment is random
• the square of each individual increment is random
• when we add many small squared increments, it becomes less random
• the total approaches the deterministic length of the interval

This also gives us the shorthand dW² = dt. With that being said, the main differences between Brownian motion and a smooth function are:

Property                    Smooth function    Brownian motion
typical increment           Δt                 √Δt
sum of squared increments   → 0                → T
quadratic variation         0                  T

Itô’s lemma

Why is this distinction important? When doing calculus, we have to keep a second order term that ordinary calculus mostly discards, such as in the 2nd order Taylor expansion below for the function f(Wₜ):

df = f′(Wₜ)dWₜ + ½f″(Wₜ)(dWₜ)²

Apply the Itô rule (dW)² = dt and we get:

df = f′(Wₜ)dWₜ + ½f″(Wₜ)dt

This is the core concept of Itô’s lemma. Other Itô rules are as such:

Expression      Itô rule
(dt)²           0
dt·dWₜ          0
(dWₜ)²          dt
dWₜ·dt          0
dWₜ²            dt
d(f(Wₜ))        f′(Wₜ)dWₜ + ½f″(Wₜ)dt
d(f(t, Wₜ))     (∂f/∂t)dt + (∂f/∂W)dWₜ + ½(∂²f/∂W²)dt

The purpose of a Taylor expansion is to approximate a smooth function near a point using its derivatives. E.g. for f(x) = x², if we want to expand around x = 3, its Taylor expression is f(3 + h) = 9 + 6h + h². Putting h = 1, we are essentially computing f(4) or f(3 + 1), which both equal 16. The results from Taylor’s expansion are used in other calculus methods such as the chain rule.

However, an ordinary differential (normal Taylor, normal chain rule) describes how a function changes due to deterministic inputs, while a stochastic differential describes how a function changes when the inputs include random processes like Brownian motion. This is where and why Itô’s lemma is applied. Below are some examples.

f(Wₜ)  = Wₜ²               → original function
f′(Wₜ) = 2Wₜ,  f″(Wₜ) = 2   → derivatives

df = f′(Wₜ)dWₜ + ½f″(Wₜ)(dWₜ)²
   = 2WₜdWₜ + (dWₜ)²
   = 2WₜdWₜ + dt      → Itô's lemma version of 2nd order Taylor expansion

df = f′(Wₜ)dWₜ + ½f″(Wₜ)(dWₜ)²
   = 2WₜdWₜ + (dWₜ)²
   = 2WₜdWₜ           → ordinary version of 2nd order Taylor expansion, 2nd order dropped

As we can see, the two results are different. And only Itô’s lemma’s version holds true to the Brownian motion rules upon integration with expectations.

Ordinary version (does not match Brownian motion requirements):

E[W_T²] = 2E[∫₀ᵀ Wₜ dWₜ]
E[∫₀ᵀ Wₜ dWₜ] = 0
E[W_T²] = 0

Itô’s lemma version (meets Brownian motion requirements):

E[W_T²] = 2E[∫₀ᵀ Wₜ dWₜ] + T
E[∫₀ᵀ Wₜ dWₜ] = 0
E[W_T²] = T

Geometric Brownian motion and log normal returns

The increment of a Brownian motion can be defined like so:

dWₜ = W(t + dt) − W(t) ∼ N(0, dt)

dWₜ = increment of Brownian motion at time t
W(t + dt) − W(t) ∼ N(0, dt) = the difference between the value at t + dt
    and the value at t, drawn from a normal distribution with mean 0
    and variance dt

It can be simplified to:

dWₜ = √dt · Z,   Z ∼ N(0, 1)

Suppose we want to model a stock price Sₜ. We can use the following formula for percentage change:

dSₜ / Sₜ = μdt + σdWₜ

μdt  = average return component
σdWₜ = random Brownian path component
σ    = scale of fluctuation magnitude

This assumes the final magnitude of μdt and σdWₜ scales with the current price (geometric Brownian motion).

Say we want to get the log of this series. We don’t use Itô’s lemma correction, and we want the log for future arbitrary purposes.

Ordinary calculus formula:

d(ln S) = (1/S)dS

Apply it to our percent change equation:

d(ln S) = μdt + σdW

Which integrates to (to prove that the calculus is wrong):

ln Sₜ = ln S₀ + μt + σWₜ

And exponentiating gives:

Sₜ = S₀ · e^(μt + σWₜ)

Remember this equation, which we will use later on to prove it’s incorrect. Now suppose the drift is zero, μ = 0. The original stock price model is dSₜ = σSₜdWₜ. Since there is no drift and the Brownian random component’s increments have mean 0, the model requires:

E[dSₜ | Sₜ] = 0

The expected value of price changes is zero. However, when using Sₜ = S₀·e^(μt + σWₜ) to predict E[dSₜ | Sₜ], it is positive. It is not zero, which contradicts the requirements, therefore it is wrong.

The correct version with Itô’s lemma applied is:

d(ln S) = (μ − ½σ²)dt + σdWₜ   → notice the ½σ² correction term

ln S_T = ln S₀ + (μ − ½σ²)T + σW_T

With the logarithm, the Brownian motion at time T follows a normal distribution, W_T ∼ N(0, T). Multiplying by σ gives σW_T ∼ N(0, σ²T).

Adding a constant changes the mean but not the variance, validating that geometric Brownian motion changes are log normal. However, arithmetic Brownian motion changes aren’t log normal because they do not scale with price.

By solving the logarithm by exponentiating, we get S_T = S₀·e^((μ − ½σ²)T + σW_T). We can interpret that solution by calculating the mean and median:

E[S_T]      = S₀ · e^(μT)
Median(S_T) = S₀ · e^((μ − ½σ²)T)

Notice that they differ by a term: −½σ². It is the difference in growth rate between the mean and median price, which gives us a skewed distribution for returns (log normal).

Randomness hedging to Black Scholes PDE

If we combine an option with the right number of shares, we can eliminate stock price randomness. Once the portfolio is riskless, no-arbitrage tells us how much it should earn, which gives us the Black Scholes PDE:

∂V/∂t + ½σ²S²(∂²V/∂S²) + rS(∂V/∂S) − rV = 0

To understand that, we can first use the simpler binomial model as an example. Suppose:

S = 100,  S_u = 120,  S_d = 90,  V_u = 30,  V_d = 5

S = stock price
V = option value

And we have a portfolio like so:

Π = V − ΔS

If we don’t have the stock hedge ΔS and only hold the option V, our portfolio value will be:

Π_u = V_u = 30
Π_d = V_d = 5

In the up and down states. However, the final portfolio is uncertain, as the up and down states have different values. That’s why the ΔS term is there. We can solve for Δ using S_u and S_d

Π_u = V_u − ΔS_u
Π_d = V_d − ΔS_d
Π_u = Π_d

Δ = (V_u − V_d) / (S_u − S_d) = 5/6

Verified by:

Π_u = 30 − (5/6)(120) = 30 − 100 = −70
Π_d =  5 − (5/6)(90)  =  5 − 75  = −70
Π_u = Π_d

For the calculus (continuous) version, it is a similar concept. We start from the stock price model from geometric Brownian motion:

dS = μSdt + σSdW

As well as the differential of option value relative to the stock price:

dV = (∂V/∂t)dt + (∂V/∂S)dS

Above is the chain rule version without Itô’s lemma. Once it is applied, we get:

dV = (V_t + μSV_S + ½σ²S²V_SS)dt + σSV_S dW

V_t  = ∂V/∂t
V_S  = ∂V/∂S
V_SS = ∂²V/∂S²

We then construct a portfolio similar to the binomial tree model:

Π = V − ΔS

Over an interval, the calculus version (instantaneous change) is:

dΠ = dV − ΔdS

Substitute the equations from earlier and group:

dΠ = (V_t + μSV_S + ½σ²S²V_SS − ΔμS)dt + σS(V_S − Δ)dW

In this scenario, instead of a certain (hedged) portfolio value remaining identical across two states, the change in the portfolio depends only on time. There will be no random influence (dW) now. To do so, choose Δ so that:

Δ = V_S  ⟹  dΠ = (V_t + ½σ²S²V_SS)dt

A delta hedged portfolio is constructed. Under Black Scholes assumptions, a riskless portfolio must earn the risk free rate r for there to be no arbitrage opportunities. Thus:

dΠ = rΠdt,   Π = V − ΔS,   dΠ = r(V − SV_S)dt

We can combine the two equations:

dΠ = (V_t + ½σ²S²V_SS)dt
dΠ = r(V − SV_S)dt

which gives us the Black Scholes PDE:

V_t + ½σ²S²V_SS = rV − rSV_S

∂V/∂t + ½σ²S²(∂²V/∂S²) + rS(∂V/∂S) − rV = 0

Thank you for reading Issue 33 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

Zero to Black Scholes PDE was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Balaena Quant Insights: Issue 32

Bootstrapping variants used in time-series Monte Carlo simulation are explored theoretically and experimentally.

1. Motivation for Stationary Bootstrapping

Bootstrapping. This is a Monte Carlo simulation technique where data observations are randomly drawn N times with replacement to form a new bootstrap sample, a process known as resampling (see Figure 1.1). This approach, however, is inappropriate for time-series data as it breaks serial dependence between successive data observations.

For example, each observation in a BMI dataset corresponds to a different individual, exerting no influence on the next observation. Datasets of this nature are known as independently and identically distributed (i.i.d.). On the other hand, in a heart rate dataset for a single person, a value at 8:00:00 influences the value at 8:00:01 to a certain degree. We do not expect our heart rate to jump erratically from 60 bpm to 180 bpm, but rather a gradual transition from 60 bpm to 70, 90, 120, 160 bpm and so on. The observations in the heart rate dataset exhibit an inherent serial relationship that is not accounted for by the default bootstrapping approach.

For exposition, we assume that the heart-rate process is approximately stationary over the interval considered.

Block bootstrapping (BB). A simple way to circumvent this issue is to group adjacent data into either overlapping or non-overlapping blocks of a fixed size B before performing resampling (see Figure 1.2). Instead of raw observations, whole blocks are drawn at random with replacement to form a bootstrap sample, preserving the serial relationship within each block, although dependence between block boundaries is still inevitably broken.

It follows that the block size B should be carefully determined based on the characteristics of the dataset, such that the resulting cross-block dependency is not a concern to our research objectives. Suppose our heart rate is recorded every second. B should be set to 1 minute if we are interested in continuous heart rates per minute. Set B too small and we obtain results equivalent to i.i.d. bootstrapping; set it too large and we deprive the resampling process of sufficient data units to generate effective bootstrap samples. Indeed, B is a difficult number to set.

Since many variants exist under this umbrella term, we introduce only two of the most notable ones in this article.

Note: the non-overlapping variant is referred to as simple block bootstrapping (SBB) below.

Moving block bootstrapping (MBB). SBB underrepresents data observations at the edge of a block; having fewer neighbours than those situated in the centre of the block, the local dependence of these edge observations is not fully captured. This approach uses overlapping blocks that move forward by one observation like a sliding window, giving each data observation an equal chance to be at the block centre, which prevents edge bias and smooths the hard dependency cutoff between block boundaries. Each observation now appears in multiple blocks.

We continue to use a block size of 1 minute as an example. Under non-overlapping BB, the heart rate at 8:00:59 is always situated at the edge of the 8:00:00–8:00:59 block, so its relationship with 8:01:00 is always severed. Under MBB, however, another block may span 8:00:30–8:01:29, placing 8:00:59 near its centre and preserving its relationship with observations on both sides.

Nevertheless, remnants of the edge bias can be found at MBB’s data-series level: observations towards the edge appear fewer than those in the centre of the series. In the same heart rate dataset, an observation recorded near the beginning of the monitoring period can belong only to the first few overlapping blocks, whereas an observation near the middle can appear in many more. Figure 1.3 is a visualisation of MBB and the edge problem.

Circular block bootstrapping (CBB). CBB modifies MBB to resolve the edge problem at the series level by treating the series as a continuous circle, allowing blocks to wrap around from the last data point to the first index. Suppose our heart rate dataset contains observations from 8:00:00 to 8:59:59. A 1-minute block beginning at 8:59:30 would therefore contain observations from 8:59:30 to 8:59:59 before wrapping around to include 8:00:00 to 8:00:29. This allows observations near 8:00:00 and 8:59:59 to occupy the same range of positions within candidate blocks as observations in the middle of the dataset, eliminating MBB’s series-level edge bias. Figure 1.4 shows a visual summary for non-overlapping BB, MBB and CBB.

CBB is not without limitations, mainly attributable to the predictability of the block boundary position. Returning to our heart rate dataset, we know that each observation at time t exhibits influence on the next observation at t+1, but it is also possible that this influence extends to t+k. This relationship between observations can be expressed as a joint distribution. For example, using values at t on the x-axis and values at t+1 on the y-axis, we can plot the coordinate for the pair 8:00:01 and 8:00:02, followed by 8:00:02 and 8:00:03, and so on until all (t, t+1) pairs in the series have been plotted. Multiple joint distributions for (t, t+2), (t, t+3), …, (t, t+k) can be plotted in the same way, generating correlation-like graphs with varying degrees of dispersion and dependence, as illustrated in Figure 1.5.

Using a fixed block length B, however, breaks serial dependence at predictable positions.

Suppose we construct a bootstrap sample using blocks of size 60, such that one sampled block occupies positions 0–59 and the next occupies positions 60–119. Consider two pairs separated by the same lag of four observations (lag-4). The pair at positions 55 and 59 lies entirely within the first block and therefore preserves the local lag-4 dependence of the original series. In contrast, the pair at positions 59 and 63 crosses the block boundary: the observation at position 59 originates from one sampled block, while the observation at position 63 originates from another independently sampled block. Their original local relationship is therefore disrupted. See Figure 1.6 for a visualisation.

Different joint distributions across different lags are not themselves an issue; (t, t+1) may naturally exhibit stronger dependence than (t, t+4). The problem is that, for any fixed lag k, whether dependence is preserved becomes predictable from t mod B. In general, a (t, t+k) pair crosses a block boundary whenever (t mod B) + k ≥ B. Consequently, the bootstrap joint distribution at a given lag depends not only on k, but also on the position of t within the fixed block. This periodic dependence on t mod B violates the stationarity property.

2. Stationary Bootstrapping

Stationarity is desirable for a number of reasons. First, it ensures that the statistical relationship between observations depends on their separation in time rather than their absolute position in the series. A lag-4 relationship, for example, should behave similarly whether it occurs near the beginning, middle or end of the sample. Second, many commonly used time-series statistics, including autocovariance and autocorrelation, are defined under the assumption that these relationships remain stable over time. Finally, a stationary bootstrap avoids introducing artificial periodic structures that are absent from the original data-generating process, allowing the bootstrap samples to reproduce its dependence structure more faithfully. We do not want a bootstrap sample of block size 60 to indicate that every 60th observation is special; that structure only exists because we chose B = 60. Lag-k pairs at any t should have the same distributional relationship.

Politis and Romano developed the stationary bootstrapping (SB) algorithm precisely to tackle this issue [1]. Their solution is to replace the fixed block size B in CBB with a random block length drawn from a geometric distribution, which models the number of trials required before the first success.

In SB, a “success” can be interpreted as the end of the current block and the start of a newly sampled one. For example, rather than forcing every heart-rate block to contain exactly 60 observations, we may set the probability of ending a block at each observation to (p=1/60). One sampled block may therefore contain 23 observations, the next 81, and another 47, while the expected block length remains 60 observations. As a result, block boundaries no longer occur predictably at positions 60, 120, 180, and so on; instead, every position has an equal chance of becoming a boundary. See Figure 2.1.

In their 1994 paper, Politis and Romano found that SB is not only better suited to preserving stationarity than MBB, but that its variance estimates are also less sensitive to the choice of the geometric parameter p. In other words, small changes in p tend to produce smaller fluctuations in the estimated variance than comparable changes in the fixed block length B under MBB. This makes parameter selection somewhat less critical in SB: although an appropriate value of p must still be chosen, the resulting inference is generally more robust to moderate misspecification of this parameter.

3. Experimental Comparison

Results in this section are contributed by Jun Han.

3.1 Simulated Data and Resampled Bootstrap Series

To illustrate the practical differences between these methods, we apply SBB, MBB and SB to the same simulated autoregressive AR(1) series. We examine whether the resampled bootstrap series retains local serial dependence while avoiding artefacts introduced solely by the resampling procedure.

Figure 3.1.1 compares individual bootstrap series generated by SBB, MBB and SB against the original AR(1) series. All three methods preserve contiguous stretches of observations because observations are resampled in blocks rather than individually. Consequently, much of the short-run movement within each sampled segment resembles movement that actually occurred in the original series.

The effect of these block transitions can also be examined through rolling statistics. Figures 3.1.2 and 3.1.3 compare the rolling mean and rolling standard deviation of the original series with those calculated from each bootstrap sample. The bootstrap curves are not expected to follow the original curves at corresponding time indices because the blocks have been reordered. Rather, these plots provide a visual indication of how local levels and variability are carried into the newly constructed series.

Within individual sampled blocks, the bootstrap series inherits the local level and variability of the corresponding portion of the original data. When a rolling window begins to include observations from two independently sampled blocks, however, observations originating from substantially different regions of the original series may be combined. This can generate abrupt movements in the rolling mean and, in particular, temporary spikes in rolling volatility.

Under SBB and MBB, these disruptions are associated with a fixed spacing between bootstrap block boundaries. Their precise appearance in the rolling statistics also depends on the width of the rolling window, since a boundary affects the statistic for as long as that boundary remains inside the window. SB exhibits the same general phenomenon: randomly concatenating two segments can still produce a large local discontinuity, but the affected locations are no longer tied to a deterministic block grid.

We reiterate that SB is not used to eliminate discontinuities entirely; instead, its objective is to prevent the resampling algorithm from systematically assigning those discontinuities to particular positions in the bootstrap series.

3.2 Testing the Stationarity Artefact

We test the stationarity argument developed in Section 2 by examining the empirical lag-1 joint distribution of the bootstrap process conditional on the position of an observation within the fixed block grid. Since the artefact exists in all fixed-block approaches (SBB, MBB, CBB), MBB is selected as the representative method for our experiment. This choice also aligns our results with Politis and Romano’s comparison between MBB and SB in their paper.

For MBB with B=10, Figure 3.2.1 separates the pairs (t, t+1), according to t mod 10. At offsets 0–8, the two observations normally belong to the same sampled block, and a strong positive relationship remains visible. Offset 9 is fundamentally different: t+1 necessarily belongs to the next sampled block. Since that block is selected independently, the original lag-1 relationship is disrupted and the joint distribution becomes substantially more dispersed.

This provides a direct empirical demonstration of the problem with fixed block lengths. Every panel represents the same lag-1 relationship, (t, t+1), yet its distribution changes according to t mod B.

The same experiment is repeated using SB with expected block length of 10, corresponding to p=1/10. Although SB does not possess a fixed block size of 10, we retain the same modulo grouping purely as a diagnostic device. If SB contained the periodic structure present under MBB, one of these offsets would exhibit systematically different dependence. The result is shown in Figure 3.2.2.

No offset now plays the special role observed at offset 9 under MBB. Because every observation has the same probability p of terminating the current block, cross-block pairs occur probabilistically throughout the bootstrap series rather than at one predictable position.

3.3 Sensitivity to Parameter Selection

Block bootstrap methods introduce another practical problem: the researcher must select a block length, and must do so with care. We therefore repeat the bootstrap procedure over a range of block lengths and estimate the variance of the bootstrap sample mean. For SBB and MBB, b represents the fixed block length; for SB, b=1/p represents the expected block length.

As shown in Figure 3.3.1, SBB exhibits considerable variation in the estimated variance as the block length changes. MBB is less erratic over much of the range, although substantial sensitivity remains, particularly at smaller block lengths. In this experiment, SB produces estimates concentrated within a comparatively narrow range as its expected block length varies.

This experiment shows that SB is less sensitive to moderate changes in the tuning parameter for this simulated process and statistic. This is a useful property when the true dependence horizon of the underlying time series is uncertain.

Another interpretation offered by Politis and Romano helps explain why SB may be less sensitive to the selection of a particular block length. For each fixed block length b, MBB produces a corresponding bootstrap distribution, or, equivalently, an estimate such as the standard error of a statistic. SB can be viewed as essentially a weighted average across these MBB distributions, with the weights determined by the geometric distribution. Rather than placing all weight on a single block length B, SB distributes weight across many possible block lengths while controlling their average through p, where E[L]=1/p. In this sense, p acts as a smoothing parameter over block lengths.

The comparatively stable SB curve observed in Figure 3.3.1 is consistent with this smoothing interpretation. Changing the expected block length alters the geometric weighting over block lengths rather than switching abruptly from one fixed block length to another.

4. Conclusion

Stationary bootstrapping was developed to address a specific weakness of fixed-block bootstrap methods: although SBB, MBB and CBB preserve local serial dependence within blocks, their fixed block lengths create predictable boundaries at which that dependence is broken. This introduces an artificial periodic structure into the bootstrap series, causing the joint distribution of a lag-k pair to depend not only on k, but also on the observation’s position relative to the block boundary, t mod B.

This problem is resolved by replacing the fixed block length in CBB with a geometrically distributed random block length. Each observation from SB therefore has the same probability of terminating the current block, removing the deterministic t mod B structure while retaining contiguous stretches of the original series. Due to comparatively stable variance estimates across changes in its expected block length, SB also has a practical advantage in parameter selection. SB can be essentially viewed as a smoothed version of MBB.

SB should therefore not be viewed as a method that eliminates block-boundary discontinuities altogether; those discontinuities remain unavoidable whenever independently sampled blocks are concatenated. Its advantage is that these breaks occur probabilistically rather than according to an artificial periodic schedule, allowing the bootstrap series to preserve the stationarity structure of the underlying process more faithfully.

Thank you for reading Issue 32 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Politis, D.N. and Romano, J.P. (1994) ‘The stationary bootstrap’, Journal of the American Statistical Association, 89(428), pp. 1303–1313. doi:10.1080/01621459.1994.10476870.

What Makes the Stationary Bootstrap Stationary? was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

The effect size r is one of the most useful companions to a p-value: significance tells us whether something moved, whereas effect size tells us how much. But there is a quiet trap in how r gets used in practice. When it is computed against a fixed benchmark, it behaves the way we expect. When it is used to rank or compare between competing features, strategies, or model variants, it can quietly mislead. This article stress-tests r across a battery of synthetic scenarios to map out exactly where that breakdown happens, and what a researcher should do about it.

This article is contributed by my colleague Jun Han!

1. Introduction

1.1 Background on Effect Size

A p-value tells us whether an observed difference is unlikely to have arisen by chance. For example, under a standard normal distribution with mean = 0 in Figure 1.1.1, the probability of observing x > 3 is 0.00135 (0.135%), way below the typical significance threshold of 0.05 (5%), where such an observation is very unlikely under this zero-mean hypothesis. We would then assume that mean != 0, given this extreme observed value. For example, it would mean that we conclude a strategy has alpha after observing a mean Sharpe of 3 out of 100 backtest trials (assuming these trials come from fully independent market data).

However, the p-value says nothing about magnitude. Two results can carry identical p-values while one reflects an economically trivial improvement and the other a substantial one.

Eq 1 shows the formula for the t-statistic, which is subsequently used to calculate the one-tailed p-value in Eq 4. Suppose we observe 100 independent Sharpe-ratio estimates. The numerator in Eq 1 is the difference between the average observed Sharpe ratio and the Sharpe ratio specified under the null hypothesis. The denominator is the standard error of the average Sharpe ratio, calculated by dividing the standard deviation of the 100 Sharpe estimates by the square root of 100. The standard error measures how much the estimated average Sharpe ratio would vary across repeated samples under the relevant assumptions. For example, an estimated Sharpe ratio of 3 with a standard error of 0.5 would have a 95% confidence interval of roughly[2.02, 3.98], under an approximately normal sampling distribution. Overall, the t-statistic measures the observed difference from the null value relative to the sampling uncertainty of that estimate.

On the other hand, Eq 2 shows the formula for Cohen’s d effect size. Eq 2 closely resembles Eq 1, and substituting Eq 2 into Eq 1 produces Eq 3. Cohen’s d does not directly depend on the sample size; it is the ratio of the observed mean difference to the variability of the individual observations. It therefore measures the magnitude of the observed effect in standard-deviation units. In contrast, the t-statistic in Equation 3 contains a term involving the sample size. As the sample size increases, the standard error decreases, making the estimated effect more precise and increasing the magnitude of the t-statistic for a fixed Cohen’s d. Consequently, the same observed effect size can produce radically different p-values depending on n.

This is precisely why effect size metrics such as Cohen’s d, its paired variant dz, or the rank-based r derived from Wilcoxon/Mann-Whitney statistics are reported alongside significance. They answer this question: assuming this is real, how large is the improvement?

However, the implicit assumption behind using r this way is that it behaves consistently as a ruler, that a larger r reliably means a larger, more decision-relevant improvement, regardless of what it is being compared to. This assumption holds reasonably well when comparing a single feature against a fixed, external benchmark. It becomes far shakier when r is used to rank multiple candidate features or strategies against each other, since each comparison carries its own variance structure, its own outliers, and its own distributional quirks. Two features can produce the same r while representing completely different underlying behaviour, or different r while representing economically similar behaviour. This article is an attempt to catalogue when and why that happens.

1.2 Experiment Design Overview

Every case in this study is run through four significance tests, split along two axes: parametric vs. non-parametric, and paired vs. grouped.

paired_parametric_test -> paired t-test, Cohen's dz
paired_nonparametric_test -> Wilcoxon signed-rank, rank-biserial r
grouped_parametric_test -> Welch's t-test, Cohen's d (pooled SD)
grouped_nonparametric_test -> Mann–Whitney U, rank-biserial r

Parametric tests are statistical significance tests that assume the data follow a specific distribution (often a normal distribution) and rely on its parameters, such as the mean and standard deviation, to compute test statistics and effect sizes. This means parametric tests preserve more information in the data. Hence, parametric tests can be more powerful (fewer false negatives—but we would need to know the ground truth to confirm this) than non-parametric tests when their assumptions are satisfied. However, their results can be misleading when key assumptions are violated (such as normality).

Non-parametric tests do not assume a specific distribution. Instead, they typically use ranks or other transformations of the data to compute test statistics. This reduces sensitivity to outliers and distributional assumptions, making the test more robust, though this can also cost statistical power when parametric assumptions are actually met, due to information loss during the rank transformation.

Orthogonal to that split is paired vs. grouped design:

• Paired tests look at the difference within each matched pair, aggregating that single difference-value across all pairs. Between-pair variability is removed by construction, since each pair is only ever compared to itself.
• Grouped tests treat the two samples as independent, unrelated populations, comparing their overall distributions (means, medians, or full empirical distributions). Every observation now contributes to estimating group-level variance, so the result reflects both within-group and between-group variation.

Because paired designs remove between-pair variance, they are more sensitive to a real effect than the equivalent grouped design; paired tests will tend to flag more positive results under the same underlying conditions. Since group tests are usually less sensitive, passing a group test will usually mean passing the pair test under the same conditions, though the reverse is not guaranteed.

2. The Base Cases

All cases share a common reference series: 100 draws from a standard normal distribution (np.random.seed(12)), which we perturb in eleven distinct ways to simulate different “improvement” regimes a researcher might realistically encounter: consistent gains, inconsistent gains, dampened gains, degradations, and pure noise.

For each case we look at the line plot and histogram of the perturbed series against the base, alongside the difference series, then report all four (p, r) pairs.

2.1 Case 1: Consistent Minor Improvements

• Paired parametric: p = 1.73e-11, r = -0.760
• Paired nonparametric: p = 2.60e-10, r = 0.621
• Grouped parametric: p = 0.597, r = -0.075
• Grouped nonparametric: p = 0.282, r = 0.041

The paired tests register both significance and a strong effect size, while the grouped tests do not, exactly as the paired/grouped variance argument above would predict. A small, consistent shift is invisible at the population level but stands out clearly once we control for the pairing.

2.2 Case 2: Consistent Major Improvements

• Paired parametric: p = 7.50e-99, r = -9.452
• Paired nonparametric: p = 1.95e-18, r = 0.868
• Grouped parametric: p = 2.96e-10, r = -0.939
• Grouped nonparametric: p = 3.10e-10, r = 0.437

All four tests now agree on significance and a large effect, as expected once the shift is big enough to separate the groups outright. Note, however, that the grouped non-parametric test reports only a moderate effect size relative to its parametric counterpart, and that the non-parametric effect sizes here differ substantially from Case 1’s, despite both being “clean, consistent improvement” scenarios. This hints at the fact that that randomness in the difference series itself, not just its mean, is already distorting rank-based effect size. Extreme Case 1 later isolates this by removing that randomness entirely.

2.3 Cases 3–6: Mixed Improvement and Degradation

These four cases combine a small, consistent improvement across most of the series with a degradation carved into a subset of observations, varying both the size of the degradation (minor vs. major) and its concentration (a contiguous late-sample block vs. a scattered majority of the sample).

Case 3: mostly consistent minor improvement, minor degradation in the last 30 points.

• Paired parametric: p = 1.28e-04, r = -0.399
• Paired nonparametric: p = 1.09e-04, r = 0.370
• Grouped parametric: p = 0.543, r = -0.086
• Grouped nonparametric: p = 0.247, r = 0.049

Paired tests agree on significance with a similar, moderate effect size; grouped tests remain insignificant, again consistent with expectations, since the net population-level shift is small.

Case 4: mostly consistent minor improvement, major degradation in the last 30 points.

• Paired parametric: p = 5.20e-05, r = 0.423
• Paired nonparametric: p = 0.564, r = -0.016
• Grouped parametric: p = 0.028, r = 0.313
• Grouped nonparametric: p = 0.960, r = -0.124

This is the first genuine surprise. The grouped tests stay (mostly) insignificant, as expected, but the paired parametric effect size has flipped sign relative to Case 3. A large, concentrated degradation is enough to drag the mean-based statistic in the wrong direction, even though the majority of individual pairs still improved. Meanwhile the paired non-parametric test collapses to near-zero significance, which runs counter to naive intuition: since most pairs still improved, a rank-based test driven by the frequency of positive vs. negative changes should still register significance. The fact that it does not tells us that the magnitude of the tail, not just sign, is already leaking into the rank statistic, a thread we pick up properly in Extreme Case 2.

Case 5: mostly inconsistent minor improvement, minor degradation across 70 points.

• Paired parametric: p = 0.189, r = -0.130
• Paired nonparametric: p = 0.110, r = 0.120
• Grouped parametric: p = 0.150, r = -0.200
• Grouped nonparametric: p = 0.083, r = 0.090

All four tests report low significance and small effect sizes, as expected. This is the mirror image of Case 3’s clean signal, now diluted across most of the sample.

Case 6: mostly inconsistent minor improvement, major degradation across 70 points.

• Paired parametric: p = 3.51e-23, r = 1.302
• Paired nonparametric: p = 1.00, r = -0.708
• Grouped parametric: p = 4.63e-11, r = 0.989
• Grouped nonparametric: p = 1.00, r = -0.458

Here the large, widespread degradation dominates the mean in both paired and grouped parametric tests, correctly flipping their sign and significance. The non-parametric tests, constrained by the one-sided greater alternative hypothesis used throughout this study, report p ≈ 1.00. This is essentially a hard ceiling rather than a graceful degradation. Their effect sizes flip in the same direction as the parametric tests. The direction is right; the significance readout, however, is a blunt instrument here.

2.4 Cases 7–8: Random, Unstructured Change

Case 7: random large changes, net improvement (loc=0.5, scale=2.5).

• Paired parametric: p = 0.188, r = -0.130
• Paired nonparametric: p = 0.173, r = 0.120
• Grouped parametric: p = 0.154, r = -0.200
• Grouped nonparametric: p = 0.083, r = 0.090

Even the grouped tests fail to detect significance despite a genuine positive shift in the mean, the scale of the noise (2.5) simply swamps the signal (0.5). This is a useful sanity check. It is not that the mean shift is undetectable in principle, but that the signal-to-noise ratio here is too low for either test design to separate it from chance at this sample size. We revisit this exact tension of magnitude vs. noise more rigorously in Extreme Case 3.

Case 8: random large changes, net degradation (loc=-0.5, scale=2.5). Mirrors Case 7 in the opposite direction, with comparably weak, insignificant readings across the board.

2.5 Cases 9–10: Dampened Series

Case 9: dampened, general improvement (loc=0.5, scale=0.1).

• Paired parametric: p = 3.41e-08, r = -0.599
• Paired nonparametric: p = 3.92e-08, r = 0.537
• Grouped parametric: p = 2.80e-08, r = -0.852
• Grouped nonparametric: p = 7.88e-10, r = 0.426

All four tests report significance, with parametric effect sizes larger than their non-parametric counterparts, consistent with the mean genuinely having shifted. Notably, the non-parametric tests also register strong significance despite not directly using the mean, suggesting that a small, tightly dampened improvement produces an almost uniformly one-sided ranking, and that uniformity (not magnitude per se) is what the rank-based tests are really picking up on.

Case 10: dampened, general degradation (loc=-0.5, scale=0.1). The mirror image, with the same pattern reversed in sign.

2.6 Case 11: Pure Noise

With case_11 drawn from the same distribution as the base series and unrelated to it, every test correctly reports insignificance, which is the expected null-case control.

3. Extreme Case 1: Fixed Improvement

This case investigates test robustness under a fixed (zero-variance) improvement: base + 1 against base + 0.1, an edge case designed to expose a specific quirk of the paired non-parametric test.

{
    "paired_parametric_test": [
        0.0,
        -1.4733519609459446e+16,
        1.4733519609459443e+17
    ],
    "paired_nonparametric_test": [
        2.8477952243481438e-21,
        0.8681770230106196,
        5050.0
    ],
    "grouped_parametric_test": [
        1.8219343140311188e-10,
        -0.9512758455404217,
        6.72653601160599
    ],
    "grouped_nonparametric_test": [
        2.610460882313259e-10,
        0.4393634812506797,
        7543.0
    ]
}
=========
{
    "paired_parametric_test": [
        0.0,
        -2109630338343290.2,
        2.10963033834329e+16
    ],
    "paired_nonparametric_test": [
        8.041198668367257e-19,
        0.8681770230106196,
        5050.0
    ],
    "grouped_parametric_test": [
        0.5019519437845776,
        -0.0951275845540422,
        0.6726536011605992
    ],
    "grouped_nonparametric_test": [
        0.22475599597627755,
        0.053559842386044325,
        5310.0
    ]
}

The paired parametric test’s r correctly scales with the size of the fixed shift. The paired non-parametric test’s r, however, remains constant across both scenarios. This is a genuine edge case. Because a fixed additive shift produces an identical rank ordering of the differences regardless of its magnitude, the Wilcoxon-derived effect size cannot distinguish between “everyone improved by 0.1” and “everyone improved by 1”. Translated into Sharpe, it would mean that the Wilcoxon r produces the same effect size for a 0.1 Sharpe vs. a 1 Sharpe improvement.

In a decision making process, this will give the illusion that two of the improvements are equal, but when inspected from another multiple other point of views, it is clear that case_1a is the better choice since the improvement has higher magnitude. In practice, such fixed or low variance improvements are rare and highly unlikely, and can be easily discerned by analysing the magnitude of differences manually. But it is a reminder that r alone, decoupled from a look at the raw data, is not a safe ranking criterion in isolation.

4. Extreme Case 2: Extreme Outliers

This case investigates the breaking point of parametric effect sizes under outlier contamination, by constructing pairs of series that arrive at similar reported effect sizes through completely different underlying mechanisms.

Grouped test pair: e_case_2a = base + 0.45 (a uniform, moderate shift) is compared against e_case_2b, which applies only a tiny +0.01 shift but then forces ten observations to a value of 11, representing extreme, isolated outliers. Despite these being economically very different scenarios, the grouped parametric test can be tuned to produce comparable effect sizes for both.

[
    0.002798804211235985,
    -0.4280741304931898,
    3.026941205222696
]
=========
[
    0.0028368352728220013,
    -0.43126310155914277,
    3.049490635880126
]

Paired test pair: the same idea is reproduced for the paired test: e_case_2c (moderate, noisy improvement, scale=0.4) versus e_case_2d (a nearly negligible loc=0.01 improvement, with a single observation forced to 1.1) — again converging on similar paired parametric effect sizes.

[
    0.016302859468366158,
    -0.24438056783618003,
    2.4438056783618
]
=========
[
    0.012434086247713342,
    -0.25461311914285856,
    2.5461311914285853
]

As we can see, the parametric tests are susceptible to outliers, producing misleading results. It is remarkably easy to construct two cases that produce the same effect size while one is contaminated by outliers. However, from a practical standpoint, e_case_2a should be the better option due to the more frequent moderate improvements rather than e_case_2b. The latter has frequent insignificant improvements with a few large outliers, which may be artefacts from other factors like method sensitivity, data errors, regime sensitivity.

The pair test counterpart is also not safe from this phenomena. We are able to produce two different sets of improvements which results with similar effect sizes as well. The reasoning for decision making will be the same as the group tests. Reproducing the outlier illusion in the paired case required more moving parts (a separate noise term and an arbitrarily placed outlier) than the grouped case, suggesting paired tests are somewhat more resistant to this failure mode, though “more resistant” is very different from “immune,” and unlike the zero-variance fixed-improvement scenario in Extreme Case 1, contaminated-outlier scenarios of this kind are entirely plausible in real return series. For example, a single anomalous trading day, a data vendor glitch, or a fat-tailed regime event.

5. Extreme Case 3: Magnitude

This final case isolates the specific weakness of rank-based effect sizes when the magnitude of improvement, not just its frequency, is the decision-relevant quantity.

e_case_3a = base + N(0.05, 0.05) produces small, tightly-clustered, highly consistent improvements. e_case_3b = base + N(1.3, 1.5) produces improvements with a much larger mean but far more variance: some pairs improve enormously, others barely move or reverse.

{
    "paired_parametric_test": [
        5.088721094582044e-96,
        -8.842837869832993,
        88.42837869832994
    ],
    "paired_nonparametric_test": [
        1.948279922547955e-18,
        0.8681770230106196,
        5050.0
    ],
    "grouped_parametric_test": [
        0.5008135252860949,
        -0.09538128592721373,
        0.6744475407742584
    ],
    "grouped_nonparametric_test": [
        0.22038630639866386,
        0.05459648449674196,
        5316.0
    ]
}
=========
{
    "paired_parametric_test": [
        8.230452535801955e-17,
        -1.0056658827798561,
        10.056658827798561
    ],
    "paired_nonparametric_test": [
        1.371561541955633e-13,
        0.7306440292663631,
        4650.0
    ],
    "grouped_parametric_test": [
        4.34867968588553e-08,
        -0.8078130763143232,
        5.712101041930239
    ],
    "grouped_nonparametric_test": [
        1.1243147885018122e-08,
        0.3954789652311466,
        7289.0
    ]
}

Both cases return broadly similar significance scores. But when we look directly at the win rate and mean of the raw differences:

win rate: e_case_3a = 0.80 e_case_3b = 0.86
mean diff: e_case_3a = 0.051 e_case_3b = 1.700

e_case_3b has both a higher win rate and a dramatically larger mean improvement, and yet the non-parametric effect sizes for the two cases fail to reflect this gap in the way we would want for decision-making. In a setting where short-term consistency is prized over long-run magnitude, this might be tolerable; but where magnitude of improvement genuinely matters (which, in most portfolio and strategy contexts, it does), the rank-based r understates just how much better e_case_3b really is. This means that the effect size cannot always distinguish between mean Sharpe of 0.05 vs. 1.7. Making decisions based on the effect size alone can be very misleading.

5.1 Introducing Skew to Sharpen the Gap

To make the magnitude gap unambiguous, e_case_3b is modified so that every positive difference receives an additional boost (+|N(0.5, 0.1)|), similarly for the negative difference (-|N(0.1, 0.1)|), deliberately skewing the improvement to be both larger and more asymmetric. This modified distribution is then compared to the e_case_3a.

{
    "paired_parametric_test": [
        1.2389811253725384e-14,
        -0.9058003602974737,
        9.058003602974736
    ],
    "paired_nonparametric_test": [
        2.3480579946009534e-12,
        0.69144712604925,
        4536.0
    ],
    "grouped_parametric_test": [
        0.7303345033830215,
        -0.04881412149518018,
        0.3451679632690591
    ],
    "grouped_nonparametric_test": [
        0.3510859195356728,
        0.027125468563254707,
        5157.0
    ]
}
=========
{
    "paired_parametric_test": [
        2.460218179401368e-15,
        -0.9380282031461347,
        9.380282031461347
    ],
    "paired_nonparametric_test": [
        5.071981353638851e-12,
        0.6804444865497093,
        4504.0
    ],
    "grouped_parametric_test": [
        3.63000393075515e-13,
        -1.1369292753629525,
        8.039304003386514
    ],
    "grouped_nonparametric_test": [
        4.892062477343201e-14,
        0.5264414185492808,
        8047.0
    ]
}

This time, the parametric effect size for e_case_3b correctly comes out higher than for e_case_3a, reflecting the genuinely larger magnitude of improvement, matching what a decision-maker would want. The non-parametric effect size still disagrees, understating the difference because it operates on ranks rather than raw magnitude.

6. Discussion

Pulling the eleven base cases and three extreme cases together, a few patterns hold consistently:

1. Paired designs are mechanically more sensitive than grouped designs, for the simple reason that they remove between-pair variance from the comparison. A result that survives a grouped test will typically also survive its paired counterpart under the same underlying conditions, but not vice versa.
2. Parametric effect sizes track magnitude faithfully, right up until outliers appear. Extreme Case 3 shows parametric r correctly rewarding larger, more consistent improvements. Extreme Case 2 shows the exact same parametric machinery being trivially fooled by a handful of extreme values into reporting effect sizes indistinguishable from a much more modest, but more uniform, improvement.
3. Non-parametric effect sizes are robust to outliers but blind to magnitude, sometimes in counter-intuitive ways. Extreme Case 1’s fixed-improvement scenario shows rank-based r being completely invariant to the size of a uniform shift. Extreme Case 3 shows the same statistic under-weighting a genuinely larger, more decision-relevant improvement relative to a smaller, more consistent one.
4. Neither test is a safe, unsupervised ranking criterion on its own. Cross-feature or cross-strategy comparison using r in isolation, without a companion look at the raw magnitude, the win rate, and the shape of the difference distribution, risks producing rankings that contradict what actually matters for a real decision (e.g., selecting a strategy offering a 0.1 Sharpe improvement over one offering 1.7, simply because the former has a higher rank-based r).

The practical recommendation, in the spirit of the conclusion from the Sharpe ratio significance testing article referenced below [1]: match the test to the property you actually care about (frequency of improvement vs. magnitude of improvement), be suspicious of any single effect size number used in isolation for ranking across distinct comparisons, and always sanity-check with the raw win rate and mean/median of the difference series before trusting r to make the call for you.

Thank you for reading Issue 31 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Ling, L. (2026) Comparing Portfolio Sharpe Ratios, Balaena Quant Insights. Available at: https://medium.com/balaena-quant-insights/comparing-portfolio-sharpe-ratios-f4b316f65738 (Accessed: 20 July 2026).

Effect Size was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dot products never made sense to me. Even more so when most explanations present two equations that are supposedly equivalent:

While Eq. 1 appears to be friendly, its geometric counterpart certainly exists to intimidate. Their relationship is what confused me for a long time: why should multiplying matching coordinates and summing them have anything to do with angles? Why is cosine included in the mix? What is the dot product even used for?

This article is my attempt to reconstruct the dot product from intuition rather than memorisation. Mathematics is the language of abstraction, but precisely for that reason, every concept we dissect should be anchored more firmly in the concrete. Hence, most steps are visualised in such a way that explanations remain tangible and, hopefully, accessible.

Coordinate View

Figure 1 shows two points v1 and v2 on a two-dimensional Cartesian plane [1], where each point has an x-coordinate and a y-coordinate defined in Eq. 3.

To make this less abstract, suppose the x-axis represents a person’s height and the y-axis represents their weight. Each point therefore represents one person using two measurable features. How might we decide whether two people are similar?

The most immediate answer is to measure the distance between their points. If two people have almost the same height and weight, their points will be close together. If their measurements differ greatly, their points will be far apart.

While humans can measure the red line using a ruler, computers cannot do that. We need to compute this distance from our existing data. A right-angled triangle can be formed with the red line acting as the hypotenuse as shown in Figure 2, where the length of the red line is known as the Euclidean distance (Eq. 3) [2].

Since the coordinates x and y are known, we can easily calculate the Euclidean distance between v1 and v2, which should tell us how similar these data points are (and hence how similar these two people are in terms of height and weight).

But distance has another description—one that brings angles into the picture.

Geometric View

The 2D x-y graph in Figure 1 is known as a Cartesian plane with an origin at (0, 0). Until now, we have interpreted each coordinate pair as a point: a location on the plane. However, the same pair of numbers can also describe a movement from the origin (see Figure 3). A coordinate of (x1, y1) means to move x1 units in the x-axis direction, followed by y1 units in the y-axis direction. The black point is the final destination of this described movement.

This same coordinate pair can therefore be represented by an arrow, known as a vector, whose length measures the distance from the origin to its endpoint. A point describes a location on the plane. Once we draw an arrow from the origin to that point, the same coordinates can also be interpreted as a vector: a movement with both magnitude and direction.

The Euclidean distance formula can also be applied to find the length of the vectors (Eq. 5).

Representing the two points as vectors yields a new graphical picture. Together with the red distance line between their endpoints, the two vectors form a triangle in Figure 4:

The red line has not changed: it is still the distance between the two original points. What has changed is our perspective. Previously, we calculated this distance using coordinate differences. Now, by treating the two points as vectors from the origin, the same red line becomes the third side of a triangle whose other two sides have lengths ||v_1|| and ||v_2||. This allows us to describe its length again, this time using the law of cosines and the angle theta between the vectors [3].

The same red line now has two valid mathematical descriptions: one in coordinates, and one in lengths and angles (see Eq. 6). Since they describe the same distance, they must be equal. The dot product emerges from the term that makes these two perspectives agree.

These equations are starting to resemble Eq. 1 and Eq. 2. Let us expand Eq. 4:

Recall Eq. 5, where the vector length is just the sum of its squared coordinates. Substitute Eq. 5 into Eq. 4 gives us:

Notice that the new Eq. 4 has exactly the same structure as Eq. 6, except for the last term. Since Eq. 4 and Eq. 6 describe the same red line, their corresponding final terms must also be equal. The shared quantity is what we call the dot product, leading us squarely back to Eq. 1 and Eq. 2.

The apparent duality of the dot product is therefore not an arbitrary rule to memorise. Both forms emerge from calculating the same red line in two different ways: once through coordinate differences, and once through vector lengths and the angle between them.

The dot product is precisely the quantity that makes the coordinate view and the geometric view agree.

What Does the Dot Product Tell Us?

Alignment

The dot product does more than connect two equations; it tells us how strongly two vectors align, taking into account both their directions and their lengths. Recall Eq. 2:

Because vector lengths are always non-negative, the sign of the dot product depends on cos theta, and therefore on the angle between the vectors. To measure whether two vectors are pointing in the same direction, the angle comes into play (see Figure 5).

When two vectors point generally in the same direction, the angle between them is less than 90 degrees. The cosine of such an angle is positive, and so the dot product is positive. When two vectors are perpendicular, the angle is exactly 90 degrees, whose cosine is zero; their dot product is therefore zero. When two vectors point generally against each other, the angle is greater than 90 degrees, whose cosine is negative; their dot product is negative.

In this case, the cosine term simply converts the angle between two vectors into an alignment score between [-1, 1]. The dot product is then the alignment between vectors scaled by the product of their lengths. If we want direction alone, independent of length, we divide by both vector norms and obtain cosine similarity.

Returning to our height-and-weight example, Euclidean distance and the dot product answer two related but different questions. The Euclidean distance asks whether two people have similar actual measurements: a person who is 160 cm tall and weighs 50 kg is closer to someone who is 162 cm tall and weighs 52 kg than to someone who is 190 cm tall and weighs 85 kg.

The angle between their vectors asks something different. It asks whether their height and weight increase in a similar proportion from the origin. A taller and heavier person may be far away in absolute distance, yet still lie in almost the same direction as a shorter and lighter person. In that case, their vectors have a small angle between them: they differ in magnitude, but share a similar height–weight profile.

In Figure 6, while v1 and v3 are far apart in terms of the Euclidean distance, both vectors point in the exact same direction with an angle of 0 degrees between them.

This is why direction matters. Two vectors pointing in the same direction reinforce one another; two vectors pointing in opposite directions conflict; and two vectors at right angles share no directional alignment. The dot product gives us a numerical way to measure this alignment.

Note: Strictly speaking, raw height and weight are only an illustrative example: their units differ, and a zero-height, zero-weight origin is not a meaningful human baseline. In real data analysis, we would usually standardise the features or measure them relative to an average person before interpreting vector angles. This example is nevertheless useful because it lets us see the conceptual difference between distance and direction.

Yet alignment is only one interpretation of the dot product. Once a particular direction matters to us, the dot product can also measure how much of another vector lies along that direction. This leads us to projection.

Projection

While alignment tells us whether two vectors point in compatible directions, projection tells us how much of one vector actually lies along another vector.

Suppose we chose v1 as a direction that we care about, and place v2 at an angle theta from it. Figure 7 shows that we can form a right-angled triangle by drawing a perpendicular line from the tip of v2 onto v1.

The green segment is the adjacent side of the triangle and, geometrically, the projection length of v2 onto v1 (which is like the “shadow” of v2 along v1). Since cos theta can be calculated from the adjacent and hypotenuse, we can rearrange the formula as such:

Substituting this formula into Eq. 2:

The dot product is therefore the length of projection of v2 along the direction of v1 scaled by the length of v1.

Suppose v1 represents a direction we care about, such as the direction of a market factor, and v2 represents an asset’s movement. The dot product tells us how much of the asset’s movement lies along that factor direction, up to the scale of the factor. If the dot product is strongly positive, the asset has a large component along the factor direction; if negative, it has a component against that direction; if near zero, little of its movement lies along that factor direction.

Thus, the dot product is not merely a score of whether two arrows face similarly. It is also a tool for extracting the component of one vector that matters in a chosen direction.

Conclusion

The dot product first appeared strange to me because its two forms seemed to speak entirely different languages: one multiplies coordinates, while the other measures lengths and angles.

However, once both are rooted in the same concrete picture—the distance between the endpoints of two vectors—their relationship becomes much less mysterious. The algebraic and geometric expressions are not separate rules that happen to agree. They are two descriptions of the same triangle, and the dot product is the shared quantity that connects them.

From this relationship, its usefulness also becomes clearer. A positive, zero or negative dot product tells us whether two vectors broadly move together, have no directional alignment, or move against each other. When normalised, it becomes cosine similarity, allowing us to compare direction without being distracted by magnitude. When viewed as a scaled projection, it tells us how much of one vector lies along a direction that matters to us.

This is especially relevant in quantitative research, where vectors may represent assets, signals, features or factor exposures. We are often interested not only in how far two observations are from each other, but also in whether they share a direction, or how much of one movement can be attributed to another. The dot product provides the mathematical foundation for asking these questions.

More importantly, reconstructing the dot product in this way has changed how I see mathematical formulas. An equation becomes much easier to trust when I can trace it back to a visible object, a measurable distance or a geometric relationship.

Mathematics may eventually speak in abstractions, but understanding begins when those abstractions are made concrete.

Thank you for reading Issue 30 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Cartesian coordinate system (2026) Wikipedia. Available at: https://en.wikipedia.org/wiki/Cartesian_coordinate_system (Accessed: 22 May 2026).
2. Euclidean distance (2026) Wikipedia. Available at: https://en.wikipedia.org/wiki/Euclidean_distance (Accessed: 22 May 2026).
3. Law of cosines (2026) Wikipedia. Available at: https://en.wikipedia.org/wiki/Law_of_cosines (Accessed: 22 May 2026).

Understanding the Dot Product was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Sharpe ratio (SR) is the dominant metric for evaluating portfolio performance, yet a single observed SR tells us little without knowing how reliable that estimate is. This article argues that comparing SRs requires formal significance testing, and that the correct test depends critically on the experimental design.

1. The Need for Significance Testing

The annualised Sharpe ratio (SR) is one of the most ubiquitous metrics used to evaluate portfolio performance in quantitative trading. It is typically computed with the formula sr = returns.mean() / returns.std() * np.sqrt(num_periods_per_year)on portfolio returns, where the numerator represents reward and the denominator measures risk. Suppose that there are four portfolios with 1000 strategies each, of which we wish to compare Portfolio A against B, and C against D.

# Case 1
Portfolio A: runs on 2020–2023
Portfolio B: same strategies as Portfolio A, runs on 2024–2026

# Case 2
Portfolio C: uses equal weights allocation, 2020–2026
Portfolio D: same strategies as Portfolio C, uses improved weight allocation, 2020–2026

For Case 1, we would be interested in whether the performances of Portfolio A and B remain stable across different market periods. If SR_a != SR_b, it could, for example, signify a regime change, alpha decay or simply a different volatility environment in our selected strategies. In contrast, Case 2 investigates the effect of pipeline changes by comparing Portfolio D against C. SR_d > SR_c would suggest that the new pipeline has indeed brought improvements; SR_d ≤ SR_c would instead hint that such changes are negligible or detrimental to performance. (Note that these two cases are associated with very different research questions; more in Section 2 later.) The obvious approach to assessing these portfolios is to calculate SRs for each of them, e.g., SR_a = 3.0, SR_b = 3.0, SR_c = 2.0, SR_d = 2.5, then make our conclusions. At first glance, Case 1 looks stable, and Case 2 looks improved.

But this raises the real research question: did we observe something robust, or did we just get lucky?

To understand the concept of randomness and luck, consider the infinite monkey theorem [1].

A monkey hitting keys at random, given enough time, will eventually type out the complete works of Shakespeare — purely by chance (luck).

But does that mean the monkey is capable of writing literature (skill)?

Imagine we live in a world that belongs to infinite parallel universes. Yesterday’s price dropped in our world but rose in another universe XYZ; today’s price rose in our world but dropped in universe XYZ. The path of price returns would eventually look very different in these two worlds; perhaps our particular set of returns data for Portfolio A and B so happen to yield SR_a == SR_b out of pure chance, or that the selected strategies for Portfolio C and D happen to perform better in Portfolio D than in C. If the four portfolios were deployed to production in a hundred more parallel universes, would this situation SR_a == SR_b and SR_d > SR_c occur in the majority of the timelines? For instance, we would be more confident in our results should both cases consistently happen 95/100 of the time vs. only 60/100 of the time.

A Sharpe ratio is estimated from one realised return path. If the market had taken a slightly different path, the same portfolio could have produced a different Sharpe ratio. A number on its own tells us nothing. A difference of 0.5 could be enormously meaningful or pure noise, depending on the length of the return series, the autocorrelation structure, and the distribution of returns. We need to know the distribution of the test statistic under the null hypothesis to say anything about significance.

2. Group and Paired Significance Testing

Significance testing gives us a disciplined way to ask how often results like ours might appear under plausible alternative histories. But before we can test anything, we need to be precise about what we are testing, because the two cases above are fundamentally different research questions, and conflating them leads to the wrong test.

2.1 Group Testing: Are These Two Populations Different?

In Case 1, Portfolio A and Portfolio B contain the exact same strategies but run over non-overlapping time windows. They share no common observations. The two SRs are computed from completely independent return histories.

This is the setting for a group hypothesis test. We treat the two portfolios as draws from potentially different distributions and ask: could these two SRs have plausibly come from the same underlying distribution? Or, more precisely: under a null hypothesis that SR_a = SR_b, how surprising is the observed difference? For example, think of it like measuring the heights of randomly sampled groups from two cities. It is unknown which individual came from which city, and the two groups have no inherent pairing. We are asking whether the populations differ (see Figure 2.1.1).

If the same set of strategies is run over 1000 different historical periods and all resulting SRs are plotted, two SR distributions would be formed, and their difference in mean is calculated. The question is whether SR_a and SR_b look like they were drawn from the same part of that distribution, or whether one came from a meaningfully different regime. Note that different group tests compare different aspects of the distributions. For example, a group t-test compares only the means of the two distributions, while a group KS test compares the entire empirical distributions.

2.2 Paired Testing: Did Our Change Actually Help?

Case 2 is structurally different. Portfolio C and Portfolio D run the same strategies over the same time period, differing only in weight allocation. Every day, both portfolios are exposed to exactly the same market environment. Their returns are not independent; they are correlated by construction, since they share underlying strategy signals.

This is the setting for a paired test (see Figure 2.2.1). Rather than comparing two independent samples, we look at the difference in SRs between D and C, where the null hypothesis is SR_d — SR_c = 0. We then ask: is the mean of this difference series significantly different from zero? The paired approach is strictly more powerful here. By subtracting out the common market noise shared between C and D, we isolate the incremental effect of the weight change. If a group test were incorrectly used, the portfolios would be treated as independent, effectively throwing away the information that they were exposed to the same market realisations, inflating variance and reducing power.

Imagine a new drug is being tested, and the old drug is given to one group and the new drug to another. That is a group test. Now instead, every patient is given both drugs at different times and the difference per patient is measured. The within-person comparison cancels out individual variation. That is a paired test, and it has higher statistical power than a group test when inputs are sufficiently correlated.

With 1000 trials for this case, the distribution of the differences in Sharpe ratio is formed, which is then compared against a null distribution. If the target distribution is meaningfully different from the null distribution, the paired portfolios are shown to have a significant difference in SR performance. Similarly, different paired tests compare different aspects of the distributions: a paired t-test looks at the difference between means, where the mean of the null distribution would typically be 0; on the other hand, a paired u-test compares the medians of both distributions.

Note: paired test results might be less reliable when the correlation between inputs is low [2]. This intuition rests on the assumption that correlation is a partial measure of dependency between samples.

3. Putting Everything Together

3.1 Assumptions of Distributions

While the Gaussian distribution is used for illustrative purposes in Figure 2.1.1 and Figure 2.2.1, the empirical distribution of our data is an important consideration that we must take into account for our use cases. In a group test, we are concerned with the distribution of Sharpe ratios; in a paired test, we are concerned with the distribution of differences of Sharpe ratios. The characteristics of these distributions are central to choosing an appropriate significance test, such as a t-test, u-test, or sign test. Note that each test also has additional assumptions that require assessment, which will not be explored in this article.

If returns were normally distributed, life would be easy: the Sharpe ratio estimator would simply approach normality under the Central Limit Theorem. However, it is already well known that returns are non-normal; they exhibit fat tails, negative skewness, and autocorrelation. The empirical (observed) sampling distribution of the Sharpe ratio depends on the skewness and excess kurtosis of the underlying returns, as characterised in [3]. The distribution of estimated Sharpe ratios across bootstrap samples of real strategy returns tends to be left-skewed and fat-tailed, not the clean symmetric bell curve the CLT would suggest. A t-test built on normal assumptions would systematically misstate p-values.

The impact of non-normal returns ripples through to the distribution of differences of Sharpe ratios. d_i = SR_di — SR_ci are not automatically normal even if both groups’ SR distributions look roughly bell-shaped. d may be approximately normal when SR_c and SR_d are jointly normal; but since SR distributions are unlikely to be nicely Gaussian, it is unsafe to assume normality for the distribution of SR differences. We also present some empirical observations of the distributions of SR differences in Figure 3.1.1, where a Shapiro-Wilk test is conducted. A p-value ≤ 0.05 indicates non-normality. Figure 3.1.1 largely supports the fact that normality and symmetry are not guaranteed for distributions of SR differences.

3.2.1 Paired Tests

In the paired setting, we work with the series of per-trial SR differences, d_i = SR_di − SR_ci, and ask whether this difference series is centred at zero.

The Sign Test (most assumption-free)

The simplest robust approach is to count: across all trials, how many d_i are positive? Under the null that the two pipelines are equivalent, positive differences should occur with probability 0.5. We test this with a binomial test or its normal approximation. The sign test requires no assumptions about the shape of the distribution, only that observations are independent. It is robust but relatively low-power, since it discards all magnitude information.

The Wilcoxon Signed-Rank Test (moderate power)

An improvement over the sign test, the Wilcoxon Signed-Rank (WSR) test uses the rank order of the absolute differences, recovering some magnitude information without assuming normality. It is the non-parametric alternative to the paired t-test. However, it does assume the distribution of differences is symmetric around the median. For return differences that are skewed, which is common when one portfolio has occasional large drawdown avoidances, this assumption can also be violated, limiting its applicability. (Asymmetry is also exhibited by the distributions in Figure 3.1.1.)

The Ledoit-Wolf Test (accounts for non-normality and varying time frames)

Ledoit and Wolf (2008) propose a bootstrap-based test for the equality of Sharpe ratios that explicitly handles non-normal returns, serial autocorrelation, and heteroskedasticity [4]. Unlike tests that only look at the empirical distribution of Sharpe ratios or SR differences (e.g. t-test, u-test), the Ledoit–Wolf test is designed for time-series return data, so it can additionally account for path dependence by using block bootstrap resampling.

Note: we can distinguish between two robustness-testing approaches. First, we can run N trials with different strategy subsets in the portfolio. This varies the portfolio constituents and measures input-level sensitivity, i.e. whether the method is robust to changes in portfolio composition. Second, we can keep the strategy set fixed but shuffle blocks of returns. This preserves the same inputs while generating alternative equity-curve paths, allowing us to measure path-level sensitivity, i.e. whether the method is robust to different realised sequences of returns. Both of these can be combined for a holistic assessment.

The procedure:

1. Compute the observed SR difference: Δ = SR₁ − SR₂
2. Resample the return series using a block bootstrap, which preserves the autocorrelation structure of returns by resampling contiguous blocks rather than individual observations
3. For each bootstrap sample, recompute the SR difference to build an empirical null distribution of Δ*
4. The p-value is the fraction of bootstrap samples where |Δ*| ≥ |Δ|

Critically, the block bootstrap does not assume any parametric form for the return distribution, nor does it assume that the SR estimator is normally distributed. It lets the data speak for itself. The Ledoit-Wolf test has been shown to have correct size (i.e., the actual false positive rate matches the nominal level) under realistic financial return distributions, unlike the t-test.

One practical consideration is block length selection. Too short and you fail to capture autocorrelation; too long and you lose resampling variability. Common heuristics suggest block lengths of order T^(1/3) where T is the sample length, though data-driven selection methods exist.

3.2.2 Group Tests

In the group setting, we compare two independent SR distributions, such as SR_a and SR_b from non-overlapping time windows, and ask whether they come from the same underlying distribution.

The Mann-Whitney U Test (assumption-free on distribution shape)

The natural non-parametric counterpart to the group t-test, the Mann-Whitney U test asks whether SR values from one group tend to be systematically larger than those from the other. It makes no assumption about the shape of either distribution, only that observations within and across groups are independent. It is equivalent to testing whether the probability that a randomly drawn SR from group A exceeds a randomly drawn SR from group B is 0.5 under the null. It is more powerful than a group sign test and more assumption-free than the t-test, making it a sensible default for group comparisons of non-normal SR distributions. It works similarly to the WSR test but is used for group tests.

4. Conclusion

Comparing Sharpe ratios is deceptively simple on the surface: subtract one number from another and check the sign. But as this article has shown, the question of whether an observed SR difference reflects genuine signal or random variation demands considerably more rigour.

The first layer of rigour is experimental design. Case 1 and Case 2 are not just different examples; they represent fundamentally different statistical problems. Using a group test when portfolios share a common return environment, or a paired test when return series are independent, leads to either inflated variance or spurious precision.

The second layer is distributional honesty. Financial returns are fat-tailed, skewed, and autocorrelated. These properties propagate into the sampling distribution of the Sharpe ratio and into the distribution of SR differences, rendering normal-assumption tests unreliable in ways that are not always obvious from looking at the data. The Shapiro-Wilk results in Figure 3.1.1 underscore that normality and symmetry cannot simply be assumed; they must be verified, and in practice they frequently fail.

The third layer is choosing a test that is honest about what it knows. The sign test sacrifices power for robustness. The Wilcoxon Signed-Rank test recovers power but requires symmetry. The Ledoit-Wolf block bootstrap makes the fewest assumptions of the three, explicitly accounting for autocorrelation and heteroskedasticity, and remains well-calibrated across realistic return distributions. For group comparisons, the Mann-Whitney U test provides a similarly assumption-light baseline.

Taken together, the practical recommendation is straightforward: always match the test to the experimental structure, never assume normality without testing it, and when in doubt, default to the more conservative test. A Sharpe ratio observed in one history is just that—one history. Significance testing is how we ask whether it would survive the others.

Thank you for reading Issue 29 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Infinite monkey theorem (2025) Wikipedia. Available at: https://en.wikipedia.org/wiki/Infinite_monkey_theorem (Accessed: 19 July 2025).
2. Hokanson, J. (2020) The (Statistical) Power of Pairs, Jim Hokanson. Available at: https://jimhokanson.com/blog/2020/2020_04_Power_Of_Pairs/ (Accessed: 06 May 2026).
3. Benhamou, E. (2021) ‘Distribution and statistics of the Sharpe Ratio’, HAL Open Science [Preprint].
4. Ledoit, O. and Wolf, M. (2008) ‘Robust performance hypothesis testing with the Sharpe ratio’, Journal of Empirical Finance, 15(5), pp. 850–859. doi:10.1016/j.jempfin.2008.03.002.

Comparing Portfolio Sharpe Ratios was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article investigates different portfolio weight allocation methods for a single-asset, multi-strategy portfolio: examining how each method performs, how sensitive they are to input quality, and what the results reveal about building a robust, scalable portfolio.

1. Introduction

1.1 More Strategies, Less Impact

There is a pattern that most systematic traders will eventually encounter: adding new strategies to a portfolio stops making a meaningful difference.

This is not a bug — it is a mathematical inevitability. As the number of strategies grows, the weight assigned to each individual strategy shrinks proportionally. A strategy that once contributed 5% of total exposure may now contribute 0.04%. Its alpha, however strong, is effectively diluted to near zero. If the portfolio grows from 100 to 2600 strategies without a mechanism to turn off underperformers, new additions do little beyond adding noise.

The right response to this problem is not to stop adding strategies. The right response is to reconsider how weights are allocated. A good allocation method should not merely distribute capital — it should filter, suppress, and concentrate where it matters.

1.2 Portfolio Optimisation Methods

This study evaluates five weight allocation methods on a live portfolio of over 2600 CTA strategies trading primarily on Bitcoin. The methods are: equal weighting (our benchmark), market optimiser (buy-and-hold on a single BTC asset), inverse volatility, hierarchical risk parity (HRP), and maximum diversification (MD). Mean-variance and mean-semivariance methods were also candidates but were eliminated during pilot testing due to persistent overfitting — a finding consistent with the broader finance literature, where estimating expected returns reliably has long been regarded as a difficult task [2, 3, 5]. These models are mostly already implemented in the package skfolio [6].

The naive equal-weighted (EW) implementation uses the EqualWeight optimiser from skfolio [6] with no additional parameters, as other parameter choices do not affect the outcome. It allocates weights evenly across all strategies in the portfolio and serves as a fixed, static benchmark.

For InverseVolatility (INV) [6], each asset weight is computed as the inverse of its volatility and rescaled so that weights sum to one. Volatility is defined using the default parameter for this method — a covariance matrix consisting of the variance of each asset's return distribution.

For HierarchicalRiskParity (HRP), the implementation follows the description by Marcos Lopez de Prado [4]. In brief, it uses a distance matrix (correlation by default) to compute hierarchical clusters, then uses a risk matrix (volatility/variance by default) to assign weights from each top-level cluster down to its children. The distance and risk matrices are left at their defaults, minimum and maximum weights are left at 0.0 and 1.0 respectively, and transaction fees are ignored as they are outside the scope of this study.

The maximum diversification (MD) method uses skfolio’s implementation [6], which optimises each asset’s weight to maximise the portfolio’s diversification ratio—a metric that accounts for the market’s total volatility relative to the weighted volatility of individual strategies. All parameters are left at their defaults, as parameter optimisation is outside the scope of this study. For reference, the parameters that would affect the outcome include the L1 and L2 regularisation coefficients (used to prevent overfitting and penalise zero-weight strategies), the solver engine, transaction fees, minimum and maximum weights, minimum and maximum budgets, and long/short weight thresholds.

2. Measuring Portfolio Quality

2.1 Defining Metrics

Before comparing methods, we need a stable definition of portfolio quality. Seven metrics were chosen to span three dimensions:

1. Sharpe ratio (higher is better)
2. Sortino ratio (higher is better)
3. Average drawdown (lower is better)
4. Max drawdown (lower is better)
5. Standard deviation (lower is better)
6. Diversification ratio [5] (higher is better)
7. Correlation with market drawdowns (lower is better)
8. Correlation with market returns (lower is better)

Volatility risk is captured by the standard deviation of returns and the diversification ratio. The diversification ratio is the ratio of the weighted average volatility of individual strategies to the total portfolio volatility [5]. A high diversification ratio signals that the portfolio is genuinely diversified, not simply holding many strategies that all move together.

Portfolio performance is captured by the Sharpe ratio, Sortino ratio, average drawdown, and max drawdown. The Sharpe ratio measures return per unit of total volatility; the Sortino ratio focuses only on downside volatility. Together, they distinguish between strategies that generate returns and those that generate risk-adjusted returns.

Market relationship is captured by the pairwise correlation of strategy returns with market returns, and the pairwise correlation of strategy drawdowns with market drawdowns. Low correlation indicates diversification benefits and resilience during market declines, while high correlation suggests greater market exposure and the potential for amplified losses, both of which are important for guiding risk allocation and hedging decisions.

2.2 Validating Metrics

To confirm that these metrics behave sensibly, we ran a sanity check using S&P 500 assets. Historical prices from the S&P 500 are loaded as individual assets (see Figure 2.2.1); an additional market asset is generated by assigning equal weights to all other assets, so that it tracks general market movement.

Four optimisers are constructed to validate different metrics:

• VolatileOptimizer. Assigns the most weight to a highly volatile asset. The portfolio should exhibit higher standard deviation and large drawdowns, producing moderate Sharpe and Sortino ratios.
• MarketOptimizer. Performs buy-and-hold on the market asset. It serves as the market benchmark for reference.
• BetaOptimizer. Puts the majority of weight on the market asset. We expect the portfolio to show strong correlations with both market returns and market drawdowns.
• MaxDiverse. Allocates weights to maximise the diversification ratio. The resulting portfolio should therefore achieve the highest diversification ratio of the four.

Portfolio compositions and equity curves are shown in Figures 2.2.2 and 2.2.3, respectively.

As shown in Figure 2.2.4, the metrics correctly ranked each portfolio according to its design intent. The volatile portfolio showed the highest standard deviation and largest drawdowns. The market-tracking portfolio showed the highest market correlations. The maximum diversification portfolio achieved the best Sharpe and Sortino ratios alongside the lowest market correlations. The metrics behave as intended.

Rankings across all methods use a Borda count system [7]: within each fold or scenario, each method is ranked per metric (1 = best, where the direction of “best” depends on the metric), and the ranks are summed. A lower total rank sum indicates better overall performance.

3. Methodology

The study runs two parallel experiments to answer different questions.

3.1 Experiment 1: Input Sensitivity

How does each method respond when the quality of its inputs degrades? Portfolio subsets of 100 strategies are constructed with controlled correlation ratios (ranging from 0.0 to 0.8 of correlated-to-uncorrelated strategies), then evaluated against the metrics from Section 2.

Given a set of in-house portfolio strategies, the input is first filtered via a uniform chance filter with a probability of 0.1 to reduce running time while maintaining a representative strategy distribution. Subsets are generated according to correlation ratios of [0.0, 0.2, 0.4, 0.6, 0.8], which dictate the ratio of correlated to uncorrelated strategies in a portfolio of 100 strategies. Pairwise Pearson correlation is computed for both strategy returns and strategy drawdowns, with a correlation threshold set at 0.5. This means that a correlation ratio of 0.8 indicates that 80 out of 100 strategies in the subset are correlated at ≥ 0.5 with each other. Poor input quality corresponds to ratios of 0% or 20%.

Gaussian blur is applied before computing correlation scores to suppress high-frequency noise without introducing look-ahead bias. A simple moving average is not used because it introduces lags and produces null values at the start of the time series.

After portfolio subsets are generated for each ratio, the allocation methods from Section 1 are evaluated against the metrics from Section 2. Experiment 1 investigates two aspects of metric sensitivity across different subsets:

• Relative value comparison. Performance metrics are regressed against the correlation ratio to produce sensitivity betas. A beta near 1.0 means the metric degrades proportionally with input quality; a beta near 0 means the method is robust to noise.
• Raw value comparison. This prevents the scenario where a method appears relatively sensitive to portfolio composition, but its raw values still outperform the alternatives.

3.2 Experiment 2: Cross-Validation

How does each method perform across time on the real portfolio? Cross-validation (CV) folds span 2020 to 2026, with the strategy pool growing from 7 to approximately 2,600 strategies over this window (see Figure 3.2.1), providing an informal stress test of method stability under structural portfolio changes.

A 6–3–1 split (six months train, three months validation, one month test) are evaluated as follows:

• Evaluate the Section 2 metrics for each CV fold.
• Within each fold, rank each allocation method per metric. Higher-is-better metrics (Sharpe ratio, Sortino ratio, diversification ratio) are ranked with the highest value first; all remaining metrics are lower-is-better and ranked with the lowest value first.
• Sum the per-metric rankings for each allocation method within the fold to produce a rank sum. The method with the lowest rank sum performs best in that fold.
• Repeat Steps 1–3 for each CV fold.
• Sum rank sums across all folds for each method. The method with the lowest final rank sum performs best overall.

To better understand the behaviour of each allocation method over time, we also construct a continuous performance record by concatenating the test data across all folds and computing the Section 2 metrics on the resulting time series (across time metrics).

A sanity check is also performed to confirm that strategy weights sum to approximately 1 for each fold and optimiser.

In addition to performance, we examine the weight distribution of each allocation method to check for undue concentration in a small number of strategies, which reduces diversification and may increase risk instead [1]. The benchmark assigns each strategy a weight of approximately 1/2500 ≈ 0.0004. Using 0.0005 as a threshold, we report: the number of strategies with weight ≥ 0.0005, the number with weight < 0.0005, and the number assigned a weight of exactly 0.

4. Results

4.1 Experiment 1

"EqualWeighted": {
            "Standard Deviation": 0.2887,
            "Average Drawdown": 0.9629,
            "MAX Drawdown": 0.9254,
            "Correlation with Market Returns": 0.9137,
            "Correlation with Market Drawdown": 0.4175,
            "Sortino Ratio": 0.9733,
            "Sharpe Ratio": 0.9685,
            "Diversification Ratio": 0.4114
        },
"InverseVolatility": {
            "Standard Deviation": 0.2236,
            "Average Drawdown": 0.9613,
            "MAX Drawdown": 0.905,
            "Correlation with Market Returns": 0.8552,
            "Correlation with Market Drawdown": 0.6138,
            "Sortino Ratio": 0.982,
            "Sharpe Ratio": 0.9856,
            "Diversification Ratio": 0.5898
        }
 "HierarchicalRiskParity": {
            "Standard Deviation": 0.8019,
            "Average Drawdown": 0.9744,
            "MAX Drawdown": 0.9301,
            "Correlation with Market Returns": 0.7638,
            "Correlation with Market Drawdown": 0.6758,
            "Sortino Ratio": 0.9516,
            "Sharpe Ratio": 0.9553,
            "Diversification Ratio": 0.2942
        }
 "MaximumDiversification": {
            "Standard Deviation": 0.9037,
            "Average Drawdown": 0.9579,
            "MAX Drawdown": 0.929,
            "Correlation with Market Returns": 0.9456,
            "Correlation with Market Drawdown": 0.9731,
            "Sortino Ratio": 0.9889,
            "Sharpe Ratio": 0.9912,
            "Diversification Ratio": 0.7628
        },

Figure 4.1.1 shows the results for Experiment 1. For EW, the correlation-with-market-returns beta is relatively high (≈0.91), meaning that as input quality worsens the portfolio’s correlation with the market declines, indicating a tendency to converge toward non-market behaviour. The standard deviation beta is low (≈0.29), suggesting that input quality has little effect on overall portfolio volatility, though this depends on which strategies the weights ultimately concentrate in. EW achieves a moderate diversification ratio beta (≈0.41), indicating that input quality does not strongly affect the diversification ratio given how the metric is constructed. Its high Sharpe (≈0.97) and Sortino (≈0.97) betas suggest that poor input quality does meaningfully degrade risk-adjusted performance.

INV shows a moderately high correlation-with-market-returns beta (≈0.86), meaning that as input quality worsens the portfolio continues to track the market reasonably well, though somewhat less tightly than EW, implying a modest drift toward market-like behaviour. Its standard deviation beta is low (≈0.22), suggesting limited sensitivity of portfolio volatility to input quality. The diversification ratio beta of ≈0.59 indicates that deteriorating input quality moderately reduces effective diversification. Its high Sharpe (≈0.99) and Sortino (≈0.98) betas indicate that poor input quality does meaningfully degrade risk-adjusted performance.

HRP shows a moderate correlation-with-market-returns beta (≈0.76) and a relatively high correlation-with-market-drawdown beta (≈0.68), suggesting that its behaviour becomes less sensitive to overall market moves as input quality worsens. Its standard deviation beta is fairly high (≈0.80), meaning that lower-quality inputs can materially increase volatility. The diversification ratio beta is low (≈0.29), indicating that input quality has limited impact on the diversification benefit. The Sharpe (≈0.96) and Sortino (≈0.95) betas are close to 1, implying that deteriorating input quality noticeably impairs risk-adjusted performance, consistent with the other methods.

MD shows a very high correlation-with-market-returns beta (≈0.95), meaning that as input quality worsens the portfolio remains tightly linked to the market and increasingly resembles a market-beta strategy rather than a differentiated one. Its standard deviation beta is also high (≈0.90), indicating that poor-quality inputs materially increase overall volatility and risk. The high correlation-with-market-drawdown beta (≈0.97) further indicates that under bad inputs, the strategy’s drawdowns move almost in lockstep with the market, offering limited downside protection. This is notable given MD’s high diversification ratio (≈0.76), which reflects structural diversification in the optimal case but does not confer robustness when inputs are noisy or degraded.

4.2 Experiment 2

Final Rank Sum (Sorted by Ascending Order)

MaximumDiversification     948.0 (best)
HierarchicalRiskParity    1117.0
InverseVolatility         1409.0
Benchmark                 1731.0
MarketOptimizer           1875.0 (worst)
dtype: float64

MD and HRP are the two best-performing methods. Figures 4.2.1–4.2.3 show the across-time metrics, the number of strategies assigned zero weight, and the one-month test equity curves. Although the benchmark achieves the highest cumulative returns, MD and HRP experience shallower drawdowns throughout the testing period, a pattern reflected in their annualised Sharpe and Sortino ratios, and confirmed by their substantially lower average and maximum drawdowns.

The benchmark is unaffected by changes in portfolio composition, as shown in Figure 4.2.4. It achieves an annualised Sharpe ratio of 4.44, an annualised Sortino ratio of 7.10, an average drawdown of 0.14%, a maximum drawdown of 14%, and a diversification ratio of 2.56. The correlation between its drawdowns and market drawdowns is 0.07 (even after Gaussian smoothing), effectively uncorrelated. Its equity curve achieves the highest average returns of all methods tested.

INV performs slightly better than the benchmark on Sharpe ratio (4.82) and Sortino ratio (7.51). Its average drawdown is 0.9% and its maximum drawdown caps at 10%. However, the INV equity curve is largely a scaled-down version of the benchmark (see Figure 4.2.3), suggesting that the two methods do not differ meaningfully in behaviour. This is corroborated by their near-identical correlations with market returns and market drawdowns. On average, approximately 500 strategies are assigned a weight above 0.0005 (see Figure 4.2.5).

Portfolio composition is also visually stable across folds, consistent with the benchmark’s behaviour (see Figure 4.2.4). The likely reason is that INV never assigns zero weights—and because it lacks a mechanism to deactivate strategies, it cannot sufficiently concentrate the portfolio to capture concentrated alpha. This is probably the primary reason INV fails to rank as highly as MD or HRP.

HRP performs similarly to the benchmark on Sharpe ratio (4.38) but achieves a modestly better Sortino ratio (7.51), indicating somewhat better drawdown characteristics. Approximately 250 strategies are assigned a weight above 0.0005 on average, and the diversification ratio is higher by approximately 0.7. However, its correlation with market drawdowns increases relative to the benchmark, likely due to its tendency to concentrate on a small number of high-performing strategies. While HRP selects 2,300+ strategies on average, it dropped as low as 44 active strategies in one fold (see Figure 4.2.2). This concentration is the primary concern with HRP: focusing too heavily on a small number of strategies reduces diversification and may increase risk. Parameter tuning—particularly for min_weight and max_weight—would likely mitigate this.

Because HRP can assign zero weights, its portfolio composition changes visibly across folds (see Figure 4.2.4), and the strategy receiving the largest weight shifts over time, indicating that HRP dynamically responds to changing market conditions and strategy performance.

MD delivers performance broadly similar to HRP but stronger in almost every dimension. It significantly outperforms the benchmark on Sharpe ratio (by ≈1.5), Sortino ratio (by ≈3), and maximum drawdown (6% vs. 14% — more than halved). Its correlation with market drawdowns increases slightly to 0.11, while its correlation with market returns decreases to 0.11. The diversification ratio nearly doubles to 4.56. On average, approximately 180 strategies are assigned a weight above 0.0005 (see Figure 4.2.5).

Like HRP, MD concentrates on a subset of strategies within each fold; however, it is more consistent in the size of that subset, averaging approximately 180 strategies. Its lowest active strategy count across all folds is 140, and the standard deviation of its selected strategy count is lower than HRP’s (446 vs. 563; see Figure 4.2.2). The same advice on parameter tuning applies here.

MD also shows visually distinct portfolio compositions across folds (see Figure 4.2.4). Notably, MD distributes its weight more evenly than HRP: weight allocations are not as concentrated in single large chunks. Both methods assign high weights to the same strategies, suggesting broad agreement on which strategies are the strongest performers.

7. Conclusion: The Turn-Off Mechanism is the Key Variable

The hierarchy that emerges from this study is MD > HRP > INV ≈ EW > market buy-and-hold, and the primary explanatory variable is not the sophistication of the optimiser—it is the presence and consistency of a zero-weight mechanism.

EW and INV cannot deactivate strategies. As the strategy pool grows, dilution is inevitable, and volatility weighting alone cannot overcome it. HRP and MD can assign zero weights, and both outperform significantly as a result. But MD’s more consistent selection range (≈180 strategies per fold vs. HRP’s range of 44–2600) may be the decisive advantage: it filters aggressively enough to capture concentrated alpha, but not so aggressively that it becomes a single-strategy bet.

The input sensitivity results add a complementary insight. EW and INV are robust floors; they degrade gracefully under poor inputs. HRP and MD are sensitive ceilings; they deliver superior performance when inputs are reasonable, but deteriorate faster when they are not. For practitioners, this suggests a conditional allocation framework: use MD or HRP as primary methods during stable regimes, with fallback logic toward EW during periods of unusual strategy correlation or degraded signal quality.

Practical directions for future work include parameter optimisation of minimum and maximum weights—particularly max_weight constraints, which limit concentration without eliminating the zero-weight mechanism —as well as more granular train-test sensitivity analysis, the inclusion of rebalancing costs, and regime-aware switching between allocation methods. Mean-variance and mean-semivariance optimisers may also warrant a second look under stronger regularisation, given that their overfitting pathology is well understood and potentially addressable.

The broader point is that a portfolio of strategies is not simply a collection of alphas. It is a system with its own dynamics, and how capital is distributed within that system is as consequential as the quality of the individual components that populate it.

Thank you for reading Issue 28 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Vyas, A. (2023) The Hierarchical Risk Parity Algorithm: An introduction, Hudson & Thames. Available at: https://hudsonthames.org/an-introduction-to-the-hierarchical-risk-parity-algorithm/ (Accessed: 11 March 2026).
2. Kimutai, T. (2025) Portfolio Optimization Explained: Mean-Variance and Risk Parity Models, Medium. Available at: https://timkimutai.medium.com/portfolio-optimization-explained-mean-variance-and-risk-parity-models-89ba519000fb (Accessed: 11 March 2026).
3. Gerát, B. (2025) Mean-Variance and Minimum Variance Portfolio Models in Python, Medium. Available at: https://medium.com/@BorisGerat/mean-variance-and-minimum-variance-portfolio-models-in-python-64a5c6b57b2d (Accessed: 11 March 2026).
4. Lopez de Prado, M. (2016) Building Diversified Portfolios that Outperform Out-of-Sample, Journal of Portfolio Management.
5. Choueifaty, Y. and Coignard, Y. (2008) Toward Maximum Diversification, Journal of Portfolio Management.
6. Delage, G. et al. (2023) skfolio: Portfolio Optimization in Python.
7. Wikipedia, Borda count (2025) Wikipedia. Available at: https://en.wikipedia.org/wiki/Borda_count (Accessed: 11 March 2026).

Comparing Portfolio Weight Allocation Methods was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Liquidity is a multi-dimensional concept spanning trading cost, immediacy, and capacity, yet it is often proxied by a single metric. This issue contrasts trade-based measures (e.g., volume) with order-based measures (e.g., spreads and depth) and shows why they can disagree.

Using Aitken & Comerton-Forde (2003), we highlight a crisis period where volume rises even as relative bid–ask spreads widen (+101%), implying worsening liquidity. The paper resolves the depth paradox by weighting order-book value by execution probability across price bands. Their approach is also met with critiques on generalisation, potential sorting bias, and limited evidence on predictability. The takeaway is methodological: liquidity measures are not interchangeable, so proxy choice should match the research objective.

1. Defining Liquidity

Liquidity in trading markets is defined as how quickly an asset can be converted into cash and vice versa without affecting its market price and at the lowest transaction costs [1, 2]. In a healthy growing market, there must always be buyers and sellers ready to transact (limit orders), so that people can quickly sell or buy assets (market orders); a market without liquidity is a dead pool with very little trading activity.

Exchanges always ensure that a reasonable level of liquidity is maintained, as they otherwise would cease to operate. Entities that provide limit orders on the order book are known as market makers; they are very much welcomed by exchanges because they offer convenience to market takers who wish to trade immediately, thereby injecting trading activity into the platform’s market.

Not every trader wants to be a market maker because of several reasons: (1) high initial capital is required to buy/sell assets from other people; (2) the position is exposed to inventory risk due to fluctuating price levels; (3) technical skills are required to maintain market-balancing algorithms. To cover such risks and costs borne by market makers, market takers usually pay a premium to market makers, which is known as the bid-ask spread. Market makers can exert certain influence over the price level, adjusting their profit margins depending on the level of risk they are exposed to. They can also refuse to buy/sell at risky price levels by adjusting their order book orders.

2. Measures of Liquidity

Having understood how markets work, we can then see how liquidity may be measured by two main approaches: trade-based and order-based measures [3].

Trade-based measures focus on using OHLCV data, such as trading volume, trading value, number of trades, turnover ratio, taker buy-sell ratio, etc. These measures are ex post rather than ex ante: they assess trading activity that has happened in the past, which is not necessarily a good indicator of what will be traded in the future.

On the other hand, order-based measures that utilise order book data can be split into different components [4]:

1. Width. This refers to the gap between the bid-ask spread.
2. Breadth. It is the volume at the best bid/ask price.
3. Depth. It measures how much a given volume is available at each price level.
4. Immediacy. The time it takes to make a trade.
5. Resiliency. The time it takes for the market to restore a temporary price imbalance.

It is important to note that the bid-ask spread can under- or overestimate the liquidity for large institutional orders, though it works effectively and accurately for small investors. While order-based measures represent future potential trading activity, such activity is not necessarily realised because it is yet to happen. It could be that limit orders are deep but no one is willing to take any trades. Both families measures have their respective limitations.

Aitken & Comerton-Forde’s work shows that trade-based and order-based measures are not strongly aligned [3]. They quantify this directly via correlations between common measures. Trade activity measures (volume, value, count) are strongly correlated with each other as expected, but they have weak and sometimes slightly negative correlations with order-based measures like relative bid–ask spread. Therefore, the authors explicitly argue that measurement choice can drive research results when studying the economic impact of market design changes.

To find out which approach is more accurate in measuring liquidity, the paper uses a practical idea: take a period that is widely viewed as a liquidity crisis, then test which measures behave consistently with that premise. Both families of measures should successfully show a decline in liquidity.

3. Which Approach Is Better?

Aitken & Comerton-Forde calculated the volume, relative volume, frequency and value for trade-based measures; and relative spreads and order depth for order-based.

Across the whole sample, average trading volume increased by approximately 51% after the crisis period begins. The top quartile of stocks shows an even larger increase (around 81%). If we equate more volume with more liquidity, we would conclude the crisis was good for liquidity. The authors call this incongruent with the accepted view that the market faced a liquidity crisis.

On the other hand, relative bid–ask spreads increased by 101% on average. That is the exact opposite conclusion from volume, as higher transaction costs indicate lower levels of liquidity.

However, the paper finds large increases in relative depth calculated using all price levels in the order book. At face value, depth suggests liquidity improved, again contradicting the spread measure and crisis narrative. However, Aitken & Comerton-Forde’s approach of including all price levels into depth calculation does not help in measuring liquidity accurately; after all, limit orders too far away from the best bid/ask price are less likely to be executed.

To deal with far-away depth, Aitken & Comerton-Forde estimate execution probabilities by price bands. They compute, for each band, the fraction of placed orders that get executed (partial fills included) with the results below:

• Orders within 1% of midpoint have about 86.65% (bid) and 82.09% (ask) execution probability.
• Orders more than 50% away have only 0.51% (bid) and 0.04% (ask) chance of execution.

They then construct a new liquidity measure that weights order values by these execution probabilities, explicitly aiming to capture the notion of latent liquidity (liquidity that becomes available as prices move).

When they apply this weighted measure, the story aligns with crisis expectations: liquidity declines for the sample as a whole by approximately 32%, with a smaller decline in the most liquid quartile (~24%). The decline is greater on the ask side (weighted ask value falls ~36% vs ~27% on the bid). They therefore conclude that order-based measures work better than trade-based measures.

4. Critique of the Paper

The Liquidity Premium and Investment Horizon paper largely agrees with the experiments conducted by Aitken & Comerton-Forde: trade-based measures perform poorly as liquidity measures in this setting [5], but it presses on several critiques regarding generalisation and predictability.

Firstly, using a known crisis as the evaluation benchmark is reasonable, but we do not know whether measures behave similarly in normal conditions. Market participants may be less rational during crises, which could change how proxies relate to true liquidity.

Another critique is regarding potential classification bias. The sample is partitioned into liquidity quartiles based on trading value (an ex post measure), which can influence how the order-based methods appear to perform.

In addition, the critique argues the paper examines how measures and liquidity change simultaneously, but does not test whether measures predict future changes in liquidity. For many research questions, predictive power matters more than contemporaneous description.

Finally, the critique warns the weighted order value measure may overestimate illiquidity because it does not account for trading within the bid–ask spread.

5. Conclusion

Liquidity is not a single observable quantity; it is an emergent property of market microstructure that combines cost (spreads and fees), immediacy (time to trade), and capacity (how much size can be transacted without meaningful price impact).

The main lesson from Aitken & Comerton-Forde is therefore methodological rather than market-specific: liquidity measures are not interchangeable. In their crisis setting, trade-based measures such as volume can move in the opposite direction from transaction-cost measures (spreads), and naïve depth measures can be inflated by orders placed far from the touch that are unlikely to execute. Their probability-weighted depth framework is an attempt to bridge this gap by separating displayed liquidity from executable liquidity via execution likelihood, producing a picture that is more consistent with crisis intuition. Their conclusion that order-based measures are more accurate than trade-based measures should also be considered with care.

Practically, the implication is to treat liquidity measurement as a modelling choice tied to the decision at hand. If the goal is to compare market quality, a spread- and execution-weighted view of depth is more defensible than volume alone. If the goal is capacity or slippage realism in backtests, the relevant question becomes whether the measure approximates tradeable liquidity near the market, not just activity or distant book volume. In other words: when a paper, dashboard, or strategy claim relies on liquidity, the first scientific step is to ask which liquidity—and whether the measurement is consistent with the mechanism being argued.

Thank you for reading Issue 27 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

Update: We are switching from weekly posts to publishing as needed.

References

1. Hayes, A. (2025) Understanding liquidity and how to measure it, Investopedia. Available at: https://www.investopedia.com/terms/l/liquidity.asp (Accessed: 03 February 2026).
2. Investor.gov (no date) Liquidity (or marketability), Liquidity (or Marketability) | Investor.gov. Available at: https://www.investor.gov/introduction-investing/investing-basics/glossary/liquidity-or-marketability (Accessed: 03 February 2026).
3. Aitken, M. and Comerton-Forde, C. (2003) ‘How should liquidity be measured?’, Pacific-Basin Finance Journal, 11(1), pp. 45–59. doi:10.1016/s0927–538x(02)00093–8.
4. Bennemark, K. and Chen, J.-F. (2007) Does tick size matter? — evidence from the Stockholm Stock Exchange. Available at: http://arc.hhs.se/download.aspx?MediumId=327 (Accessed: 03 February 2026).
5. Vorster, B. (2008) Liquidity Premium and Investment Horizon, University of Pretoria. Available at: https://repository.up.ac.za/bitstream/handle/2263/27193/dissertation.pdf?sequence=1 (Accessed: 03 February 2026).

Measuring Liquidity was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dear all,

Wishing you a Merry Christmas and Happy New Year 2026!

Our newsletter will resume normal update frequencies after the month of January 2026.

Best regards,

Liana from Balaena Quant

Merry Christmas and a Happy New Year was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

For beginners in quantitative research, 2D time series is the simplest data to start with: one scalar per day, easy to standardise, easy to backtest, easy to optimise. But many of the most information-rich features are not scalars at all, but snapshots—distributions over buckets that evolve over time. That is 3D data: time × bucket × mass. This issue suggests a simple, explainable pipeline to convert distribution snapshots (price buckets) into a 2D signal without leaning on black-box modelling. In essence, we are engineering our own data features.

1. Comparing Snapshots

Suppose we receive a daily histogram of some market-related quantity by price bucket, as shown in Figure 1.1. Each day we have:

• An x-axis: bucket boundaries (e.g. price ranges)
• A y-axis: mass in each bucket (counts, volume, supply, whatever)

Our hypothesis involves looking at the change in the shapes of the histogram: if the shape of the histogram changes tremendously, it might suggest that price volatility will be high or price changes will be large.

To validate our hypothesis, the naive approach is to compute distance between histograms day-to-day. But there is an immediate trap for using the metric: the buckets themselves might change over time. If the bucket grid moves (new upper bound, new spacing, new bin edges), then the histogram can “change” even if the underlying reality did not. That is not a signal. That is a coordinate system bug. So the first step is to ensure all snapshots to lie on the same grid before we measure change.

Below is a toy example with deliberately simple numbers.

Day 1 grid (linear):

• Buckets: 0–100, 100–200, 200–300
• Mass: 50, 30, 20

Day 2 grid changes because the range expanded:

• Buckets: 0–150, 150–300, 300–450
• Mass: 50, 30, 20

If we compute a distance between these two histograms as-is, we will conclude that something changed. In fact, nothing did; it is simply because the x-axis changed.

This is exactly what happens in real datasets whenever the grid definition depends on a moving quantity (an expanding range, a rolling max, a recalibrated scale). It produces fake differences that are indistinguishable from real differences unless we fix the grid by performing re-binning.

When carrying out re-binning, we must avoid forward looking bias by two possible methods: (1) arbitrary grid with static min-max and number of bins; (2) rolling grid with adaptive min-max and number of bins. However, Method (2) can be more complicated than necessary as it involves the choosing of yet another hyper-parameter. We select Method (1) to keep it simpler and more auditable.

2. Preparing the Data for Computing Distance

A fixed grid needs three ingredients:

1. A minimum x (avoid zero if log space is applied)
2. A maximum x (a cap)
3. Number of bins

Despite being arbitrary, it is transparent and easy to implement. We can justify it as an engineering choice rather than an alpha claim. A practical rule is to pick a maximum that is high enough that it is difficult to be exceeded most of the time. If new data in the future goes over the maximum, researchers will need to update it in a new version of the research with a clean cutover date.

After fixing our constants, we conduct re-binning, which means taking an old histogram defined on old bin edges and redistributing its mass onto a new set of edges. By doing this, we assume that mass is uniformly distributed within each original bin. While it might not be perfectly true, this assumption is explainable, deterministic, and stable. Here is minimal Python-style pseudocode for mass-preserving rebinning:

def rebin_mass_preserving(old_edges, old_mass, new_edges):
    # old_edges: length N+1, old_mass: length N
    # new_edges: length M+1
    out = zeros(M)

    for each old bin [a,b] with mass m:
        for each new bin [c,d] that overlaps:
            overlap = max(0, min(b,d) - max(a,c))
            out[new_bin] += m * overlap / (b - a)
    return out

In addition, we need to consider scale of the y-axis; many distributions drift in total size over time. If we compare raw masses, our distance will often be dominated by total growth rather than shape change. If the research hypothesis is “shape changes matter”, then we would want to remove scale by converting each histogram into probabilities (fractions that sum to 1).

This is not a philosophical statement, but rather a statement about what researchers want their metric to respond to. Keeping raw mass means that the distance becomes a mixture of shape shift and total size shift. Normalise instead, and it becomes mostly about shape. This also solves a common visual artefact: early-era buckets with huge raw piles stop dominating everything.

Once everything is on the same grid and normalised, there is still a subtle choice: for x-axis with price buckets, should we choose linear or log scale? On a linear axis, moving from (1) $10k to $20k costs the same as moving from (2) $90k to $100k (both are $10k apart). However, the same increase in (2) has lesser impact than in (1) because markets tend to behave multiplicatively; a move from $10k to $20k is double, but a move from $90k to $100k is not. Therefore, when buckets are price-like, the log space is a better coordinate system, where distances correspond to ratios. $10k to $20k is the same “distance” as $50k to $100k because both are double.

Figure 2.1 shows an example of the scaled data on a fixed log grid, revealing a very different shape than Figure 1.1.

3. Wasserstein Distance (Earth Mover Distance)

There are many ways to measure the distance between two probability distributions: L2 distance, Hellinger distance, etc. [1]. The common feature is that they are binwise; they compare distribution 1, bin n to distribution 2, bin n at the same index and then add up the differences. In other words, the only thing that matters is how much mass sits in each bin, not where those bins live on the x-axis.

These binwise metrics ignore the underlying geometry of the space. They work on aligned grids, but they still do not price in how far mass moved along the x-axis. A one-bin shift and a fifty-bin shift can look similarly different if the same amount of mass is displaced.

This is exactly why these distances become brittle in our use case. Our data is a time series of histograms where the x-axis has economic meaning (price-like buckets), and the bucket grid can change when the scale changes. If yesterday’s bins and today’s bins do not line up, e.g., L2-style distance will tell us that “everything is different” because it is forced to compare the wrong bin pairs.

Wasserstein distance does the opposite: it is geometry-aware. It treats a distribution as mass sitting on a line, and measures how much work is required to move this yesterday’s mass to match today’s mass [2, 3]. Moving mass to a nearby bin is cheap; moving it far is expensive. That work interpretation is precisely what we want if our hypothesis is large redistributions across price regimes matter.

Wasserstein-1 is especially convenient because it can be computed from cumulative sums. If we already have binned probabilities, we can compute it quickly and deterministically. The result is a daily time series where high values represent a big redistribution event, and low values suggest a quiet reshuffling, which can produce data that detects regime intensity.

However, as a distance is always non-negative, we cannot tell the direction from the absolute magnitude of Wasserstein alone. If the strategy requires the direction of the change, we can add a simple calculation as an add-on heuristic: the net shift of the distribution’s centre in the chosen coordinate system (use log centres if we are using log edges). We compute a weighted average bucket centre each day and take the day-to-day difference; a positive value means the centre moved to higher buckets, negative means it moved lower buckets. This calculation returns a signed series that can be combined like so: signed magnitude = Wasserstein magnitude × sign(net shift).

Nevertheless, we should keep the components of the net shift too, because they answer different questions. Importantly, it separates two behaviours: (1) two-way churn with lots of movement but centre stays similar; (2) net migration where centre clearly shifts up or down. This is also more honest for research. Many markets have days where the distribution changes a lot in both directions; forcing a sign onto that can be misleading.

Here is a toy distribution with five bins on a fixed grid. Say the bins represent increasing price regimes.

Day 1 probabilities:

• [0.40, 0.30, 0.20, 0.07, 0.03]

Day 2 probabilities:

• [0.25, 0.30, 0.25, 0.12, 0.08]

This is a clear low → high drift: mass left the first bin and appeared in the right tail. Wasserstein magnitude will be large because mass must be transported rightward. Net shift is positive because the centre moved higher.

Now a different Day 2:

• [0.30, 0.25, 0.25, 0.10, 0.10]

There are still changes, but less directionally. It spreads into both mid and high. The centre might move only slightly.

A third Day 2:

• [0.40, 0.10, 0.20, 0.10, 0.20]

This is violent reshuffling with mixed direction. Wasserstein can be large, but net shift might be near zero. That is the point: magnitude and direction are not the same thing, and we should not force them to be. At the end of the pipeline we have one or more daily series, which can then be plugged into standard CTA mechanics at the sole discretion of researchers:

• w1_mag. Regime intensity.
• net_shift. Signed migration.
• optionally signed_w1. Direction × Intensity.

4. Conclusion

Turning snapshot distributions into 2D signals is not a modelling flex but rather a hygiene problem. Most false positives in this area do not come from subtle statistics; they come from sloppy comparability: moving grids, wrong geometry, scale drift, and distance measures that do not match the story we tell.

The Wasserstein distance could be an approach too simple for someone’s liking; there is always the temptation is to do something clever (fit a curve, train a model, compress with PCA), but the danger is that clever usually means quietly leaking assumptions, and in a cheap-backtest world, leaked assumptions turn into false positives.

This approach is not tied to any one dataset. It is a general way to treat time series of distributions as first-class research objects, rather than awkward charts to be eyeballed and then ignored. Once we have this machinery, we can argue about whether the market cares about distribution shifts. But at least we are arguing about the market, not about our bin edges.

Thank you for reading Issue 26 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Chung, J.K. et al. (1989) ‘Measures of distance between probability distributions’, Journal of Mathematical Analysis and Applications, 138(1), pp. 280–292. doi:10.1016/0022–247x(89)90335–1.
2. SciPy (2025) Wasserstein_distance#, wasserstein_distance — SciPy v1.16.2 Manual. Available at: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wasserstein_distance.html (Accessed: 19 December 2025).
3. Kolouri, S. (no date) Optimal Transport and Wasserstein Distance, Carnegie Mellon University. Available at: https://www.stat.cmu.edu/~larry/=sml/Opt.pdf (Accessed: 19 December 2025).

Converting 3D Price Bucket Snapshots Data to 2D Signals was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a crypto CTA setting with essentially unbounded on-chain factor space and cheap backtests, every extra configuration increases our chances to find a false positive, quietly inflating tools like DSR and PBO. We introduce the classical secretary problem as a stylised optimal stopping rule for backtesting, and contrast this with random search. Framed this way, secretary-style search becomes “random search plus a fixed minimum exploration budget and an adaptive performance threshold”, giving us a numerical way to reduce our number of tested configurations.

1. Optimal Stopping Rule

In systematic research, while we put a lot of focus on how to pick a strategy, most of the danger actually lives one level higher: when do we stop looking? Given enough time and enough parameter combinations, we can always find a Sharpe that looks exciting on a backtest (for more information on false positives, refer to False Discovery Rate (FDR)). That is not a sign of skill; it is what happens when you give randomness enough chances to impersonate skill (for a mathematical explanation, refer to Sharpe Ratio: Distributions, Hypothesis Testing, and P-Values).

In crypto CTA this problem is amplified. The factor space is basically unbounded; on-chain data alone contain flows, holdings, age bands, labels, and entities in absurdly many ways. Every reasonable idea spawns a huge parameter grid: windows, thresholds, models, entry/exit rules… Moreover, backtesting loop is cheap, where we can sweep thousands of configurations on a laptop in an afternoon. Therefore, if we do not have a clear stopping rule, the effective number of trials just drifts upwards, arbitrarily inflating the Deflated Sharpe Ratio or The Probability of Backtest Overfitting (PBO).

All the anti-overfitting machinery (DSR, Rademacher, PBO, etc.) is always better paired with a smaller number of trials. So the point of an optimal stopping rule is to cut down the number of configurations using well-defined policy, other than applying theoretical restraints to our backtest. The secretary problem gives one stylised way to think about this.

2. Secretary Algorithm

The classical secretary problem goes like this [1, 2]:

• We have (N) candidates who arrive in random order.
• After each interview we must either hire them (and stop) or reject them (and never see them again).
• We want to maximise the chance of picking the single best candidate.
• However, each additional candidate increases our chances of finding a false positive.

The famous solution is to

• Observe the first approximately 37% (N/e) candidates, hire none, just remember the one with the best performance (where e is the Euler’s constant).
• Then hire the first candidate who is better than everyone we have seen so far.

We do not literally have HR constraints in research, as we can always choose from the pool of candidates seen so far, but the framework maps surprisingly well:

1. Candidates = strategy configurations. For a given idea, define a set of plausible parameter combinations, e.g. window {10, 30, 50, …, 390}, threshold {0.0, 0.1, …, 4.0}. This is best filtered based on the theoretical assumptions for the strategy. For example, if a strategy is believed to profit from mean reversion, do not test the trend-following entry-exit logic.
2. Random arrival = randomised grid. Randomly permute the list of (window, threshold) pairs. The order in which we run backtests is now a random variable, which is the arrival order.
3. Exploration phase (37% of configs). Backtest the first N/e or so configurations, compute Sharpe and other diagnostics, keep track of the best Sharpe so far, but do not select any of them as the final strategy. This phase is pure observation as we learn what “typical” performance looks like for this idea.
4. Stopping phase (online selection). Starting from the configuration number N/e + 1, evaluate strategies one by one in the random order. For each new configuration, if its Sharpe is not better than the best so far, we move on to the next one. However, if its Sharpe beats the best so far, we stop the search and choose this as our final strategy.

The crucial discipline is that we do not go on to test the remaining configs just to see if there is something nicer. Those unexplored strategies remain unknown.

import numpy as np
import pandas as pd
from math import e


def secretary_search(
    df,
    factor,
    window_list,
    threshold_list,
    fees=0.0006,
    sr_multiplier=24,
    random_state=42,
):

    param_grid = [
        (int(w), float(round(th, 3)))
        for w in window_list
        for th in threshold_list
    ]

    N = len(param_grid)

    rng = np.random.default_rng(random_state)
    rng.shuffle(param_grid)

    # 1/e rule: number of configs used purely for exploration
    n_explore = int(N / e)

    results = []
    best_sr = -np.inf
    best_config = None
    best_report = None

    # Exploration phase
    for i in range(n_explore):
        window, threshold = param_grid[i]
        report = backtesting(
            df=df.copy(),
            threshold=threshold,
            window=window,
            fees=fees,
            sr_multiplier=sr_multiplier,
            factor= factor,
        )
        report["window"] = window
        report["threshold"] = threshold
        report["config_idx"] = i
        results.append(report)

        if report["SR"] > best_sr:
            best_sr = report["SR"]
            best_config = (window, threshold)
            best_report = report

    # Sequential selection phase
    stop_idx = N - 1

    for i in range(n_explore, N):
        window, threshold = param_grid[i]
        report = backtesting(
            df=df.copy(),
            threshold=threshold,
            window=window,
            fees=fees,
            sr_multiplier=sr_multiplier,
            factor=factor,
        )
        report["window"] = window
        report["threshold"] = threshold
        report["config_idx"] = i
        results.append(report)

        # Secretary rule: stop at the first config that beats everything seen so far
        if report["SR"] > best_sr:
            best_sr = report["SR"]
            best_config = (window, threshold)
            best_report = report
            stop_idx = i
            break

    results_df = pd.DataFrame(results)

    meta = {
        "N": N,  # Total number of theoretically justifiable configs
        "n_explore": n_explore,  # Size of the 1/e exploration set
        "stop_idx": stop_idx,  # Index at which we stopped (how many backtests we actually ran)
    }

    return best_config, best_report, results_df, meta

window_list = np.arange(10, 400, 20)
threshold_list = np.arange(0, 4, 0.1)

best_config, best_report, secretary_results, meta = secretary_search(
    df=df,
    factor=factor,
    window_list=window_list,
    threshold_list=threshold_list,
    fees=0.0006,
    sr_multiplier=24,
)

print("Total configs in grid (N):", meta["N"])
print("Exploration configs (≈ N/e):", meta["n_explore"])
print("We stopped after index:", meta["stop_idx"])
print("Chosen config:", best_config)
print("Chosen SR:", best_report["SR"])

Figure 2.1 and Figure 2.2 show heat maps generated from the secretary algorithm and the default grid search respectively for the same Strategy A. The configurations chosen with secretary search yields a similar Sharpe of approximately 2 when compared to the exhaustive grid search. Secretary search successfully selects the best configuration from the two dark green areas in Figure 2.2 (around (70, 1.4) and (290, 2.5)), showcasing its ability to choose good quality parameters.

In the literal secretary problem, we cannot go back to past candidates, but in backtesting we can. However, once recall is allowed, 37% is no longer guaranteed to be optimal in a strict mathematical sense, although it still serves as a reasonable exploration budget that balances information and search cost. The exact optimal percentage is dependent on the search space’s true Sharpe distribution (which is unknown)—for example, if the distribution is flat, a smaller exploration fraction is enough; but if the space is spiky, more exploration is needed. Further research can be done on this through simulation.

Also note that the algorithm does not guarantee that we pick the globally best Sharpe in the grid. It trades some probability of hitting the absolute maximum for a controlled, finite number of tests, and a non-insane chance of getting something near the top. In fact, we can treat the secretary algorithm as an optimisation method.

3. Comparing With Random Search

The secretary algorithm shares some similarities with something simpler and more familiar: random search with fixed coverage, which we will define more strictly than the ML textbooks usually do. Suppose we have N plausible configurations. By random search with coverage C, e.g. C = 37%, we specifically mean:

1. Randomly permute the list of N configs.
2. Take the first K = floor(cN) of them.
3. Backtest exactly those K configurations.
4. Pick the configuration with the highest Sharpe within that subset.
5. Ignore the remaining (N — K) configurations completely.

In this version of random search, K is fixed without any adaptive stopping. If the first 10 configs are terrible, we still run all 37. If the very first config is amazing, we still run all 37. From an overfitting perspective, this is quite clean, as we fix our effective trials = K with nothing more and nothing less. However, this non-adaptive optimisation method means that we have a higher risk missing the global best configuration in the full grid.

When comparing the secretary algorithm to random search, (1) both start by randomising the order of all plausible configs; (2) both naturally suggest something like C = 37% as a meaningful chunk of the grid; (3) both are, in spirit, random search over a reasonable parameter space. The difference is what happens after that initial chunk.

Under our definition:

• Random search with 37% coverage. Evaluate K = 0.37N configs, chosen at random. Stop. Choose the best Sharpe among those K as the final strategy. Effective trials is exactly K.
• Secretary-style search. Evaluate the first 0.37 N configs as a mandatory exploration budget. Then continue sequentially, configuration by configuration, with a moving threshold equal to the best Sharpe seen so far. Stop at the first config that beats that threshold. Choose the best-so-far among everything we have seen up to that stopping point. Effective trials is a random K, always ≥ 0.37 N but usually < N.

Essentially, we can define the secretary algorithm as random search + fixed minimum exploration budget + adaptive extra coverage driven by a dynamic performance threshold. The fixed minimum budget (say 37%) guarantees that we do not jump at the first shiny backtest; we have seen enough of the landscape to calibrate what “good” even means for this idea. The adaptive part also acknowledges reality. Sometimes a clear outlier shows up later, and it is rational to keep looking until we see something materially better than our exploration set. But we do not keep going forever; the moment that condition is met, we stop and freeze our choice.

A pure random-37% policy is stricter and simpler. It is great when our priority is to control overfitting above all, killing the idea after exploring 37% of the dataset. This guarantees reducing backtesting time to a fixed 37% of the original grid search algorithm. On the other hand, a secretary-style policy is more forgiving as it gives us a principled way to extend the search when the exploration set is underwhelming, while still producing a clear stopping time and a smaller K than grid search for our DSR and Rademacher penalties. However, its run time will vary depending on luck, with a worst case scenario of using equal amounts of time as grid search.

4. Conclusion

In practice, neither grid search, random search, nor the secretary algorithm is “the answer”. What matters is that we stop pretending the search process is invisible. Once we acknowledge that every extra configuration is another roll of the dice, it becomes natural to pair our favourite performance metrics with a clear experimental budget and a stopping rule. The secretary algorithm is useful not because 37% is magical, but because it forces us to separate two phases: first we learn what typical performance looks like, then we only extend the search if a new configuration materially beats that baseline.

For crypto CTA and on-chain factors, this kind of discipline is not optional. Our data is rich, noisy, and extremely easy to over-exploit, and anti-overfitting tools only make sense when they are fed a realistic trial count. Thinking in secretary-style terms, or in terms of fixed-coverage random search, gives us a vocabulary to talk about that trial count explicitly: how big a search did we run, what rule did we follow, and how much luck did we use to get this result. From there, extending the framework—different exploration fractions, other stopping rules, or more sophisticated optimisation schemes—becomes an engineering choice rather than folklore.

Thank you for reading Issue 25 of Balaena Quant Insights. We look forward to sharing more of our research and perspectives with you in the next issue.

References

1. Ferguson, T.S. (1989) ‘Who solved the secretary problem?’, Statistical Science, 4(3). doi:10.1214/ss/1177012493.
2. Bailey, D.H. and López de Prado, M. (2014) ‘The deflated Sharpe Ratio: Correcting for selection bias, Backtest overfitting and Non-Normality’, SSRN Electronic Journal [Preprint]. doi:10.2139/ssrn.2460551.

Secretary Algorithm: When to Stop Backtesting was originally published in Balaena Quant Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.