Explore detailed Vision Transformer

Introduction

Every architecture in this series has assumed a discrete vocabulary, and an image has none.

Tokens must therefore be invented, and the patch size that invents them fixes everything downstream: n = (S/P)² + 1, so at 224 pixels with P = 16 there are 197 tokens.

Three exact identities follow. The patch embedding is a strided convolution, not an analogy for one, verified at 3.553×10⁻¹⁵.

At ViT-Base’s P = 16 with d = 768 the map is exactly square, since 3 × 1⁶² = 768, so patchification is a change of basis that discards nothing, while P=32 compresses 4× and loses at least 2304 dimensions per patch by rank alone.

And without position embeddings a ViT is exactly permutation-invariant over patches, measured at 1.776×10⁻¹⁵: a shuffled image is literally the same input, with 196 patches collapsing 1214.8 bits of arrangement.

Table of Contents

• Introduction
• Table of Contents
• What is a Vision Transformer
• Core Components of a Vision Transformer
• Mathematical Foundation
• Types of Patch and Position Schemes
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is a Vision Transformer

Definition

• A Vision Transformer cuts an image into non-overlapping P × P patches, maps each through one shared linear layer, adds a position embedding, and applies a standard transformer encoder:
• Everything after E is Article 91’s architecture unchanged, so the 12d² block and the n/(2d+n) cost share both apply directly.

What changes relative to a language model

• The fourth row is the substantive one.
• In a language model n is a property of the input; here it is a hyperparameter, and Section 5 shows it to be the most consequential one in the architecture.

Core Components of a Vision Transformer

The patch size sets the sequence length and the cost

• At P = 4 the quadratic term is 67.13% of attention cost, so the sequence-length problem that Article 91 located at n = 2d arrives through a choice of patch size rather than through a long document.

Mathematical Foundation

The P⁻⁴ scaling, and where it is not exact

Since n ≈ (S/P)², the quadratic term 2n²d scales as P⁻⁴:

The measured ratios approach 16 without reaching it, and the entire shortfall is the single class token: removing it recovers 16.0000 exactly at every step.

The power law is exact in the patch count and only asymptotic in n.

Per-layer cost at d = 768, using Article 91’s 4nd² + 2n²d:

Halving the patch size from 16 to 8 costs 5.3368×, not 16×, because the projection term is only linear in n and still dominates at these lengths.

The patch embedding is a strided convolution

Cutting into non-overlapping patches, flattening each, and applying one shared linear map is identical to a convolution with kernel P and stride P. This is an identity, checkable directly:

The identity is that flattening a patch and applying a shared matrix is the same contraction as a convolution evaluated at a stride-P offset:

So a ViT’s first layer is a convolution, and the only one in the model.

The consequence is exact: a stride-P kernel-P convolution has receptive field exactly P with no overlap, so within-patch structure is seen once by a linear map and no convolution ever crosses a patch boundary.

At ViT-Base the patch embedding is exactly square

Whether patchification can be lossless depends entirely on 3P² against d:

The rank of the map is bounded by the smaller of its two dimensions, which decides the question outright:

At ViT-Base/16 the numbers coincide: 3 × 1⁶² = 768 = d. The patch embedding is a square matrix, so patchification is a change of basis that discards nothing in principle.

Solving for the original patch recovers it to 3.972×10⁻¹³ at a condition number of 7.6398×1⁰³.

At P = 32 the map is 3072 → 768, discarding at least 2304 dimensions per patch by rank alone and retaining 0.2500. At P = 8 it is 192 → 768, an expansion whose rank is at most 192, or 25.0000% of the embedding space.

That is Article 99’s rank argument again, now on the input side of a vision model rather than a vocabulary.

Without position embeddings a ViT is a bag of patches

Article 91 measured attention as permutation-equivariant to 1.332×10⁻¹⁵, and for images the consequence is sharper because patches carry no intrinsic order.

A shuffled image is literally the same input, and the positive control confirms the measurement means something: with position embeddings the same quantity reaches 5.380.

Article 95 priced permutation at log₂ s! bits, and the same formula applies:

At P = 16 that is 1214.8 bits, larger than any corruption measured in Article 95, whose maximum was 1510.92 bits for deleting 77 tokens.

Layers needed to relate opposite corners

The two counts come from different growth laws, one linear in depth and one geometric:

The stride-1 figure of 112 layers is not a fair comparison, and it is worth saying so.

Real convolutional networks downsample, and with a stride-2 reduction every second layer the receptive field grows geometrically, spanning 224 pixels in 12 layers.

So the honest statement is 1 against roughly a dozen. What remains exactly true is that attention needs one layer at any image size, while any convolutional stack needs a count that grows with the image.

The embedding problem nearly vanishes

All three land within 1.2%, the closest agreement in this series. And the embedding share collapses:

Articles 98 and 99’s compression problem essentially does not exist here. There is no vocabulary to factorize, and the ceiling on embedding-side savings is under one percent rather than 21.77%.

What remains is the block share at 99.13% to 99.83%, where layer removal and cross-layer sharing apply unchanged.

Types of Patch and Position Schemes

Learned position embeddings tie the model to one resolution

• The table is n × d, so changing S or P changes n and invalidates it.
• Interpolating the table is the standard remedy and it is an approximation, whereas Article 91’s sinusoidal encodings satisfy PE(p+k) = RₖPE(p) exactly at any length.
• The choice is between a table that must be resized and a function that need not be.

Overlapping patches break the identity

• Section 5’s equivalence holds because stride equals kernel.
• With overlap the first layer is still a convolution but no longer a partition, so token count rises above (S/P)² and the clean P⁻⁴ accounting no longer applies.

Evaluation Metrics

The permutation check is the same diagnostic as Article 91's

• A residual near 10⁻¹⁵ means position embeddings are not reaching the model, and the consequence for images is that the model sees a set.
• The positive control matters equally: with embeddings present the same quantity reached 5.380, so a small residual is informative only against a large one.

Report the patch size before any cost figure

• Since P determines n and n determines the quadratic share, a cost or latency claim about a ViT is meaningless without it.
• The same model at P = 32 and P = 8 differs by 22.98× in per-layer arithmetic, which is a larger spread than most architectural changes in this series produce.

Selecting Evaluation Metrics

Choosing the patch size

What this series has established about where costs are exact

Two of this series’ recurring lessons apply directly

• First, an exact ceiling is not an achievement: the 99.83% block share of ViT-Huge bounds what compression can save without indicating what it costs in accuracy, exactly as Article 98’s 77.69% did.
• Second, a rule verified across many models is still a rule about a configuration.
• Article 91’s 12d² reproduced BERT, all four GPT-2 sizes, BART and every ViT here to within 1.2%, and failed by 16× for T5–11B in Article 94.

Assumptions and Practical Considerations

On the convolution identity

• The equivalence is exact for non-overlapping patches with a shared linear map and holds at machine precision across the configurations tested.
• It says nothing about whether a ViT behaves like a CNN overall: after the first layer the two differ entirely, and the identity concerns only patchification.

On the corner-distance comparison

• The 112-layer figure assumes stride-1 convolutions throughout, which no practical network uses, and the strided calculation gives 12.
• The article reports both because the first alone would overstate the case. What is exactly true is the scaling: one attention layer at any image size against a convolutional count that grows with it.

Explore detailed ALBERT

Introduction

Article 98 left two parameter blocks to attack, and ALBERT attacks both.

Factorizing the embedding through a width-E bottleneck helps exactly when E < VH/(V+H), which at V = 30,000 and H = 768 is 748.83: a threshold so close to H that any real bottleneck saves.

Cross-layer sharing saves exactly (L-1)/L of the block parameters, which is 0.9167 at twelve layers, and exactly zero compute, since the same matrices are applied the same number of times.

That zero is the central point. Sharing decouples parameters from arithmetic completely, so the two move in opposite directions: ALBERT-xxlarge has 0.7012 times the parameters of BERT-large and 6.8000 times the compute at n = 512.

By count it is a compression of BERT-large; by arithmetic it emphatically is not. Sharing also changes what the stack computes, since applying one function L times is an iterated map.

The residual form does not converge at all, and it does so whether or not weights are shared.

Table of Contents

• Introduction
• Table of Contents
• What is ALBERT
• Core Components of ALBERT
• Mathematical Foundation
• Types of Parameter Reduction
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is ALBERT

Definition

• ALBERT is BERT with two parameter reductions and a replaced auxiliary objective. The embedding becomes a product through a bottleneck, and one encoder block is reused at every depth:
• The single parameter set θ is shared, so the stack computes fθˡ rather than f(θ_L) ∘ ⋯ ∘ f(θ_1).

What each change targets

Core Components of ALBERT

Which technique contributes more

• Sharing contributes 3.93 times more than factorization at base scale, 71.21% against 18.12%.
• And the gap widens with width, because a block grows as H² while the factorized embedding grows only as VE + EH:

Mathematical Foundation

The factorization break-even is exact

A single V × H matrix becomes V × E plus E × H, so factorization helps precisely when

The threshold approaches H as V grows, so it is not a real constraint: any bottleneck narrower than the hidden width saves, and the saving tends to V(H-E).

The last row is the control. At E = H the factorization must be a pure loss, since it inserts an H × H matrix for nothing, and it costs exactly 589,824 extra. A saving reported at E = H would indicate a bug in the accounting.

What it costs is rank. The product has rank at most E, so the embedding is confined to a rank-128 subspace: 16.6667% of the available dimensions at H = 768 and only 3.1250% at H = 4096.

The constraint applies to the input embedding alone and the model still uses all H dimensions internally, which is why so narrow a bottleneck is tolerable.

Sharing saves exactly (L-1)/L and exactly zero compute

At L = 12, d = 768, n = 512 the stack costs 19, 327, 352, 832 multiply-accumulates whether shared or not.

Sharing is the extreme case of Article 98’s finding that a parameter count is not a latency proxy: up to (L-1)/L of block parameters saved, and none of the arithmetic.

Parameters and compute, fully decoupled

Sharing divides the parameter count by L while leaving the cost untouched, so the two ratios are governed by different quantities entirely:

ALBERT-xxlarge has 0.7012 times the parameters of BERT-large and 6.8000 times the compute.

It is the smaller model by parameter count and nearly seven times the larger by arithmetic.

Parameters per billion multiply-accumulates falls from 5,202,086.77 to 536,423.54, a factor of 9.70, and that ratio is exactly what sharing manipulates.

The same divergence appears within ALBERT’s own family:

A shared stack is an iterated map

Unshared layers apply L different functions; a shared stack applies one function L times, so the representation follows an orbit.

If f is a contraction with modulus κ < 1, the step size decays geometrically and the orbit converges to the unique fixed point:

Every layer past convergence then contributes nothing.

Measured on a plain, non-residual map, the step size collapses:

The shared stack contracts harder than the unshared one at both gains, by 0.076678 against 0.000000 and 0.372474 against 0.074773, which is what iterating a single contraction predicts.

Iterating to convergence locates the fixed point exactly:

At gain 0.5 the orbit is within 10⁻¹² of its fixed point in 20 layers, so effective depth is set by the contraction rate rather than by L, and layers past that point change the output by less than machine precision.

But the residual form does not converge

Every residual configuration grows rather than shrinking, by factors from 1.667962 to 2.397955 over 24 layers.

A residual update x + h(x) is not a contraction unless h is small, so the fixed-point argument does not apply and the stack keeps moving.

Crucially, the growth appears whether the weights are shared or not, so failing to contract is a property of the residual connection rather than of sharing.

Which of the two grows faster is not stable: shared exceeds unshared at gain 0.5, 2.397955 against 1.669411, and the ordering reverses at gain 1.0, 1.667962 against 1.989650.

That reversal is why the comparison between them should not be read as a finding.

This bounds the iterated-map result to the plain case rather than to transformers as built, and it is why real shared architectures do not collapse.

Configurations against published totals

All four land below their published figures, by 7.6% to 12.4%, and the deficit grows with width.

That is a consistent under-count rather than rounding, so this accounting captures the structure of both techniques but is missing a component that scales with H.

The honest claim is that the mechanisms and their exact savings are established here; the totals are not reproduced, unlike the BERT, GPT-2 and BART counts in earlier articles.

Sentence-order prediction carries exactly what NSP did

At V = 30,000 the MLM term is 1142.2214 bits and a binary sentence label is 1 bit, so the auxiliary objective is 0.0875% of the signal, matching Article 97’s 0.0873%.

Replacing NSP with SOP changes nothing about the size of the signal, because both ask one binary question.

What changes is the shortcut available. NSP negatives come from a different document, so a topic cue suffices.

SOP negatives are the same two sentences swapped, so the topic is identical and order must be represented.

Article 95 priced exactly that operation: sentence permutation over two sentences is log₂ 2! = 1 bit and preserves the token multiset entirely.

Types of Parameter Reduction

Sharing and layer removal are opposites at equal parameter saving

• Removing half the layers saves 0.5 of block parameters and 0.5 of compute.
• Sharing across twelve layers saves 0.9167 of block parameters and 0 of compute.
• So at a larger parameter saving, sharing delivers no speedup whatsoever, and choosing between them depends entirely on whether the binding constraint is memory or latency.

Factorization is the only one that leaves compute essentially untouched for a different reason

• Its saving is in a lookup table, and a lookup is constant time regardless of vocabulary.
• The added E × H projection does cost arithmetic, but nEH against the block’s 4nH² is small: at E=128, H=768 it is 128/(4 × 768) = 4.17% of one block’s projection cost.

Evaluation Metrics

Reporting a parameter count alone is misleading for a shared model

• ALBERT-xxlarge is smaller than BERT-large by parameters and 6.80× larger by arithmetic.
• Since sharing saves exactly zero compute, the parameter count of a shared model carries no information about its cost to run, and the two numbers must be reported together.

Measure the step size before claiming effective depth

• The iterated-map argument gives a striking result for plain maps and does not apply to residual ones, where the step size grew by 2.12 instead of shrinking.
• So the diagnostic is the measured step norm across depth, not the sharing scheme: a shared residual stack is not obviously wasting its layers, and the plain-map figures do not transfer.

Selecting Evaluation Metrics

Choosing between the two techniques

When factorization is worth more than sharing

• Never, at any width tested: the ratio ran from 4.01× at H=768 to 18.37× at H = 4096, always favoring sharing.
• Factorization wins only when V is enormous relative to H² L, since its saving is linear in V while sharing’s is quadratic in H.
• At a 250,000-token vocabulary the break-even E is 765.65, so the technique still applies, but the block term dominates any realistic configuration.

What carries over from earlier articles

Assumptions and Practical Considerations

On the iterated-map result

• The convergence figures come from a plain two-matrix map with tanh, not from a transformer block, and the residual measurement shows the conclusion reversing for the form actually used.
• What transfers is the question, not the answer: a shared stack may be applying an inert map, the step norm across depth is the way to check, and the check is cheap. The specific layer counts describe the plain case only.

On the parameter totals

• Unlike the BERT, GPT-2, T5-small and BART counts in earlier articles, these do not reproduce their published values, falling short by 7.6% to 12.4% with the deficit growing in H.
• The savings formulas are exact and verified; the totals indicate a missing component and are reported as such rather than adjusted to fit.

Explore detailed DistilBERT

Introduction

Article 97 ended on the arithmetic ceilings for compression: blocks are 77.69% of BERT-base and embeddings 21.77%, so a method that removes layers cannot save more than the first figure however clever it is.

Distil BERT halves the depth, which saves exactly half the block share at 38.84%, and adds the pooler and token-type table for a total of 39.3848%.

The compute saving is a different number entirely, and it is exactly 50% at every sequence length, because depth multiplies both the projection and quadratic terms equally while embeddings hold parameters without costing arithmetic.

Distillation itself has a quantifiable rationale. A hard label gives the target’s identity; a soft target gives the whole conditional row, and at τ = 4 that row carries 2.678132 bits of shape over 8 classes.

The consequence is a sample-complexity result rather than a vague one: a student learning from soft targets reaches exactly the teacher’s error after enough observations to visit each state once, while a hard-label student trained on twenty times more data still carried excess.

Table of Contents

• Introduction
• Table of Contents
• What is DistilBERT
• Core Components of DistilBERT
• Mathematical Foundation
• Types of Compression
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is DistilBERT

Definition

• DistilBERT is a six-layer BERT trained to match a twelve-layer teacher’s output distribution rather than the data’s hard labels.
• The loss combines a temperature-scaled soft term with the original masked-language objective:
• The τ² factor is not arbitrary, and Section 5 derives exactly what it corrects for.

What is removed

Core Components of DistilBERT

The reduction, decomposed exactly

• The residual is exactly 109,482,240–43,119,360 = 66,362,880, matching the published figure.
• The token-type table is a rounding error at 0.0014%, so removing it is a tidiness decision rather than a compression one.

The embedding share rises under compression

• The embedding numerator barely changed while the denominator shrank, so layer removal makes the embedding matrix the dominant remaining target.
• That is a structural consequence of compressing one part, and it sets up the next article’s approach.

Mathematical Foundation

What fraction of the ceiling is reached

Halving the depth reaches exactly half the layer-removal ceiling, which is arithmetic rather than a finding.

What is not arithmetic is how much accuracy survives, and no amount of parameter counting predicts that.

The τ² factor, derived

Differentiating the soft loss with respect to the student logits gives

For large τ the softmax approaches uniform with pᵢτ ≈ 1/K(1 + (zᵢ — z̄)/τ), so the difference carries a second factor of 1/τ and the gradient scales as 1/τ². Multiplying by τ² therefore normalizes it.

Verified by central finite differences, the analytic form matched to a worst relative deviation of 1.342×10⁻⁸.

Whether the normalization actually works is a separate question, and a single logit pair answers it badly. Averaged over 500 independent logit pairs over 8 classes: Each row doubles τ, so the predicted ratio is 4 throughout.

The measured ratio rises to 4 from below, slightly overshoots to 4.0065 at τ = 16, then settles from above at 4.0032, 4.0014, 4.0007 and 4.0003 as τ continues doubling to 256, where the scaled value reaches its limit of 0.904667.

The scaled gradient is within 1% of that limit from τ = 4 onward, while the ratio is within 1% of 4 only from τ = 8.

Over the range actually used in distillation, τ ∈ [2, 8], the scaled gradient varies by only 1.0296×, so the τ² factor does its job in practice even though it is exact only asymptotically.

The asymptotic regime begins where τ exceeds the logit spread, which is why τ = 1 sits outside it:

With a logit standard deviation of 2.0031 and a mean range of 5.6594, larger logits push the 4× ratio further out in τ, exactly as the near-uniform argument predicts.

The dependence is not merely qualitative. Scaling the logits by s and the temperatures by s together leaves the ratio exactly unchanged, since softmax(sz/sτ) = softmax(z/τ) and the two 1/τ prefactors cancel in the ratio:

Measured at s=2 this gives 3.367496 against 3.367496, agreeing to six decimals.

Only τ relative to the logit scale matters, so the temperature is not a transferable constant: a value tuned for one teacher is the wrong setting for another whose logits differ in scale.

What temperature exposes

With log₂ K = 3.0000 bits, τ = 0.5 gives an effective class count of 1.014634, so the soft target is nearly a hard label and carries almost nothing extra.

At τ = 32 the effective count is 7.972540 of 8, so the distribution is nearly uniform and carries almost nothing about the input.

The usable signal is in between, which is why temperature is a hyperparameter and not a limit to be pushed.

The sample-complexity result

Article 97 gave the hard-label law 𝔼[D(KL)] = K(K-1)/(2N): each observation supplies one sampled outcome, so estimating a row takes many visits.

A soft target supplies the entire row per visit, so the student needs only to visit each context, which is a coupon-collector count of order Klog K.

Measured with the true chain as teacher, an upper bound on any real teacher:

The two requirements scale differently, which is the whole result:

The hard-label count grows as the accuracy target tightens while the soft-target count does not depend on ε at all.

The coupon-collector expectations are 8.05 and 23.03 steps, and the soft-target error hits exactly zero once all states have been seen.

The crossover is at the visit count, not at any accuracy target: at K=10, N=100 the soft student is exact while the hard-label student at N=1000, ten times more data, still carries 0.04317260 nats.

And the teacher’s error is a hard floor

The student matches the teacher exactly, at a ratio of 1.0000, using one to two orders of magnitude fewer samples than the teacher needed.

It never beats the teacher, because the soft targets are the teacher’s rows. That is the precise statement of what distillation transfers and what it cannot: sample efficiency, not accuracy beyond the source.

Inference cost

Depth multiplies both attention terms equally, so per-token compute falls by exactly the layer ratio:

Depth appears as a common factor in both cost terms, so it cancels out of the ratio entirely:

independent of n and d. So compute falls 50% while parameters fall 39.38%, and the two differ because embeddings hold parameters but cost essentially no arithmetic.

A lookup is constant time regardless of vocabulary.

Types of Compression

Three of these have very different compute profiles at the same parameter saving

• Layer removal cuts compute proportionally, which is why DistilBERT’s 39.38% parameter reduction comes with a 50% compute reduction. Embedding factorization saves parameters with almost no compute effect.
• Weight sharing saves parameters with zero compute effect, since the same matrices are still applied the same number of times.
• A parameter count is therefore not a proxy for latency, and the three approaches are not interchangeable despite reporting comparable headline numbers.

Width reduction is the only one that is quadratic

• Since a block is 12d², halving d removes 75% of block parameters and 75% of the projection compute, against 50% for halving depth.
• It also halves the representation size, which layer removal does not, so the two are not comparable at equal parameter saving.

Evaluation Metrics

The student/teacher ratio has a floor of one

• Soft targets are the teacher’s distributions, so a student trained purely on them cannot fall below the teacher’s error.
• Measuring a ratio of 1.0000 means distillation worked perfectly.
• A ratio below 1 would mean something other than the soft targets was supplying information, which in practice is the retained hard-label term.
• Reporting a student that beats its teacher is therefore a claim about the data term, not about distillation.

Check the effective class count before tuning anything else

• At τ = 0.5 the effective class count was 1.014634, so the soft target was nearly a hard label and the distillation term was contributing almost nothing regardless of α.
• That is a one-line diagnostic and it distinguishes a badly chosen temperature from a badly weighted loss.

Selecting Evaluation Metrics

Choosing what to compress

Deciding the temperature from the teacher’s logits

• The asymptotic table showed the 4× ratio arriving at τ = 8 for logits of standard deviation 2.0031, and at τ = 2 when the logits were halved.
• So the temperature should be set relative to the teacher’s logit scale, and a value transferred from another model with different logit statistics is not the same setting.

What carries over from earlier articles

• The last row connects directly. At τ = 32 the teacher’s distribution had 7.972540 effective classes of 8, which is the same saturation-toward-uniform phenomenon Article 91 measured from the opposite direction.
• There the concern was a softmax collapsing to a hard argmax at large d, here it is a softmax flattening to uniform at large τ.
• Both are the same statement about where a softmax carries information.

Assumptions and Practical Considerations

On the sample-complexity measurement

• The soft-target student here copies the teacher’s row on every visited state, which makes its advantage exact but also idealized.
• A real student fits a parametric function and has approximation error the measurement excludes.
• What transfers is the mechanism and its scaling, that one observation conveys a whole distribution rather than one sample from it, and that the requirement is coverage rather than accuracy.
• The absolute figures describe the idealized case.

On what parameter counting cannot say

• Every ceiling in this article is arithmetic and therefore certain. How much accuracy survives removing half the layers is not, and nothing here bears on it.
• The counts bound what can be saved; they say nothing about what it costs to save it.

Explore detailed RoBERTa

Introduction

RoBERTa changes no part of BERT’s architecture, so Article 91’s 12d² block and Article 92’s parameter totals carry over untouched.

What changes is the recipe, and several of its choices were quantifiable before any ablation. Article 92 put next sentence prediction at 1.2853% of the training signal by prediction count.

That accounting is generous to it: a binary label carries at most 1 bit while 76.8 masked tokens over a 30,522-token vocabulary carry 1144.1328 bits, so NSP is 0.0873% by information.

The largest change, an order of magnitude more data, obeys an exactly derivable law.

For a plug-in estimator the expected excess loss is K(K-1)/(2N), in which the stationary distribution cancels exactly, so doubling the data halves the excess loss regardless of the state count.

Static masking is the other quantifiable choice: ten fixed patterns leave 19.69% of every sequence never predicted at all, while dynamic masking has about 1⁰⁹³ patterns available.

Table of Contents

• Introduction
• Table of Contents
• What is RoBERTa
• Core Components of RoBERTa
• Mathematical Foundation
• Types of Recipe Changes
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is RoBERTa

Definition

• RoBERTa is BERT with the next-sentence objective removed and the training procedure changed. The objective reduces to masked language modeling alone:
• The expectation over mask sets M is the substantive difference from Article 92: BERT fixed M per copy of the data, while RoBERTa resamples it.

The five changes, and whether each was predictable

Core Components of RoBERTa

NSP measured two ways

• Article 92 counted predictions: 76.8 masked tokens against one sentence label. Counting information instead requires the vocabulary:

• Writing both shares side by side shows where the discrepancy comes from:
• The denominators differ by exactly the factor log₂ V, so the count accounting overstates NSP by log₂ V = 14.8976, up to the additive terms. Both accountings predicted that dropping NSP would cost little.
• The information one predicted it would cost about fourteen times less.

Mathematical Foundation

Static masking leaves part of every sequence unpredicted

BERT duplicated the corpus ten times with one fixed mask each. A position is masked in a given pattern with probability r, so after P patterns:

Ten static patterns leave 19.69% of every sequence never predicted, which is 100.80 of 512 positions. Forty patterns leave 0.15%.

Dynamic masking draws from C(512, 77) patterns, which Article 95 measured at 308.40 bits, or about 1⁰⁹³. The birthday bound over 40 epochs gives a collision probability of roughly 10⁻⁹⁰, so a mask is never reused.

The cost is zero: the same compute yields 40 distinct prediction problems per sequence instead of 10.

The data scaling law

For a multinomial over K categories estimated from m samples, the expected KL from the truth is asymptotically (K-1)/(2m).

A Markov chain has K rows, and row a receives mₐ = Nπₐ samples, so

The stationary distribution cancels exactly. The excess loss depends on K and N only, not on how unevenly the states are visited. Measured over 400 trials per point:

Ratios span 0.9400 to 1.0045 with a mean of 0.9679. They sit systematically below one, which is expected: the estimator uses add-0.5 smoothing, and shrinking toward uniform reduces the KL slightly.

The formula’s constant is asymptotic and shows visible bias at small N, which is exactly where the approximation is not supposed to hold.

The slope is the part the claim rests on:

And measured directly, quadrupling the data cut excess loss to 0.2441, 0.2660, 0.2434, 0.2490, 0.2569 and 0.2600 of its previous value, against a predicted 0.2500.

So RoBERTa’s roughly tenfold data increase removes about 90% of the estimation component of the loss, independent of vocabulary size, since K appears only in the numerator.

What the scaling law does not touch

The law governs estimation error alone. The process entropy is irreducible:

At a conditional entropy of 1.280264 nats, the estimation term is already under 1% of the total by N = 1⁰³ for this five-state chain.

More data has sharply diminishing returns once estimation error falls below the irreducible entropy, and the crossover is computable in advance as N = K(K-1)/(2ε H).

Batch size arithmetic

For a fixed token budget, steps = tokens/(B · n):

The gradient standard error falls as 1/√B, so the signal-to-noise ratio per step improves by √32 = 5.6569.

Under the linear scaling rule the learning rate scales with B, and the two factors cancel:

The batch size drops out entirely, so the product is exactly constant: the same total displacement in fewer, larger steps.

Tokenizer cost

Byte-level BPE includes every byte, so any string is representable and the UNK rate is exactly zero rather than small. The cost is 15,162,624 extra parameters, or 13.85% of BERT-base’s 109,482,240 total from Article 92.

Sequence length

BERT trained 90% of steps at length 128 and 10% at 512, giving a mean of 0.9 × 128 + 0.1 × 512 = 166.4. RoBERTa used full length throughout, a ratio of 3.0769. The consequences are Article 91’s and Article 92's:

Types of Recipe Changes

Two of these are free

• Dropping NSP and resampling the mask both cost nothing in compute or parameters.
• That makes them unusual: the first removes a head, and the second is a different choice of the same number of random draws.
• Any recipe change that is free and measurably positive should be adopted without an ablation, and both were computable in advance.

Full-length sequences are not free

• Going from a mean of 166.4 to 512 raises the quadratic share from 0.0769 to 0.2500, so a larger fraction of attention cost becomes the term that scales badly.
• The compensation is exact: 4× the masked predictions per sequence, from 19.2 to 76.8, and 4× the mean context.

Evaluation Metrics

Counting predictions is the wrong unit for comparing objectives

• A binary classification and a 30,522-way prediction are not one unit each.
• Article 92’s 1.2853% and this article’s 0.0873% differ by 14.72× purely from the choice of unit, and the bits accounting is the defensible one because it reflects how much the model must learn to succeed.
• Any comparison of auxiliary objectives should use bits, or it will systematically favor whichever objective has the smallest output space.

The slope matters more than the constant

• The scaling law’s constant K(K-1)/2 is asymptotic and was measured 3% to 6% low because of smoothing.
• The exponent is what supports the practical claim, and it measured -0.9930 and -0.9846.
• Reporting the constant as exact would be overclaiming; reporting the exponent as approximately -1 is supported.

Selecting Evaluation Metrics

Deciding whether more data will help

• The last row is the one the scaling law cannot answer.
• K(K-1)/(2N) is the error of an estimator that can represent the truth exactly, so it bounds what data alone can fix.
• A model that cannot express the target pays a separate, non-vanishing cost, which is exactly Article 96’s position-unaware floor of 0.114901 bits: no quantity of data removes it.

What carries over from earlier articles

• Article 92’s row is a revision rather than an extension.
• The earlier figure was correctly computed for the unit it used, and the unit was the wrong one for the comparison being made.

Assumptions and Practical Considerations

On the scaling law’s scope

• The derivation assumes a correctly specified model estimated by counting, and it gives the estimation component only.
• Real pretraining has approximation error and optimization error on top, and text is not a first-order chain over a handful of states.
• What transfers is the exponent and the cancellation: the 1/N rate, and the fact that state imbalance does not change it. The absolute magnitudes are specific to the chains tested.

On the batch-size figures

• Steps and gradient variance are exact arithmetic.
• The claim that a √B SNR gain translates into equivalent optimization is not: the linear scaling rule holds empirically over a range and fails at extremes, and nothing measured here tests that.
• The step counts and the constant displacement product are exact; the conclusion that large batches are equivalent is not established by them.

Explore detailed XLNet

Introduction

Article 95 ended on the possibility of predicting tokens in a permuted order rather than corrupting the input.

The chain rule licenses it exactly: every permutation of the factorization gives the identical joint probability, verified here at a worst relative error of 4.190×10⁻¹⁶ across eight orders including the reverse.

With log₂(512!) = 3875.17 bits of distinct orderings, a sequence is essentially never seen the same way twice, and no corruption is needed because conditioning on an arbitrary subset is already licensed.

Two exact facts make the method work. The standard softmax parameterization has no dependence on which position is being predicted, so it must emit one distribution for every target sharing a context, and that degeneracy costs 0.114901 bits per prediction on a test chain.

No training removes it. The two-stream fix has a checkable requirement: perturbing the target must leave the query stream exactly unchanged, and across 1200 perturbations spanning five orders of magnitude not one produced any change.

Partial prediction turns out to be load bearing rather than an optimization, since over full permutations the mean context is (n-1)/2 = 255.50, only 0.5871 of BERT’s 435.20.

Table of Contents

• Introduction
• Table of Contents
• What is XLNet
• Core Components of XLNet
• Mathematical Foundation
• Types of Factorization and Prediction Schemes
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is XLNet

Definition

• XLNet maximizes the expected log-likelihood over random factorization orders. For a permutation z of the positions,

• The chain rule holds for every z, not just the identity, so each term is a genuine conditional of the true joint:
• Which means the objective needs no corrupted tokens at all.

What that removes

• The third row is the substantive one. BERT predicts all masked positions in parallel from the same context, treating them as conditionally independent.
• A permuted factorization conditions each target on the previously predicted ones, so no such assumption is made.

Core Components of XLNet

Partial prediction, and the premium it fixes

• XLNet predicts only the final 1/K of the permutation, using the first n(1–1/K) tokens as context:

• Predicting the final 1/K of the order gives

• So the premium is exactly K, since a fraction 1/K of positions is predicted.
• At K = 6 that is 6, marginally better than Article 92’s 6.67 for BERT, while yielding both more predictions and more context:

Mathematical Foundation

Order invariance, verified

The chain rule gives P(x) = Πₜ P(x(z_t) ∣ xz(<t)) for every permutation z, not just the identity.

On a five-position Markov chain over three states, where each conditional is computable by exact marginalization:

Worst relative error over eight orders: 4.190×10⁻¹⁶. This is what makes the objective unbiased, and it is a property of the chain rule rather than of the architecture, so it holds for any z at any n.

The parameterization cannot see its own target

The usual output layer is P(x(z_t) = v ∣ xz(<t)) = softmax (e(v)⊤ h(xz(<t))). The hidden state depends only on the context, and nothing in the expression mentions zₜ.

So for a fixed context the model must emit one distribution for every candidate target, while the true conditionals differ:

The optimal position-unaware distribution is the mixture, and the excess cross-entropy is the weighted average KL to it:

No amount of training removes this, because the parameterization cannot express a dependence on zₜ. It is a representational floor, not an optimization failure.

The size of the floor depends on how much the conditionals differ, and the pattern is not simply about how far apart the targets are:

Positions 3 and 4 are adjacent and cost almost nothing, while 1 and 4 are the most expensive pair.

The driver is the spread in mixing distance from the context, not adjacency.

Both 3 and 4 are far enough from position 0 that the chain has largely mixed, so both conditionals sit near the stationary distribution and a single distribution serves both.

The control confirms the computation rather than the claim. For an i.i.d. chain every conditional equals π regardless of position, so a position-unaware model should lose nothing, and the measured excess is -2.030×10⁻¹⁷.

The two-stream requirement, as a perturbation identity

Two streams resolve the degeneracy. The content stream h(z_t) sees both x(z_t) and zₜ and serves as context for later predictions; the query stream g(z_t) sees zₜ but not x(z_t) and does the predicting.

The requirement is exact:

Averaging over 200 independent perturbation directions at each of six scales:

Across all 1200 perturbations, not a single one produced any query-stream change: the count of nonzero cases is exactly zero, and the invariance holds even at scale 10⁻⁶ where the content response is only 6.881×10⁻⁷.

The separation is exact at every scale, not merely where the content response is large.

Both halves of the check are needed. The first is the correctness condition; the second is the positive control, since a query stream that ignored the target token and everything else would also pass the first test.

The content response is monotone increasing and does not saturate, growing almost exactly linearly with the perturbation over the first four decades.

That is worth pinning down, because Article 91’s entropy collapse suggests the softmax should saturate and stop responding. It does saturate, and the output still grows:

The weight is bounded in [0,1] and reaches 1.000000 exactly as Article 91 predicts, but the norm of the value vector it is averaging grows linearly, because perturbing x(z_t) scales V(tgt) = x(z_t)Wᵥ itself.

A bounded weight times an unbounded value is unbounded, so saturation of the weights does not bound the output.

A single-stream model that uses the target’s own content as its query is exactly the failure this prevents: its query would move by the amounts in the content column, which is the leak.

Why partial prediction is load bearing

Predicting every position of a random permutation gives n predictions, but the token at rank t sees only t-1 context tokens, and t is uniform, so the mean context is (n-1)/2:

So permutation language modeling without partial prediction has strictly worse context than the masking it was meant to improve on, at 0.5871 of it. The distribution is what makes this concrete:

The bottom decile of predictions sees 51 tokens or fewer. Restricting to the final 1/6 raises the minimum context to 426.7 and the mean to 468.83, above BERT’s 435.20.

Partial prediction is not an efficiency tweak; it is what makes the method competitive at all.

Types of Factorization and Prediction Schemes

The first two rows have the same mean context

• A left-to-right factorization and a random one both give a rank-t token exactly t-1 predecessors, so the mean is (n-1)/2 either way.
• What differs is which tokens those are: left-to-right always supplies a contiguous prefix, while a permutation supplies a scattered subset that can include positions from the far right.
• The gain from permutation is in the diversity of contexts, not their size, and Article 92’s measurement applies directly, since that diversity is only useful insofar as right context carries information the left does not.

Choosing K is choosing a point on an exact curve

• Raising K raises both the premium and the mean context, monotonically. Going from K = 2 to K = 8 takes the premium from 2 to 8 while the mean context rises only from 383.50 to 479.50, a gain of 25% for a 4× cost.
• The returns on context are sharply diminishing because the mean is already bounded above by n-1.

Evaluation Metrics

The two-stream check needs both halves

• A query stream that is invariant to the target token passes the correctness condition.
• So does a query stream that is invariant to everything, including a bug that zeroes it out entirely.
• Only the content-stream column distinguishes the two, which is why the positive control is not optional here any more than it was for Article 93’s prefix consistency.

And it needs more than one perturbation direction

• A single direction at a single scale is a poor probe: at scale 1⁰⁰ the content response ranged from 0.236304 to 3.904082 across directions, a spread of 16.5×, and the mean of 0.900566 sits well above the median of 0.645062.
• Averaging over directions turns a noisy single number into a monotone curve. Sweeping scales as well costs nothing and confirms the response is linear rather than saturating.

Selecting Evaluation Metrics

Reading the diagnostics together

• The second row is the dangerous one for the same reason it was in Article 93: a model that conditions on the answer trains to a very low loss and has learned nothing, and the loss curve gives no warning.

Deciding whether the objective is worth its complexity

• The second row is worth restating. XLNet removes BERT’s corruption and its independence assumption, but both fixes only matter to the extent that bidirectional information exists in the data.
• Article 92’s cyclic chain gained exactly zero from bidirectional context while changing 35.0% of predictions, and on such data XLNet’s entire apparatus buys nothing while costing two attention streams.

What carries over

Assumptions and Practical Considerations

On the scope of the excess-loss figure

• The 0.079643 nats comes from a specific three-state chain with a single observed position, and it is exact for that setting rather than an estimate for text.
• What generalizes is the argument: the excess equals a Jensen gap between the true conditionals and their mixture, so it is zero exactly when the conditionals coincide and grows with their spread.
• The i.i.d. control at -2.030×10⁻¹⁷ is the boundary case.

On the perturbation check

• Perturbing the input embedding tests one layer of one attention block, which is where the leak would occur, but a full model has residual connections and normalization that could reintroduce the dependence.
• The check should be run end to end on the actual query stream output, not on a single block, and the positive control run alongside it.

Explore detailed BART

Introduction

At n = 512 document rotation creates only 9.00 bits of ambiguity, sentence permutation over 20 sentences creates 61.08, and deleting 77 tokens creates 1510.92.

Masking and deletion differ by exactly log₂C(n, k), which is 308.40 bits at those settings: that is the price of not knowing where the gaps are.

Rotation is weaker than deletion by a factor of 167.9, so calling both corruption hides a two-order-of-magnitude difference.

Whether that ambiguity translates into lost accuracy is a separate question, and on a Markov chain both posteriors can be enumerated exactly.

Masking beat deletion at every setting tested, with deletion still well above the chance floor. BART’s other choice is to reconstruct the entire document rather than only the corrupted spans, which removes Article 92’s 6.67× premium outright.

It is not free: BART pays 24.81× the decoder attention for 6.67× the predictions, so cost per prediction is 3.72× worse.

Table of Contents

• Introduction
• Table of Contents
• What is BART
• Core Components of BART
• Mathematical Foundation
• Types of Corruption
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is BART

Definition

• BART is an encoder-decoder transformer trained as a denoising autoencoder over text.
• A corruption C damages the input, and the decoder reconstructs the original document in full:
• The sum runs over all n positions, not only the corrupted ones. That single choice separates BART from T5 and drives most of what follows.

How that differs from the two neighbors

Core Components of BART

The zero-length span is deliberate

• BART draws span lengths from Poisson(λ = 3), and a length-zero span inserts a mask token where nothing was deleted:
• So P(0) = e⁻³ = 0.049787, and 4.98% of spans mark nothing at all. At r = 0.30 and n = 512 that is 51.20 spans of which 2.55 are empty.
• Their purpose is exact: they teach the model that a mask token does not guarantee missing content, which is the same shortcut-breaking role Article 92’s 1.5% unchanged tokens played for BERT.
• Because every position is predicted regardless, the corruption rate does not change the prediction count at all.
• BART uses twice T5’s rate and still gets 512 predictions. Raising r changes only the difficulty.

Mathematical Foundation

Ambiguity in bits, per corruption

A corruption leaves some set of originals consistent with what the model sees.

The log of that count is what the model must supply from its prior:

Subtracting the two gives the position term alone, with the identity term cancelling exactly:

which is 0.2565 of the 1202.53 bits of identity ambiguity. Deletion is strictly harder, and by a computable amount rather than a vague one.

The spread across corruptions is the striking part. Rotation costs 9.00 bits against deletion’s 1510.92, a factor of 167.9.

And permutation over 20 sentences costs 61.08 bits, only 6.7864 times rotation, so both of the order-scrambling corruptions sit far below the content-destroying ones.

But bits are not the same as unrecoverability

Rotation and permutation preserve the token multiset exactly, so no content is destroyed and the task is purely one of ordering.

Only masking preserves length while destroying content, which is why it alone permits a position-aligned target; the others force the decoder to infer the output length as well.

Exact reconstruction accuracy, masking against deletion

On a Markov chain both posteriors can be enumerated. With masking, the corrupted positions are known, so there are Kᵏ fillings.

With deletion they are not, so there are C(n, k)Kᵏ hypotheses, each scored by the chain likelihood. Reconstructing with the per-position marginal MAP maximizes expected accuracy:

Masking wins at every setting, and that ordering is forced rather than empirical. The deletion posterior is the masking posterior with the gap locations summed out:

A mixture of posteriors is no sharper than its sharpest component, so the deletion accuracy cannot exceed the masking accuracy for any data distribution.

The control that makes the enumeration credible is the chance floor.

Deletion stays above 0.333333 everywhere, and at concentration 3.0 it sits at 0.337500, only just above it, which is the regime where the chain is nearly uniform and neither corruption can be inverted.

The gap is non-monotone in concentration, at 0.075000, 0.137500, 0.075000.

At low concentration the chain is so predictable that knowing the positions adds little; at high concentration nothing helps; the cost of positional uncertainty is largest in between.

Full reconstruction removes the premium and pays for it

T5 emits corrupted spans plus sentinels, so its target is about 0.2n. BART emits everything:

BART pays 24.81× the decoder attention for 6.67× the predictions, so cost per prediction is 3.72× worse.

The extra predictions are not free, and the exchange rate is exact.

At the whole-architecture level the picture is much milder, because the encoder is identical and dominates:

So a 24.81× decoder becomes only 2.0497× overall.

Article 94’s finding that the encoder carries 88.27% of the budget is what absorbs it, and it means the choice of target length matters far less to total cost than it appears in isolation.

Parameter counts

BART uses the standard configuration, so Article 91’s 12d² encoder block and Article 94’s 16d² decoder block both apply, unlike the large T5 variants:

Both land within 1.6%. The measured block multipliers are 12.0169d² and 16.0247d² at d = 768, the excess being biases and layer norms, with a decoder-to-encoder ratio of 1.3335 against the exact 4/3 Article 94 derived.

Types of Corruption

Rotation is the weakest by a wide margin

• At 9.00 bits it asks the model to identify one of 512 starting points, against deletion’s 1510.92 bits.
• Since the multiset is preserved, no content inference is involved at all. Reporting that rotation “underperforms” as a pretraining objective is not an empirical surprise; the objective supplies 167.9 times less to invert.

Why infilling is the strongest single corruption

• Infilling combines all three unknowns: which content is missing, where the boundaries are, and how many tokens each span held. Masking supplies the position and the count.
• Deletion supplies neither but has no marker; infilling marks the location while hiding the count, and the Poisson(3) distribution means the count is unknown over an effectively unbounded range with P(0) = 0.049787 of the marks being empty.

Evaluation Metrics

Bits overstate the difference between order and content corruptions

• Rotation’s 9.00 bits and deletion’s 1510.92 differ by 167.9×, but rotation preserves the entire token multiset while deletion destroys 77 tokens permanently.
• A corruption with few bits of ambiguity can still be the one that removes recoverable information, and a corruption with many can leave everything present in scrambled order.
• The bit count measures how many hypotheses exist, not whether the right one is identifiable, and the multiset column is the check on it.

The chance floor is the control that validates the accuracy numbers

• At concentration 3.0, deletion reconstructs at 0.337500 against a floor of 0.333333.
• That near-coincidence is the signal that the enumeration is behaving: a result below chance would mean the marginal MAP was misimplemented, and a result far above it on a near-uniform chain would mean the posterior was leaking the answer.

Selecting Evaluation Metrics

Choosing a corruption

Choosing the target length

The isolated decoder comparison makes full reconstruction look expensive at 24.81×, and the whole-architecture comparison makes it look cheap at 2.0497×.

Both are correct, and the second is the one that matters for a total budget.

So the argument against full reconstruction is not cost, it is that 3.72× worse cost per prediction may buy predictions that are individually less informative, since most positions in a lightly corrupted document are trivially copied.

What carries over from earlier articles

Assumptions and Practical Considerations

On the scope of the reconstruction measurements

• The exact accuracies come from a first-order Markov chain of length 8 with 2 corrupted positions over 3 states, enumerated in full.
• They establish the ordering and the mechanism, not the magnitude on text: real language is neither first-order nor over three symbols, and the gap between masking and deletion on a real corpus is not 0.0750.
• What transfers is that the gap is forced to be non-negative by a marginalization argument, and that it is largest at intermediate predictability rather than at either extreme.

On the bit counts

• These count hypotheses under an idealized corruption with k and s known to the model.
• A real implementation reveals the number of mask tokens, which removes part of the deletion ambiguity, so 1510.92 is an upper bound on what an actual BART must resolve.
• The comparison between corruptions is the useful content; the absolute figures are ceilings.

Explore detailed T5

Introduction

Counting it properly reverses the conclusion. A decoder-only model on a concatenated stream is causal, so it pays (m+k)(m+k+1)/2 score entries, while an encoder-decoder pays a full m² on the input because the encoder is bidirectional.

At m = 512, k = 64 that is 296, 992 against 166, 176, a ratio of 1.7872, and the ratio approaches exactly 2 as the input grows.

The encoder-decoder is the more expensive architecture, and 88.27% of its attention budget goes to reading the input rather than to cross-attention.

Article 91’s rule that a transformer block costs 12d² also fails here, and not marginally.

T5–3B and T5–11B hold d(model) = 1024 fixed while scaling the attention inner width to 4d and 16d and the feed-forward width to 16d and 64d, giving true block multipliers of 48 and 192 rather than 12: understatements of exactly 4× and 16×.

Yet one ratio survives untouched. The decoder-to-encoder block ratio is exactly 1.3333 for all five variants, because T5 scaled both widths in the one proportion that keeps attention at precisely half the feed-forward.

Table of Contents

• Introduction
• Table of Contents
• What is T5
• Core Components of T5
• Mathematical Foundation
• Types of Corruption and Position Schemes
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is T5

Definition

• T5 casts every task as mapping an input string to an output string, and implements that with an encoder-decoder transformer. The encoder reads the input bidirectionally.
• The decoder generates the output causally while attending to the encoder output:
• Because targets are always text, classification and regression need no task-specific head, so the output layer has exactly zero task-dependent parameters.

Where the three attention blocks sit

• The encoder row is the one that matters. It is unmasked, so it pays a full square where Article 93’s causal prefill paid a triangle, and Section 5 shows that this single term dominates the architecture’s cost.

Core Components of T5

The second stack’s parameter cost

• An encoder block is Article 91’s 12d². A decoder block adds cross-attention, so it is 4 + 4 + 8 = 16d²:
• So matching an L-layer decoder-only model with L encoder plus L decoder layers costs exactly 1.166667 times the parameters, independent of L and of d.

Relative position bias is nearly free in parameters

• T5 discards absolute positional encodings entirely and adds a learned scalar bias to each score, indexed by a bucketed relative distance.
• The table is heads × buckets and is shared across all layers:
• Against Article 91’s sinusoidal encodings, which cost zero parameters and satisfy PE(p+k) = RₖPE(p) exactly, T5 pays a few hundred parameters.
• What it buys is translation invariance that does not have to be learned; what it gives up is measured in Section 5.

Mathematical Foundation

The corrected cost comparison

A decoder-only model on the concatenation is causal, so the naive (m+k)² overstates it by nearly half. The correct comparison is

The encoder-decoder is more expensive in every case.

The ratio approaches exactly 2 as the input grows with the output fixed, because m² against m(m+1)/2 dominates everything else:

And the budget does not go where the architecture’s name suggests. At m = 512, k = 64: So an encoder-decoder does not spend its budget on cross-attention.

It spends it on reading the input bidirectionally, which is precisely the thing being purchased, and Article 92 measured that purchase as worth between 0.026172 and 0.251591 in accuracy, or exactly zero.

When the premium is recovered

The encoder output does not depend on the output sequence, so it can be computed once and reused across many candidate outputs. Generating N alternatives for one input:

The premium is fixed and the per-candidate cost is nearly identical, so the ratio falls toward 1 as N grows. At N = 64 it is 1.0554.

That makes the architecture well suited to beam search and multi-sample decoding and poorly suited to producing one output per input.

What the relative position bucketing loses

T5 keeps small distances exact, then buckets logarithmically to a maximum distance, then clamps. With 32 buckets and a maximum distance of 128, over distances 0 to 200:

Only 8 distances get their own bucket. Everything past the maximum distance collapses into one, so over the range 0 to 2000 the final bucket holds 1910 distinct distances:

Two tokens 200 apart and two tokens 1000 apart are indistinguishable to the model by construction.

That is a deliberate trade: resolution where it matters against a fixed parameter budget and unbounded extrapolation range.

Span corruption, accounted exactly

T5 corrupts a fraction r of tokens in contiguous spans of mean length ℓ, replacing each span with one sentinel token:

At ℓ = 1 the scheme reduces to single-token masking and the target is 153.6 tokens, exactly twice the 76.8 corrupted tokens, because every corrupted token needs its own sentinel. Longer spans amortize sentinels away.

Critically, the corrupted count is 76.8 at every span length, so span corruption does not reduce Article 92’s 6.67× premium at all.

What it reduces is the decoder’s cost, since the target length enters quadratically: k²/2 falls from 11,796 entries at ℓ=1 to 3,732 at ℓ=8.

Where the 12d² rule breaks

Article 91 assumed heads partition d, so the projections are d × d, and that d(ff) = 4d. T5 violates both above T5-large:

The block cost becomes 4d(h dₖᵥ) + 2d d(ff) rather than 12d²:

Splitting T5–11B’s encoder block: attention is 4 · 1024 · 16,384 = 67,108,864 = 64d² and the feed-forward is 2 · 1024 · 65,536 = 134,217,728 = 128d², totaling 201,326,592 = 192d².

The feed-forward is 66.7% of it, so T5–11B is largely a stack of very wide feed-forward layers rather than a wider attention model.

The ratio that survives

The natural expectation is that scaling the feed-forward so aggressively would swamp the extra cross-attention and drive the decoder-to-encoder ratio toward 1.

It does not move at all:

A decoder block is an encoder block plus one more attention, so the ratio is

which is fixed whenever attention and feed-forward scale together. T5 chose h dₖᵥ = 4d with d(ff) = 16d, and h dₖᵥ = 16d with d(ff) = 64d: in both cases attention is 4 · (h dₖᵥ/d) = exactly half the feed-forward’s 2 (d(ff)/d).

The 4/3 ratio is preserved by design across a 16× change in block size.

Totals against published figures

Four of five land within 3%. T5-large is off by 4.2%, which is too large to be rounding, so the honest claim is that this accounting captures the structure of the standard variants and demonstrates that 12d² fails for the large ones, not that it reproduces every published total.

Types of Corruption and Position Schemes

Choosing the span length is choosing the decoder cost

• The corrupted-token count is rn regardless of ℓ, so the training signal by prediction count is fixed.
• What ℓ controls is how many sentinels appear, and therefore the target length, which enters the decoder’s self-attention quadratically.
• Going from ℓ = 1 to ℓ = 8 cuts decoder self-attention entries from 11, 796 to 3, 732 while leaving the number of predicted tokens untouched at 76.8.

Why the bucketing has no exact shift property

• Article 91’s sinusoidal encodings satisfy an exact rotation identity, so relative position is a fixed linear function of the encodings.
• A bucketed bias has no analogous structure: the bias for distance 91 and distance 200 is literally the same learned scalar.
• The scheme is not an approximation to the sinusoidal one; it is a different object with a different failure mode, namely deliberate blindness beyond the maximum distance.

Evaluation Metrics

The metric that misleads

• Comparing (m + k)² against m² + k² + km suggests separation saves km entries and gives a ratio of 0.75 at m = k.
• That comparison forgets the causal mask on the decoder-only baseline, and correcting it moves the ratio from 0.75 to 1.2493 at m = k and to 1.7872 at T5-style summarization shapes.
• The direction of the conclusion reverses, which is why the mask has to be in the count.

Check the block multiplier before quoting any d² rule

• The multiplier 4(h dₖᵥ/d) + 2(d(ff)/d) takes one line to compute and reduces to 12 only under the standard configuration.
• Quoting 12d² for T5–11B understates its block by 16×, so a capacity or memory estimate built on the rule would be wrong by more than an order of magnitude.

Selecting Evaluation Metrics

Deciding between the architectures

• The last two rows are the operative ones, and Article 92 supplied the test: measure the error reduction against a matched causal baseline on the actual data.
• A cyclic-chain-like dataset yields exactly zero gain, in which case the encoder-decoder pays between 1.25× and 1.99× the attention cost plus 1.166667× the parameters for no benefit at all.

What carries over from earlier articles

• Article 91’s row is the one worth noting, because that rule reproduced BERT and all four GPT-2 sizes to within a percent and then failed here by 16×.
• A rule that has held across six models is still a rule about a configuration, not about transformers.

Assumptions and Practical Considerations

On the cost figures

• These count score-matrix entries and are proportional to the attention arithmetic, not to wall-clock time.
• An encoder’s m² is fully parallel while a decoder’s k steps are sequential, so a ratio of 1.7872 in entries can correspond to a much better or much worse latency depending on batch size and hardware.
• The entry count is the right quantity for comparing architectures and the wrong one for predicting throughput.

On the parameter totals

• Four of the five variants match published figures within 3% and T5-large is 4.2% low.
• That residual is unexplained here, so the parameter tables should be read as establishing the structure and the failure of the 12d² rule rather than as an exact reproduction of every checkpoint.

Explore detailed GPT

Introduction

A causal model pays none of that, and the reason it can is a property checkable to machine precision.

Causal attention at position i is identical to running the model on the prefix x₁ … xᵢ alone, measured here at a worst residual of 8.882×10⁻¹⁶, while an unmasked model fails the same test by 2.107.

That identity is what makes teacher forcing valid: all n predictions come from one pass because each one already equals its own inference-time computation.

The costs are equally countable. A causal mask discards (n-1)/(2n) of the score matrix, which at n=512, d=768 is 12.48% of attention’s arithmetic and at n=32768 is 47.76%.

Generating n tokens without a key-value cache is O(n³d) against O(n²d) with one, a measured speedup of 555.16× at n=1024.

And the standard claim that autoregressive errors compound without bound turns out to be false for a mixing model.

On a Markov chain the drift saturates at exactly TV(π, π̂), agreeing to ten decimal places, so exposure bias has a computable ceiling rather than an unbounded trajectory.

Table of Contents

• Introduction
• Table of Contents
• What is GPT
• Core Components of GPT
• Mathematical Foundation
• Types of Decoding Strategies
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is GPT

Definition

• GPT is a transformer decoder stack in which attention is restricted to earlier positions, so the model factorizes the sequence probability by the chain rule and is trained to predict every token from its prefix:
• The restriction is applied by setting the score to -∞ wherever j > i before the SoftMax, which makes those positions receive exactly zero weight.

What the restriction buys and costs

• The final two rows are the same fact.
• A model can generate exactly when its training-time computation at each position equals its inference-time computation, and Section 5 measures that as an identity rather than asserting it.

Core Components of GPT

What the mask discards, exactly

The mask keeps entry (i,j) only for j ≤ i, so it keeps n(n+1)/2 of n² entries:

• The kept fraction approaches 1/2 from above, never below. Combining with Article 91’s quadratic share n/(2d+n) gives the discarded work as a fraction of all attention cost:
• So at d = 768 and n = 512 a naive implementation that computes the full matrix and masks it wastes 12.48% of attention arithmetic, which is real but not fatal.
• At n = 32768 it wastes 47.76%, and the waste is nearly all of the quadratic term.

Mathematical Foundation

Prefix consistency, and the control that must fail

Teacher forcing computes all n predictions from one pass over the full sequence.

That is valid only if the output at position i does not depend on positions after i, which is checkable directly by comparing the full-sequence output against the prefix-only output:

The causal residual is at machine precision everywhere; the unmasked residual reaches 2.107.

The positive control is what makes this measurement worth anything: an unmasked model fails at every position except the last, where the full sequence is its own prefix and the residual is exactly zero.

That final row is the reason a masked encoder cannot generate, stated as an identity rather than an intuition.

Key-value caching, exactly

Generating one token at a time without a cache recomputes the whole prefix at every step, so step t costs 4td² + 2t²d. Summing gives a cubic:

With a cache, step t projects one token and attends over t stored keys, costing 4d² + 2td:

The speedup grows without bound because the exponents differ, and the price is memory linear in n: the cache holds 2ndL floats, which is 75.50 MB at n = 1024, L = 12 in fp32 and 1, 207.96 MB at n = 4096, L = 48.

At d = 768, L = 12 the cache equals the block parameter count of 84,934,656 at exactly n = 4,608.

Exposure bias has a closed-form ceiling

Teacher forcing conditions on true prefixes while generation conditions on the model’s own output, and on a Markov chain the consequence is exactly computable.

With true transitions T and learned T̂, the one-step error is maxₐ TV(Tₐ, T̂ₐ) and does not depend on t, while the generated marginal drifts:

The drift grows, slightly overshoots, and then flattens. It does not grow without bound, and the ceiling is not approximate:

The saturation value is exactly the total variation between the two stationary distributions, agreeing to ten decimal places, because both chains forget their starting point at their own mixing rate.

So for an ergodic model the compounding stops, and the often-repeated claim that autoregressive errors accumulate indefinitely does not hold. Two details are worth keeping honest.

The drift is non-monotone, peaking at 0.012210 for δ = 0.02 before settling to 0.012092.

And the drift at t=1 is below the one-step TV, 0.008760 against 0.014806, because the one-step figure is a maximum over rows while the drift averages over π.

Temperature and truncation

Temperature rescales logits, pᵢ ∝ exp(zᵢ/τ). On a synthetic logit vector over a 50,257-token vocabulary, where log V = 10.824905 is the entropy ceiling:

Halving the temperature from 1.00 to 0.50 cut the tokens needed for 90% of the mass from 11, 895 to 259, a factor of 45.9. That is the quantitative content of “temperature controls diversity.”

Top-k retains a mass that depends on how peaked the step is, from 0.016289 at k=1 to 0.476944 at k=1000 here, while top-p fixes the mass and lets k vary.

These figures describe a synthetic distribution, not a trained model, and are meant to show the mechanism rather than to characterize GPT’s output.

Parameter counts across the GPT-2 family

All four reproduce from Article 91’s 12d² block plus tied embeddings.

The embedding share collapses with scale, from 31.65% to 5.27%, while blocks rise from 68.35% to 94.73%, and the quadratic share at the full 1024-token context falls from 0.4000 to 0.2424 as d grows.

Types of Decoding Strategies

Why top-p exists

• The tables above make the difference concrete. At k=40 the retained mass on this distribution is 0.116046.
• On a sharply peaked step the same k would retain nearly everything. Top-k therefore applies a fixed cut to a varying distribution, while top-p applies a fixed mass and lets the cut move, which needed k = 11,895 here for 0.90.

Why beam search suits translation better than open generation

• Beam search maximizes total sequence log-probability. For a task with one roughly correct output that is what is wanted.
• For open generation the highest-probability continuation is systematically bland, because probability mass concentrates on generic completions.
• That is a property of the objective, not a defect of the search.

Evaluation Metrics

Perplexity measures the wrong regime

• Teacher-forced perplexity conditions every prediction on a true prefix, which is the one situation that never occurs at generation time.
• Section 5 quantified the gap: the one-step error was 0.014806 while the free-running drift settled at 0.012092, and the two are different quantities computed under different conditioning.
• A model can have identical teacher-forced loss and very different generated distributions, since the drift depends on T̂ through its stationary distribution rather than through its rows individually.

The one diagnostic that should be run once

• The prefix-consistency residual is cheap and catches a whole class of bugs: an off-by-one in the mask, a bidirectional layer left in the stack, or a normalization that mixes across positions.
• A residual of 2.107 rather than 10⁻¹⁶ means the model cannot be run forward at all, and no amount of training will fix it.

Selecting Evaluation Metrics

Reading the diagnostics together

• The third row matters because the loss looks fine. A model that leaks future information trains to a low teacher-forced loss and cannot generate, so the loss curve gives no warning at all and only the residual check does.

Deciding whether drift is worth attacking

• Since the drift saturates at TV(π,π̂), that ceiling is the right target rather than the per-step error.
• At δ = 0.10 the per-step error was 0.074030 while the ceiling was 0.056805, so reducing per-step error by a given factor does not reduce the ceiling by the same factor.
• The ceiling is a property of the stationary mismatch, and methods aimed at per-step accuracy address it only indirectly.

What carries over from earlier articles

Assumptions and Practical Considerations

On the scope of the drift result

• The saturation is exact for a finite ergodic Markov chain, where both the true and learned chains mix to unique stationary distributions.
• A model with absorbing states, or one whose learned chain is not ergodic, has no such ceiling, and a long-range dependency structure is not first-order to begin with.
• What transfers is the structural point: compounding error is bounded by a mixing property, so “errors accumulate” is a claim requiring a non-mixing model rather than a general truth.
• The non-monotonicity, with an overshoot to 0.012210 above a limit of 0.012092, is a further sign that the trajectory is not the simple exponential the usual argument assumes.

On the sampling figures

• The temperature and truncation numbers come from a synthetic Gaussian logit vector, so they demonstrate the mechanism and the exact relationships but are not measurements of any trained model.
• A real model’s logits are far more peaked, so the tokens-for-90% figures would all be much smaller.

Explore detailed BERT

Introduction

At n = 512 with a mask rate of 0.15, a masked objective yields 76.8 training predictions per sequence against an autoregressive model’s 512, so reaching the same number of prediction events costs exactly 6.67 times more passes over the corpus.

The naive optimum is not 0.15 either: maximizing predictions times context, p(1-p)n², peaks at p = 0.50, so the product is evidently the wrong criterion.

What the premium buys can be computed exactly rather than asserted. On a stationary Markov chain both the left-only and the bidirectional Bayes accuracies have closed forms, and across nine random chains the gain ran from 0.026172 to 0.251591.

But a noisy cyclic shift gave a gain of exactly zero at every noise level, despite carrying 0.239605 nats of conditional mutual information and changing the predicted token on 35.0% of contexts.

Information gain, prediction changes, and accuracy gain are three different quantities, and only the third is what bidirectionality is bought for.

Table of Contents

• Introduction
• Table of Contents
• What is BERT
• Core Components of BERT
• Mathematical Foundation
• Types of BERT Objectives and Variants
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is BERT

Definition

• BERT is a transformer encoder stack trained without a causal mask, so every position attends to the entire sequence.
• Since an unmasked model can see the token it would be asked to predict, the next-token objective is unusable, and BERT replaces it by corrupting a fraction p of positions and predicting them from the rest:

What changes relative to an autoregressive stack

• Nothing in Article 91’s parameter or cost accounting changes: the block is still exactly 12d², and the quadratic share is still n/(2d+n).
• Only the objective differs, and every consequence below follows from that.

Core Components of BERT

The 80/10/10 corruption, accounted exactly

• Of the 15% selected, 80% become [MASK], 10% become a random token, and 10% are left unchanged:
• At fine-tuning time [MASK] never appears, so the input distribution shifts by exactly the 12.0% of positions that carried it.
• The remaining 3.0% exists to break a shortcut: on those positions the model must predict from an uncorrupted or wrongly corrupted input, so it cannot use the presence of the marker as the cue to do work.

Next sentence prediction is small by construction

• At n = 512 and p = 0.15, one sequence yields 76.8 token predictions and exactly one binary sentence label:
• So NSP is 1.2853% of the training signal by prediction count. That is the quantitative form of the later finding that removing it costs very little, and it was computable before running the ablation.

Mathematical Foundation

An exact bound on what bidirectional context is worth

Take a stationary Markov chain over K states with transition matrix T and stationary distribution π. Predicting xᵢ from the left neighbor alone, the Bayes-optimal accuracy is

Given both neighbors, the rest of the chain is conditionally independent, so the posterior is a product of the forward and backward factors:

Both are closed form. No training, no seeds, no estimation error.

The gain is never negative, which is forced: conditioning on more variables cannot lower the Bayes accuracy. Two exact controls confirm the computation rather than the claim.

For an i.i.d. chain, where identical rows make neighbors uninformative, both accuracies equal maxⱼ πⱼ = 0.4177311877 with a gap of 1.110×10⁻¹⁶; any nonzero gap there would mean the bidirectional code was cheating.

The result that contradicted the expectation

The second control was meant to be the opposite extreme. A noisy cyclic shift, T(a,a+1) = 1-ε with the rest spread uniformly, is not reversible, so the right neighbor should have carried something the left one lacked. It did not:

The gain is exactly zero while the conditional mutual information is large. The right neighbor supplies up to 0.392908 nats about xᵢ beyond the left neighbor and converts none of it into accuracy.

Neither information nor prediction changes explain the gain

The natural repair is to bound the gain by that mutual information. The bound holds, with zero violations across all twelve chains, but it is loose enough to be useless as a predictor:

The second repair is to count how often the right neighbor moves the predicted token, that is, how often arg maxⱼ TₐⱼT(jc) ≠ arg maxⱼ Tₐⱼ. That fails too, and more sharply:

The cyclic chain changes the predicted token on 35.0% of contexts and gains exactly nothing, because the symmetry of the chain makes those changes correct exactly as often as they are wrong: the gains and the losses cancel to machine precision.

So a bidirectional model can differ from a causal one on a third of its predictions while being no better, and measuring disagreement between the two tells you nothing about which is more accurate.

Verifying the closed forms with an estimator

Analytic optima are worth only as much as their agreement with something fitted. Simulating 400,000 steps and using plug-in argmax predictors over the empirical conditional tables:

Worst deviation 0.001804, so the closed forms are the right quantities and the gain is a property of the data rather than a modeling artifact.

Parameter counts, from Article 91’s accounting

BERT-large is 3.06× base in total parameters but 3.55× in the block stack alone, since embeddings scale with vocabulary rather than depth.

And at BERT’s full 512-token context the quadratic share is 512/(2d+512), giving 0.2500 for base and 0.2000 for large: even at maximum context the projections dominate.

Types of BERT Objectives and Variants

Two variants attack two different halves

• The parameter table above makes the split explicit. In BERT-base, embeddings are 21.77% and blocks 77.69%.
• In BERT-large the same figures are 9.48% and 90.20%.
• A method that factorizes the embedding matrix therefore has four times the headroom on base than on large, while layer sharing or distillation has more on large.
• These are the addressable fractions, and they bound what either approach can save before any of it is implemented.

ELECTRA addresses the 6.67× directly

• Section 5’s premium came from predicting at only pn positions. Replaced-token detection asks a binary question at every position instead, restoring n predictions per sequence and removing the factor entirely.
• That is the one variant aimed at the sample-efficiency term rather than at the parameter count.

Evaluation Metrics

The disagreement metric is the trap

• It is tempting to validate a bidirectional model by showing it predicts differently from a causal one.
• Section 5 measured the exact counterexample: the cyclic chain disagrees on 35.0% of contexts with a gain of 0.000000.
• Disagreement rate and accuracy gain are unrelated quantities, and only a matched-baseline comparison separates them.

Why raw MLM accuracy will not compare across mask rates

• Changing p changes both the number of predictions and the context each one gets, so accuracy at p = 0.15 and p = 0.50 are not measuring the same task.
• The error-reduction column in Section 5 is the comparable quantity, because it normalizes by the headroom the baseline left: at K = 3, c = 0.3 a gain of 0.178836 is a 0.446025 error reduction, while at K = 10, c = 0.3 a larger gain of 0.251591 is a nearly identical 0.445021 reduction.

Selecting Evaluation Metrics

Deciding whether bidirectionality is worth the premium

• The last row is the operative one. A dataset can look like it needs bidirectional context, in the sense that the right neighbor carries real information, and still yield no accuracy at all.

The only defensible test is a matched causal baseline on the same data.

What carries over from earlier articles

• Article 82 found aggregate coverage of 0.9488 against an exact 0.9491 while pointwise variances were wrong by factors of 0.70 to 1.38.
• An aggregate can be exactly right while the quantities composing it are individually wrong in offsetting directions, so matching on an aggregate is never evidence that the parts match.

Assumptions and Practical Considerations

On the scope of the Section 5 results

• The closed forms are exact for a first-order stationary Markov chain, where two neighbors screen off the rest of the sequence.
• Natural language is neither first-order nor stationary, so the numbers are not estimates of what BERT gains on text.
• What transfers is the structural conclusion, which does not depend on the order: conditioning on more context cannot hurt, can be worth a great deal, and can be worth exactly nothing while still changing a third of the predictions.
• The cyclic chain is a proof that the third case exists.

On the plug-in verification

• The estimator matched the analytic optimum to 0.001804 across four chains, which validates the closed forms but not any claim about neural estimators.
• A trained model has approximation and optimization error on top of the statistical error measured here, so these accuracies are ceilings.

Explore detailed Transformers

Introduction

Attention removes the bound by comparing every position with every other in one step, and both the benefit and the cost are exactly countable.

Relating positions n apart needs ⌈ (n-1)/(k-1)⌉ convolution layers, which is 2,048 layers at n = 4096 with k=3, against exactly one attention layer at any length.

That capability comes with an exact requirement and an exact price. Attention is permutation-equivariant to machine precision, measured at 1.332×10⁻¹⁵, so without positional encoding the architecture cannot represent order at all.

It is a set function. Sinusoidal encodings satisfy PE(p+k) = Rₖ PE(p) for an orthogonal Rₖ depending on k alone, verified to 2.142×10⁻¹⁴, which is why relative position is recoverable by a linear map.

The 1/√d factor exists because Var(q· k) = dσ⁴: without it, attention entropy on length-16 sequences fell monotonically from 1.306609 to 0.064045 as d went 8 to 2048, against a maximum of log 16 = 2.772589.

And the parameter count is exactly 12d² per block, which reproduces BERT-base at 109,482,240 and GPT-2 small at 124,439,808.

Table of Contents

• Introduction
• Table of Contents
• What is a Transformer
• Core Components of a Transformer
• Mathematical Foundation
• Types of Attention
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is a Transformer

Definition

• A transformer replaces recurrence and convolution with scaled dot-product attention, in which every position forms a query and compares it against every key:
• Every pair of positions is related in a single operation, so there is no receptive field to grow layer by layer.

What that replaces

• The recurrence row is Article 68’s result: a signal crossing n steps is multiplied by n Jacobians and decays as ρⁿ.
• The convolution row is Article 90’s. Attention’s column of ones is the property being purchased, and Section 5 counts what it costs.

Core Components of a Transformer

Attention cannot see order

• Permuting the input rows permutes Q, K and V identically, so the score matrix is conjugated by the permutation and the output is permuted:

• And the consequence for an order-invariant readout is that two permutations of the same sequence are indistinguishable: mean-pooled outputs differed by 1.110×10⁻¹⁶.
• Positional encoding is not an enhancement. Without it the architecture is a set function and order is unrepresentable.

Mathematical Foundation

Sinusoidal encodings shift by an orthogonal rotation

With ωᵢ = 10000(-2i/d) and PE(p)₂ᵢ = sin(ωᵢ p), PE(p)₂ᵢ₊₁ = cos(ωᵢ p), angle addition gives for each frequency pair

so PE(p+k) = Rₖ PE(p) with Rₖ block-diagonal and dependent on k alone:

Worst residual over all pairs tested: 2.142×10⁻¹⁴. Two further consequences are exact:

At d = 64 the dot products were 30.916832 at k=1, 21.051629 at k=10 and 15.673797 at k=50, identical across four widely separated base positions.

So relative position is available as a fixed linear function of the encodings, which is what allows a learned projection to extract it. A learned positional embedding table has no such guarantee.

Why the 1/√d factor is necessary

For q, k with independent zero-mean entries of variance σ², the dot product is a sum of d independent terms:

Entropies are averaged over 2000 independent sequences, and the averaging is not cosmetic.

Over 400 seeds, a single-draw estimate is close to unbiased but badly dispersed, and the dispersion grows relative to the gap it must resolve:

So one draw ranks the 512 → 2048 pair backwards 14.75% of the time, and every adjacent pair is invertible by some seed pair.

The bias is small, with a single-draw mean of 1.296976 at d=8 against the 2000-sequence value 1.306609; the problem is variance. The overall trend is robust regardless, with P(H₈ < H₂₀₄₈) = 0.0000 over 400 paired draws.

Maximum entropy for a length-16 sequence is log 16 = 2.772589.

Unscaled, attention collapses toward a hard argmax as width grows, monotonically from 1.306609 at d = 8 to 0.064045 at d = 2048, and a saturated softmax has vanishing gradient by the same mechanism Article 83 measured for the discriminator.

The decay rate is itself informative. Each fourfold increase in d roughly halves the entropy, with successive ratios 2.171, 2.168, 2.066 and 2.098, so

which is what the variance result predicts: the softmax sharpens in proportion to the spread it is given.

Scaled, the entropy is essentially flat, ranging only 2.355421 to 2.370321 across a 256-fold change in width, a spread of 0.014900, and staying above 84.95% of the maximum at every width.

A block has exactly 12d² parameters

Multi-head attention uses four d× d projections regardless of head count, since heads partition d rather than adding to it. The feed-forward block is d → 4d → d:

Reproducing published totals from that accounting alone:

Both land within a percent, so 12d² per block plus embeddings is the whole story. Note the two models have identical block parameters at 85,054,464 and differ only in vocabulary and context length.

The cost crossover against convolution is exact

Counting multiply-accumulates per layer: attention projections cost 4nd², scores and mixing cost 2n²d, and a k-wide convolution costs nkd². So

Below 1,152 tokens at d = 768, attention’s quadratic term costs less than a single 3-wide convolution layer.

The quadratic cost is not intrinsically prohibitive; it becomes so only at long context.

Within attention itself the quadratic share has a closed form, 2n²d/(4nd² + 2n²d) = n/(2d+n), so it passes one half at exactly n = 2d:

At d = 768 and n = 512 the quadratic term is 25% of attention’s own cost and the projections dominate. It reaches half at n = 1536 and 95.5% at 32,768, which is why long-context work targets that term specifically and leaves the projections alone.

Layers needed to connect two positions

The path length matters for gradients as well as expressiveness: Article 68 measured that a signal crossing L multiplicative stages decays geometrically, and here L is 2,048 against 1.

Types of Attention

Local attention reintroduces exactly what attention removed

• A window of w gives a receptive field that grows layer by layer, so relating positions n apart needs ⌈ (n-1)/(w-1)⌉ layers, the same recursion as Article 90’s convolution.
• That is a coherent trade rather than a defect, but it means local attention inherits the blindness result: block permutations above the effective field are invisible at that layer.

Flash attention changes the constant, not the exponent

• It computes the same function with the same O(n²d) arithmetic, avoiding the O(n²) materialisation of the score matrix.
• So it moves the memory bound without moving the crossover in Section 5, and reports of it “solving” quadratic attention conflate memory with arithmetic.

Evaluation Metrics

The two cheapest diagnostics are exact formulas

• The quadratic share needs no profiler: at d and n it is n/(2d+n), so a model at n = 512, d = 768 spends 25% of its attention cost on the quadratic term and no optimisation of that term can save more than a quarter.
• And the logit variance should be O(1) after scaling; measuring O(d) means the scaling is missing or misapplied, with the entropy consequence in Section 5.

A permutation sensitivity check that should be run once

• Section 4 established that attention without positional information is exactly permutation-equivariant.
• So permuting the input and measuring the output difference tests whether positional information is reaching the model at all.
• A residual near 10⁻¹⁵ means it is not, and that is a wiring bug rather than a training problem.

Selecting Evaluation Metrics

Reading the diagnostics together

• The third row is the one worth care. Low entropy is a symptom of saturation only when the logit variance is also large.
• A well-scaled model that has learned to attend sharply is behaving correctly.
• The diagnostic is the variance, not the entropy, because the variance distinguishes a scaling bug from a learned preference.

On the quadratic cost as a design consideration

• Section 5 makes the tradeoff quantitative rather than rhetorical.
• At n ≤ 2d the projections dominate, so a sparse-attention variant that leaves 4nd² untouched can save at most n/(2d+n) of the attention cost, which is under half.
• At n = 32,768 the same variant addresses 95.5%. The value of any long-context method is bounded by that fraction at the target length, and it is computable before implementing anything.

What carries over from earlier articles

• The last row is worth noting: Article 81 wanted permutation equivariance because node ordering was arbitrary, and here the same property is an obstacle because sequence order is meaningful.
• The mathematics is identical; only the desirability differs.

Assumptions and Practical Considerations

On the feed-forward multiplier

• The 12d² accounting assumes d(ff) = 4d. At d(ff) = 8d the block is 20d² and at 2d it is 8d², so a model described only by d and depth is underspecified by a factor approaching two.
• The published BERT and GPT-2 totals reproduce exactly because both use 4d.

On what these results cover

• The equivariance, rotation and variance results are algebraic and hold at any size.
• The parameter accounting is exact for the standard block and was checked against two published models.
• The cost figures count multiply-accumulates only and ignore memory traffic, softmax and normalization arithmetic, which is why Flash attention improves wall-clock without changing the crossover.
• The crossover is a statement about arithmetic, not about time.