Data Science & Machine Learning — Article 99
Vision Transformer: A Comprehensive Guide
Explore detailed Vision Transformer
Introduction
Every architecture in this series has assumed a discrete vocabulary, and an image has none.
Tokens must therefore be invented, and the patch size that invents them fixes everything downstream: n = (S/P)² + 1, so at 224 pixels with P = 16 there are 197 tokens.
Three exact identities follow. The patch embedding is a strided convolution, not an analogy for one, verified at 3.553×10⁻¹⁵.
At ViT-Base’s P = 16 with d = 768 the map is exactly square, since 3 × 1⁶² = 768, so patchification is a change of basis that discards nothing, while P=32 compresses 4× and loses at least 2304 dimensions per patch by rank alone.
And without position embeddings a ViT is exactly permutation-invariant over patches, measured at 1.776×10⁻¹⁵: a shuffled image is literally the same input, with 196 patches collapsing 1214.8 bits of arrangement.
Table of Contents
- Introduction
- Table of Contents
- What is a Vision Transformer
- Core Components of a Vision Transformer
- Mathematical Foundation
- Types of Patch and Position Schemes
- Evaluation Metrics
- Selecting Evaluation Metrics
- Assumptions and Practical Considerations
What is a Vision Transformer
Definition
- A Vision Transformer cuts an image into non-overlapping P × P patches, maps each through one shared linear layer, adds a position embedding, and applies a standard transformer encoder:
- Everything after E is Article 91’s architecture unchanged, so the 12d² block and the n/(2d+n) cost share both apply directly.
What changes relative to a language model
- The fourth row is the substantive one.
- In a language model n is a property of the input; here it is a hyperparameter, and Section 5 shows it to be the most consequential one in the architecture.
Core Components of a Vision Transformer
The patch size sets the sequence length and the cost
- At P = 4 the quadratic term is 67.13% of attention cost, so the sequence-length problem that Article 91 located at n = 2d arrives through a choice of patch size rather than through a long document.
Mathematical Foundation
The P⁻⁴ scaling, and where it is not exact
Since n ≈ (S/P)², the quadratic term 2n²d scales as P⁻⁴:
The measured ratios approach 16 without reaching it, and the entire shortfall is the single class token: removing it recovers 16.0000 exactly at every step.
The power law is exact in the patch count and only asymptotic in n.
Per-layer cost at d = 768, using Article 91’s 4nd² + 2n²d:
Halving the patch size from 16 to 8 costs 5.3368×, not 16×, because the projection term is only linear in n and still dominates at these lengths.
The patch embedding is a strided convolution
Cutting into non-overlapping patches, flattening each, and applying one shared linear map is identical to a convolution with kernel P and stride P. This is an identity, checkable directly:
The identity is that flattening a patch and applying a shared matrix is the same contraction as a convolution evaluated at a stride-P offset:
So a ViT’s first layer is a convolution, and the only one in the model.
The consequence is exact: a stride-P kernel-P convolution has receptive field exactly P with no overlap, so within-patch structure is seen once by a linear map and no convolution ever crosses a patch boundary.
At ViT-Base the patch embedding is exactly square
Whether patchification can be lossless depends entirely on 3P² against d:
The rank of the map is bounded by the smaller of its two dimensions, which decides the question outright:
At ViT-Base/16 the numbers coincide: 3 × 1⁶² = 768 = d. The patch embedding is a square matrix, so patchification is a change of basis that discards nothing in principle.
Get Arko Sengupta’s stories in your inbox
Join Medium for free to get updates from this writer.
Solving for the original patch recovers it to 3.972×10⁻¹³ at a condition number of 7.6398×1⁰³.
At P = 32 the map is 3072 → 768, discarding at least 2304 dimensions per patch by rank alone and retaining 0.2500. At P = 8 it is 192 → 768, an expansion whose rank is at most 192, or 25.0000% of the embedding space.
That is Article 99’s rank argument again, now on the input side of a vision model rather than a vocabulary.
Without position embeddings a ViT is a bag of patches
Article 91 measured attention as permutation-equivariant to 1.332×10⁻¹⁵, and for images the consequence is sharper because patches carry no intrinsic order.
A shuffled image is literally the same input, and the positive control confirms the measurement means something: with position embeddings the same quantity reaches 5.380.
Article 95 priced permutation at log₂ s! bits, and the same formula applies:
At P = 16 that is 1214.8 bits, larger than any corruption measured in Article 95, whose maximum was 1510.92 bits for deleting 77 tokens.
Layers needed to relate opposite corners
The two counts come from different growth laws, one linear in depth and one geometric:
The stride-1 figure of 112 layers is not a fair comparison, and it is worth saying so.
Real convolutional networks downsample, and with a stride-2 reduction every second layer the receptive field grows geometrically, spanning 224 pixels in 12 layers.
So the honest statement is 1 against roughly a dozen. What remains exactly true is that attention needs one layer at any image size, while any convolutional stack needs a count that grows with the image.
The embedding problem nearly vanishes
All three land within 1.2%, the closest agreement in this series. And the embedding share collapses:
Articles 98 and 99’s compression problem essentially does not exist here. There is no vocabulary to factorize, and the ceiling on embedding-side savings is under one percent rather than 21.77%.
What remains is the block share at 99.13% to 99.83%, where layer removal and cross-layer sharing apply unchanged.
Types of Patch and Position Schemes
Learned position embeddings tie the model to one resolution
- The table is n × d, so changing S or P changes n and invalidates it.
- Interpolating the table is the standard remedy and it is an approximation, whereas Article 91’s sinusoidal encodings satisfy PE(p+k) = RₖPE(p) exactly at any length.
- The choice is between a table that must be resized and a function that need not be.
Overlapping patches break the identity
- Section 5’s equivalence holds because stride equals kernel.
- With overlap the first layer is still a convolution but no longer a partition, so token count rises above (S/P)² and the clean P⁻⁴ accounting no longer applies.
Evaluation Metrics
The permutation check is the same diagnostic as Article 91's
- A residual near 10⁻¹⁵ means position embeddings are not reaching the model, and the consequence for images is that the model sees a set.
- The positive control matters equally: with embeddings present the same quantity reached 5.380, so a small residual is informative only against a large one.
Report the patch size before any cost figure
- Since P determines n and n determines the quadratic share, a cost or latency claim about a ViT is meaningless without it.
- The same model at P = 32 and P = 8 differs by 22.98× in per-layer arithmetic, which is a larger spread than most architectural changes in this series produce.
Selecting Evaluation Metrics
Choosing the patch size
What this series has established about where costs are exact
Two of this series’ recurring lessons apply directly
- First, an exact ceiling is not an achievement: the 99.83% block share of ViT-Huge bounds what compression can save without indicating what it costs in accuracy, exactly as Article 98’s 77.69% did.
- Second, a rule verified across many models is still a rule about a configuration.
- Article 91’s 12d² reproduced BERT, all four GPT-2 sizes, BART and every ViT here to within 1.2%, and failed by 16× for T5–11B in Article 94.
Assumptions and Practical Considerations
On the convolution identity
- The equivalence is exact for non-overlapping patches with a shared linear map and holds at machine precision across the configurations tested.
- It says nothing about whether a ViT behaves like a CNN overall: after the first layer the two differ entirely, and the identity concerns only patchification.
On the corner-distance comparison
- The 112-layer figure assumes stride-1 convolutions throughout, which no practical network uses, and the strided calculation gives 12.
- The article reports both because the first alone would overstate the case. What is exactly true is the scaling: one attention layer at any image size against a convolutional count that grows with it.
