Vision Transformer, Part 1 — Embeddings: From Pixels to a Token Sequence

Abstract. A Vision Transformer doesn’t process an image the way a CNN does — it first turns the picture into a sequence of tokens, the same kind of input a language model reads, and only then applies a standard transformer. This opening part builds that translation from the ground up: how a 4×4 patch of pixels becomes a 128-dimensional vector through a single learned projection, how a learnable [CLS] token and positional embeddings join the sequence, and how the model — starting from pure noise — discovers the image’s 2-D layout entirely on its own. By the end you’ll have the exact [65, 128] token sequence the rest of the network operates on, with every number traced to a ViT trained from scratch on CIFAR-10. No prior transformer knowledge assumed.

In this 6-part series of Vision Transformer:

1. Part 1 — Embeddings: from pixels to a token sequence — you are here
2. Part 2 — Attention: how tokens relate to each other (single head)
3. Part 3 — The full block: multi-head, LayerNorm, residuals, MLP, and the prediction
4. Part 4 — Building the ViT in PyTorch, trained on CIFAR-10
5. Part 5 — Do overlapping patches help? A controlled experiment
6. Part 6 — How it learns: loss, backprop, the chain rule, the weight update

How a patch becomes a token

Attention operates on tokens. So where do tokens come from? A patch of an image is not a vector — it’s a little square of coloured pixels. Turning one into the other is two moves, and neither is mysterious.

1 · A 4×4 RGB patch is just 48 numbers

Figure1: Image of size 32x32 converted into 48 patches, each patch size is 4x4 RGB.

A 4×4 patch has 16 pixels. Each pixel has 3 colour channels. So the patch is 16 × 3 = 48 numbers, and we simply lay them out in a row:

patch [3, 4, 4]  ──flatten──▶  vector [48]

The values are negative as well as positive because the image has been normalised (mean-subtracted). Nothing clever has happened yet — we’ve only rearranged pixels. No information gained, none lost.

2 · 48 numbers × a learned matrix = one 128-dim token

Figure 2: Transformation of 48 values of patch embedding into 128 values embedding

Now the learning. We keep a matrix W of shape [48 × 128] — 128 columns, each one a learned 48-long weight vector — and multiply:

token[k]  =  Σ_i  patch[i] × W[i, k]  +  bias[k]        for k = 0 … 127

Look closely at that formula: each output dimension is a dot product — the the same dot product we unpack in Part 2 (figure 1), between the patch and one column of W. Do it for all 128 columns and the patch has become a 128-dim token.

That’s 48 × 128 = 6,144 weights plus 128 biases = the 6,272 parameters of the patch embedding. In code it's a single Conv2d(3, 128, kernel_size=4, stride=4) — a convolution whose kernel exactly covers one patch is this matrix multiply. (The figure was produced by pulling those conv weights out of the trained model and doing the matmul by hand; it matches the model's own output to 4 decimal places.)

Get Chandrashekhar Padole’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Why go from 48 up to 128? Not to add information — you can’t; the pixels are the pixels. It’s to re-describe them in a learned space where useful properties (edges, textures, colour blobs) become directly readable, rather than tangled up in raw intensities. It’s the visual analogue of an LLM’s token embedding, except the vector is computed from pixels rather than looked up by ID.

3 · Assembling the sequence: patches + [CLS] + positions

Figure 3: Adding CLS token and Poisitional Embeddings

(Each token is drawn as a vertical column of 128 values — same orientation as figure 2. Token index runs left→right, embedding dimension top→bottom.)

Run 0a–0b on all 64 patches and you have a [64, 128] block. Two things are added before the encoder sees it:

1. The [CLS] token. A single learnable 128-dim vector — not computed from any patch — is prepended, giving [65, 128]. It belongs to no particular location, which is exactly the point: it exists to accumulate a summary of the whole image, and at the very end its final state is what the classifier reads.

2. Positional embeddings. A learned table of shape [65, 128] — one 128-dim vector per position — is added element-wise:

sequence = [CLS ; patch₀ … patch₆₃]  +  positional_embeddings
  [65,128]         [65,128]                  [65,128]

Note it is added, not concatenated. Concatenating would grow the width to 256 and cost parameters everywhere downstream; addition keeps it at 128 and lets the model learn to encode “what” and “where” in the same 128 numbers. After this step, a token vector means “this content, at this place.”

Why is this needed at all? Because of the order-blindness we prove in Part 2 (figure 7) — attention has no built-in notion of position, so we bake it into the token itself.

4 · The model really does learn 2-D layout

Here’s the part I find genuinely satisfying. Those positional embeddings start as random noise and are learned by backprop. The model is never told the patches form an 8×8 grid — it only ever sees a flat list of 65 tokens.

So did it figure out the geometry?

Figure 4: Spatial Structure Learning

Taking each position’s learned embedding and measuring cosine similarity against all the others — then laying the result back out on the 8×8 grid — every position is most similar to itself and its immediate neighbours, fading with distance. Measured across all pairs:

Table 1: Similarity fading with distance

A clean monotonic decay. The 2-D spatial structure of the image was learned from data alone, purely because it helped classify — nothing in the architecture encodes it.

That’s also a neat sanity check on the whole idea: if positional embeddings were useless, they’d have stayed noise.

5 · The whole stage in one picture

Figure 5: Summarized View: Image to Tokens

image [3,32,32]
  → 64 patches, flattened      [64,  48]
  → × W (48→128) + bias        [64, 128]     ← learned  6,144 + 128
  → prepend [CLS]              [65, 128]     ← learned  128
  → + positional embeddings    [65, 128]     ← learned  8,320
  → encoder

Two things worth noticing in the summary:

• Only three things are learned in this entire stage — the projection W, the single [CLS] vector, and the positional table. Everything else is reshaping: cutting the image into patches and flattening them moves numbers around without changing them.
• It’s a tiny fraction of the model. 6,144 + 128 + 128 + 8,320 = 14,720 parameters — roughly 1% of the ViT's 1.2M. The other 99% is the encoder. Turning pixels into tokens is cheap; relating the tokens to each other is where the model actually spends its capacity.

Next

We now have the [65, 128] sequence — 65 tokens, each a 128-dim vector carrying what it is and where it sits. In Part 2 we build attention from first principles: the one operation that lets these tokens finally share information.