Vision Transformer, Part 5 — What Overlapping Patches Actually Do

Abstract. A controlled experiment: what happens if the image patches overlap instead of tiling the picture edge-to-edge? This part runs the sweep from 0% to 75% overlap — and tells the story of a hidden confound (an out-of-necessity batch-size change) that nearly produced the exact opposite conclusion, before de-confounding it into a clean, honest result. A small case study in doing empirical machine learning carefully, with real numbers throughout.

In this 6-part series of Vision Transformer:

1. Part 1 — Embeddings: from pixels to a token sequence
2. Part 2 — Attention: how tokens relate to each other (single head)
3. Part 3 — The full block: multi-head, LayerNorm, residuals, MLP, and the prediction
4. Part 4 — Building the ViT in PyTorch, trained on CIFAR-10
5. Part 5 — Do overlapping patches help? A controlled experiment — you are here
6. Part 6 — How it learns: loss, backprop, the chain rule, the weight update

Part 5 of the series. Part 4 built the ViT from scratch and measured it: ~1.2M parameters, 81.25% on CIFAR-10, with its single worst confusion being cat mistaken for dog, 164 times. This part asks: can overlapping the patches fix that?

A note on novelty. Overlapping patch embeddings are not my idea — they appear in published work such as T2T-ViT, PVTv2, and CvT. This is an educational, isolated ablation of a known technique, not a research claim. The value here is clarity and a controlled measurement.

Quick recap of Part 4

In Part 4 we built a standard ViT. Its patch embedding tiles the image with a 4×4 patch and a stride of 4 — a clean 8×8 grid of 64 non-overlapping patches (the top row of the figure in the next section).

It reached 81.25% test accuracy, and its hardest job was telling cats from dogs. That’s the number we’re trying to move.

The idea: let the patches overlap

A feature that straddles a patch boundary — a whisker, an ear edge — gets split across two tokens. What if patches overlapped, so no feature falls in a seam?

Mechanically, this is a one-parameter change: shrink the conv stride below the patch size. The patch content stays 4×4 (same 48 values, same projection); only the step between patches shrinks, so neighbours share pixels and the patch count grows. Here it is in 1-D (a single row of pixels, patch width 4):

Figure 1: Overlapping patches

I expose it as a percentage of the patch (so it’s meaningful at any patch size), resolved to whole pixels:

overlap_pixels = round(patch_size * overlap_percentage / 100.0)
overlap_pixels = min(overlap_pixels, patch_size - 1)      # keep stride >= 1
stride = patch_size - overlap_pixels
self.grid_size = (img_size - patch_size) // stride + 1
self.num_patches = self.grid_size ** 2

The overlap is 2D and symmetric — the window steps by stride along both height and width, so every interior patch overlaps its 8 neighbours (4 edges + 4 corners). And crucially, everything downstream is untouched: the CLS token, the positional-embedding table, the encoder, and the head all adapt to sequence length automatically. Only the patch count changes.

How overlap changes the model

The crucial observation: across the whole sweep, total parameters grow by only +8% — because only the positional-embedding table changes size. The conv projection, attention, MLP, and head are byte-for-byte identical. Yet the attention compute — which is O(N²) in the token count — explodes by ~170×:

Figure 2: Accuracy with diffrent amout of overlap; no-overlap, 0%, 25%, 50% & 75% in 4x4 patch

Overlap buys spatial resolution and compute, not model capacity. The MLP doesn’t get bigger; there are just more tokens flowing through the same weights.

The experiment — and a cautionary tale

My hypothesis: more overlap → finer detail → fewer cat/dog confusions. So I ran the sweep at batch size 128… and immediately hit a wall.

The 75% run ran out of GPU memory. 842 tokens means 842×842 attention matrices stashed for backprop — too much for a 6 GB card at batch 128. The quick fix: drop that one run to batch size 32.

Get Chandrashekhar Padole’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That single expedient decision is where the story gets interesting:

Figure 1: 75% ovelap experiment was in need of more memory for a batch of 128 image samples

At a glance: accuracy climbs steadily and then surges at 75% (+1.83 over 50%) — overlap looks better and better, with the biggest jump at the end! I nearly wrote exactly that conclusion.

⚠️ The confound. But 75% used a different batch size. Smaller batches inject more gradient noise, which acts as regularization and is worth +1–2% on CIFAR-10 all by itself. The 75% result was conflating two effects — overlap and batch size. I couldn’t honestly credit the surge to overlap.

De-confounding

The fix is dull but essential: re-run 0/25/50% at batch 32 too, so every point shares one batch size.

Figure 2: All experimetns were done with 32 image samples batch

The very first comparison quantified the confound:

0% overlap @ batch 128 → 81.25%, and the same model @ batch 32 → 82.68%. That’s a +1.43% lift from the smaller batch alone — before any overlap.

Figure 3: Reducing batch size increase the accuracy

So more than a third of the flashy “85% vs 81%” headline was just the batch change. With all four points at batch 32, the real shape of the overlap curve appears:

Here it is as a chart:

And the honest conclusion is the opposite of the confounded one:

• 0 → 25%: +2.07 — the overwhelming majority of the benefit
• 25 → 50%: +0.58 — reaches the peak (85.33%)
• 50 → 75%: −0.29 — worse, despite ~14× more compute

Overlap helps — a real +2.65% from 0→50% at matched batch — but with sharp diminishing returns. It peaks around 50%, and 75% is pure waste: more compute, no accuracy. The confounded data’s “returns keep rising” was an artifact of the batch change on that one run.

The overfitting smoking gun

Look at the train–val gap: it grows monotonically — 4.9 → 5.6 → 7.9 → 9.5 — even as validation accuracy plateaus after 50%.

Past the peak, the extra overlap capacity goes entirely into memorizing the training set, not generalizing — train marches toward 95% while validation flatlines. A clean, visual illustration of overfitting.

Did it help cat vs dog?

Yes. Measured consistently at batch 32 (so the 0% baseline here reads cat→dog 159, versus 164 at batch 128), the hardest class improves steadily as overlap increases:

Interestingly, cat accuracy keeps climbing all the way to 75% (64% → 71%) even though overall accuracy peaks at 50% — the extra overlap keeps helping this one detail-hungry class while slightly hurting easier ones. The cat→dog error count trends down (159 → 130) with a small non-monotonic wobble.

So finer tokenization genuinely helped the hardest, most detail-dependent class. But it reduced the confusion, it didn’t eliminate it — because the binding constraint was never the patching. It was the 32×32 resolution. Overlap re-samples the same pixels more cleverly; it cannot conjure detail that a thumbnail doesn’t contain.

Limitations & honest caveats

• Single seed. Each point is one training run. The 25/50/75% results sit within ~0.6% of each other — enough to establish the plateau, not enough to confidently rank the peak. Proper rigor needs 2–3 seeds per setting.
• One tiny dataset, one small model. CIFAR-10 at 32×32 with a 1.2M-param ViT. The resolution ceiling dominates; conclusions may differ at higher resolution.
• Not novel. Overlapping patch embeddings are established prior art (T2T-ViT, PVTv2, CvT). This is an isolated, educational measurement.

Takeaways

1. Overlap helps, with steep diminishing returns. ~+2.6% at matched batch, front-loaded: 25% captures most of it, 50% is the peak, 75% is wasted compute.
2. Overlap adds tokens and compute, not capacity (+8% params, ~170× attention at 75%) — and past the peak, that extra compute just fuels overfitting.
3. Control your variables. A single out-of-necessity batch-size change nearly produced the exact opposite conclusion. The most valuable step in the whole experiment was the boring one: re-running to de-confound.
4. Resolution is the ceiling. Better tokenization refines what’s already there; it can’t add detail a 32×32 image never had.

If you missed it, Part 4 covers the from-scratch implementation — patch embedding, attention, and the training loop — that everything here is built on.

Code and full experiment notes were built and documented from scratch. The point was never to beat a benchmark; it was to understand every number.