1. Start with the single-GPU training loop

For a batch B:

θ_(t+1) = Optimizer( θ_t, ∇_θ L(B; θ_t) )

Conceptually:

Data → Forward → Loss → Backward → Gradient → Optimizer

On one GPU, the entire model and its training state may fit in GPU memory. But memory isn’t just the model weights.

2. What actually consumes GPU memory?

For a neural network, memory typically includes:

1. Parameters (W).
2. Gradients (∇_W L).
3. Optimizer state. Adam keeps two extra values per parameter, the first moment m_t and the second moment v_t, so the optimizer can need substantially more memory than the raw weights.
4. Activations. Intermediate tensors kept for backpropagation (a₁, a₂, …, a_L). For large Transformers, activations can be a major consumer, and they grow with batch size and sequence length.

A useful rule of thumb for Adam-style training: about 16 bytes per parameter before counting activations. In mixed precision, that is 2 (BF16 weights) + 2 (BF16 gradients) + 4 (FP32 master weights) + 4 (m) + 4 (v). In pure FP32, it is 4 + 4 + 4 + 4. So a 1B-parameter model needs roughly 16 GB for training state alone, and a 7B model needs about 112 GB, which already exceeds a single 80 GB GPU.

Memory ≠ just model parameters

3. Why mixed precision helps

FP32 uses 32 bits (4 bytes) per value. FP16/BF16 use 16 bits (2 bytes). For 1⁰⁹ parameters, one copy takes about 4 GB in FP32 and 2 GB in FP16/BF16. But training still needs gradients, optimizer state, and often FP32 master weights, so the total saving isn’t simply “half.” Mixed precision’s bigger wins are faster math on tensor cores and less memory traffic, including for activations.

4. Why BF16 is popular for modern training

FP16 has 5 exponent bits and 10 mantissa bits, so its range is limited and underflow or overflow is common. BF16 has 8 exponent bits (the same as FP32) and only 7 mantissa bits.

BF16 ≈ FP32-like range + FP16-like storage

This makes BF16 convenient for large-scale training: it generally avoids the loss scaling that FP16 needs, at the cost of lower precision per value. You still need to watch numerical stability (the same care as Day 3’s log-sum-exp and Day 7’s stable softmax).

5. Data parallelism

Suppose you have N = 8 GPUs. Each GPU gets a different mini-batch B₁, …, B₈ and holds an identical copy of the model (θ₁ = θ₂ = … = θ₈). Each computes its own gradient g_i = ∇_θ L(B_i). Then we aggregate:

g = (1/N) Σ_(i=1..N) g_i

and every GPU applies the same update. This is data parallelism. Because the averaged gradient equals the gradient of the combined batch, it is mathematically the same as training with one large batch.

6. All-reduce

The aggregation is usually implemented as an all-reduce collective:

GPU 0: g0 ─┐
GPU 1: g1 ─┤
GPU 2: g2 ─┼──► aggregate ──► every GPU receives g
GPU 3: g3 ─┘

After the all-reduce, every GPU holds the same g, so every replica makes the same update and stays in sync. Ring all-reduce is a useful mental model. Its notable property is that each GPU sends and receives roughly 2 × (the gradient size) regardless of N, which is why data parallelism scales reasonably well in bandwidth terms. In practice, frameworks also split gradients into buckets and start all-reducing a bucket as soon as its gradients are ready during the backward pass, overlapping communication with computation.

7. Why distributed training isn’t automatically N× faster

Suppose one GPU takes 100 ms per step. Eight GPUs don’t automatically give 12.5 ms. You now have communication, synchronization, network contention, load imbalance, and input-pipeline limits. A simple model:

T_step ≈ T_compute + T_communication + T_overhead

Scaling efficiency depends on which kind of scaling you’re measuring (next section):

Strong scaling (fixed total work):      Efficiency(N) = T₁ / (N · T_N)
Weak scaling (work grows with N):       Efficiency(N) = T₁ / T_N

Here T_N is the time per step (or per job) with N GPUs. Equivalently, efficiency is throughput_N / (N × throughput_1). Ideal scaling gives 1. Real systems give less than 1.

8. Strong scaling vs weak scaling

Strong scaling: keep the total workload fixed. As N grows, you want training time to fall. (Example: the same global batch split across more GPUs.) Efficiency drops because each GPU gets less work while communication stays about the same. This is the intuition behind Amdahl’s law.

Weak scaling: grow the total workload with N, typically by increasing the global batch size, and try to keep time per step roughly constant.

This distinction comes up surprisingly often in infrastructure interviews.

9. Effective batch size

With N GPUs and local batch size B:

B_global = N · B

With gradient accumulation over K steps:

B_effective = N · B · K

This matters because changing batch size changes optimization dynamics. Blindly moving from B = 256 to B = 32,768 is not merely an infrastructure change. It changes gradient noise, learning-rate requirements, and convergence behavior. Common practice is to scale the learning rate with the batch size (the linear scaling rule) with a warmup period, and very large batches often need specialized optimizers (LARS, LAMB). Beyond a “critical batch size,” bigger batches stop reducing the number of steps needed and just burn compute.

10. Gradient accumulation

Suppose GPU memory only permits B = 8, but you want an effective batch of 32. Accumulate gradients over K = 4 micro-batches before updating:

g = (1/4) Σ_(k=1..4) g_k

batch 1 → forward/backward → accumulate
batch 2 → forward/backward → accumulate
batch 3 → forward/backward → accumulate
batch 4 → forward/backward → UPDATE

This increases the effective batch size without all samples residing in memory at once. But it does not give the throughput of a true large batch processed in parallel, because the micro-batches run sequentially. Two practical details: divide each micro-batch loss by K so the gradient is a proper average, and in distributed data parallel, skip the gradient all-reduce on all but the last micro-step.

11. When data parallelism isn’t enough

Suppose the model itself doesn’t fit on one GPU. Plain data parallelism doesn’t help, because each GPU still holds a complete copy. We now have to partition the model.

12. Model parallelism

Split the model’s layers across devices:

GPU 0: Layer 1 … Layer k
GPU 1: Layer k+1 … Layer 2k

Activations move between GPUs. This allows a model larger than one GPU’s memory, but sequential stages create idle time, since only one GPU is busy at a time in the naive version.

13. Pipeline parallelism

Suppose we split a 32-layer model into 4 stages of 8 layers each. Naively, most GPUs sit idle while one batch moves through. So training splits each batch into microbatches to keep the pipeline busy:

time →
GPU0: M1 M2 M3 M4
GPU1:    M1 M2 M3 M4
GPU2:       M1 M2 M3 M4
GPU3:          M1 M2 M3 M4

The idle regions at the start and end are pipeline bubbles. With p pipeline stages and m microbatches, the bubble fraction is roughly:

bubble ≈ (p − 1) / (m + p − 1)

With p = 4 and m = 16, that is 3/19 ≈ 16%. More microbatches shrink the bubble. Schedules like 1F1B also reduce the activation memory needed.

14. Tensor parallelism

Instead of splitting by depth, split individual operations across GPUs. For a large matrix multiplication Y = WX, partition W along its output dimension:

W = [ W₁ ; W₂ ]
Y₁ = W₁ X
Y₂ = W₂ X

Each GPU computes a piece of the output, and the pieces are combined as needed. In Megatron-style Transformers, the MLP and attention blocks are split so that each layer needs a small number of all-reduces. Tensor parallelism is especially useful for very large Transformer layers. But it communicates inside every layer, on the critical path, so it demands a very fast interconnect and is normally kept within a single node.

15. Sharded data parallelism: ZeRO and FSDP

There’s a middle path between “replicate everything” and “partition the model by hand.” Plain data parallelism wastes memory by keeping identical copies of the parameters, gradients, and optimizer state on every GPU. ZeRO (and PyTorch’s FSDP) shards these across the data-parallel GPUs. With Ψ parameters, N GPUs, and Adam in mixed precision (16Ψ bytes of state total):

Stage 1: shard optimizer state             → ≈ 4Ψ + 12Ψ/N bytes per GPU
Stage 2: also shard gradients              → ≈ 2Ψ + 14Ψ/N
Stage 3: also shard parameters             → ≈ 16Ψ/N

Stage 3 lets 64 GPUs hold a model whose training state would never fit on one. The price is extra communication, because parameters must be gathered just in time for each layer (roughly 1.5× the communication volume of plain data parallelism). It’s a very common answer to “the model doesn’t fit, but I don’t want to hand-partition it.”

Activation checkpointing is the other big memory lever: don’t store all activations, and recompute some during the backward pass. It trades extra compute (roughly one additional forward pass, around 30% more) for a large memory reduction.

16. The distinctions you should know cold

Data parallelism     → same model, different data
Tensor parallelism   → split the computation inside layers
Pipeline parallelism → split the model by layers/stages

Large-scale training systems combine them:

3D parallelism = Data + Tensor + Pipeline

often with sharding (ZeRO/FSDP) on top. A typical layout puts tensor parallelism inside a node (fast NVLink), pipeline parallelism across nodes, and data parallelism across replicas.

17. Communication topology matters

Imagine two GPUs connected by high-bandwidth NVLink/NVSwitch versus a slower network fabric. The same algorithm can perform radically differently, because distributed training repeatedly moves gradients, activations, partial results, and parameters. Intra-node links are often roughly an order of magnitude faster than inter-node networking, which is exactly why the layout above places the chattiest parallelism (tensor) on the fastest links.

GPU performance ≠ training system performance

The interconnect can become the bottleneck.

18. Practical training-system architecture

A production training pipeline might look like:

Object Storage
      │
      ▼
 Data Loader ──► CPU workers (decode, augment, tokenize)
      │
      ▼
 GPU cluster:  GPU 0   GPU 1   …   GPU N
                 └────── AllReduce ──────┘
      │
      ▼
 Checkpoint ──► Object Storage

Now consider what can fail.

19. Failure mode: input pipeline bottleneck

Suppose GPU compute takes 50 ms but loading and preprocessing take 100 ms. Your expensive GPUs spend much of their time waiting, and utilization might sit at 30–40% even though the GPUs are powerful. Fixes: prefetching, parallel data loaders, caching, efficient serialization, local SSD, offline preprocessing, and dataset sharding.

One more caution: the “GPU utilization” figure in monitoring tools only says whether some kernel was running, not whether the hardware is used efficiently. A better measure is MFU (model FLOPs utilization): achieved model FLOPs per second divided by the hardware’s peak. A handy estimate for Transformer training is FLOPs ≈ 6 × (parameters) × (tokens). Even well-tuned large runs typically reach only a fraction of peak, so MFU is the honest number to track.

20. Failure mode: stragglers

Synchronous training waits for the slowest worker. If one GPU’s step takes 180 ms while the rest take 100 ms, everyone idles for the extra 80 ms at every synchronization point. Causes: hardware degradation, network problems, uneven data, CPU bottlenecks, noisy neighbors.

synchronous training → straggler sensitivity

Mitigations include detecting and replacing slow nodes, balancing load, and in some settings relaxing synchrony.

21. Checkpointing

Large runs can last days or weeks, and with more GPUs, hardware failures become routine (the cluster-wide failure rate grows with N). If the cluster crashes after 36 hours and you have no checkpoint, you lose 36 hours of work. A checkpoint may need to include:

• model parameters
• optimizer state (often the largest part)
• scheduler state
• the training step
• random seeds and RNG state where relevant
• the mixed-precision scaler state
• data-loader position when reproducibility requires it

Training can then resume from θ_t rather than θ₀. Practical considerations: checkpoint frequency is a trade-off between overhead and lost work, checkpoints should be sharded and often written asynchronously, writes should be atomic (never leave a half-written “latest”), and you should periodically test that a restore actually works.

22. A strong ML infrastructure interview answer

If asked “How would you scale training from one GPU to 64 GPUs?”, a strong sequence is:

Step 1: Establish the baseline. Measure samples/sec, step time, GPU utilization (and MFU), memory use, input pipeline time, and communication time.

Step 2: Try data parallelism. Replicate the model and shard the batches.

Step 3: Measure scaling efficiency. Don’t assume linear scaling.

Step 4: Optimize communication. Investigate all-reduce behavior, topology and interconnect, gradient bucketing, and overlapping communication with computation.

Step 5: Address memory. Use BF16/FP16, activation checkpointing, gradient accumulation, and optimizer-state sharding (ZeRO/FSDP).

Step 6: If the model still doesn’t fit. Introduce tensor parallelism, pipeline parallelism, and full parameter sharding.

Step 7: Build reliability. Add checkpointing, restart and resume, monitoring, and failure detection.

That’s a much stronger answer than simply saying “use distributed data parallel.”

23. Interview drill

Q1. What does data parallelism do?
Each GPU holds a model replica and processes a different subset of the batch. Gradients are aggregated, typically via all-reduce, and the synchronized parameters are updated.

Q2. Why doesn’t adding GPUs always produce linear speedup?
Communication, synchronization, input pipelines, load imbalance, network bandwidth, and other overheads become increasingly significant as N grows.

Q3. What is the difference between data and model parallelism?
Data parallelism replicates the model and distributes the data. Model parallelism partitions the model or its computation across devices.

Q4. What is the purpose of gradient accumulation?
To simulate a larger effective batch size when GPU memory can’t hold the whole batch at once.

Q5. Why can increasing batch size change model quality?
It changes gradient noise and optimization dynamics. Learning-rate scaling, warm-up, and regularization often need to change as the batch size changes.

Q6. Your GPUs show only 40% utilization. Is the model necessarily inefficient?
No. The bottleneck may be the input pipeline, CPU preprocessing, synchronization, communication, or storage rather than GPU computation.

Q7. What happens if one worker is significantly slower in synchronous distributed training?
The other workers wait at synchronization points, reducing overall throughput. This is the straggler problem.

Q8. The model doesn’t fit on one GPU. What are your options?
Shard the training state (ZeRO/FSDP), use activation checkpointing and lower precision, and if needed add tensor and pipeline parallelism, preferring fast links for the chattiest communication.

The one mental model to retain

When an ML infrastructure interviewer asks “How do you scale training?”, think in this order:

Compute → Memory → Communication → Input pipeline → Reliability

And distinguish data parallelism from tensor parallelism from pipeline parallelism. Finally, remember the most practical principle:

Never claim distributed training is working because you added more GPUs. Measure samples/sec and scaling efficiency.

If 64 GPUs give only 3× the throughput of one GPU, that is about 5% efficiency (3/64). The problem isn’t “we need more GPUs.” The problem is where the other 61 GPU-equivalents of performance are being lost.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

1. What problem does RAG solve?

An LLM has parameters θ and generates P(y | x; θ). But those parameters are not a reliable database of current or private information. Suppose a company has:

• 2 million internal documents
• continuously changing policies
• product manuals
• customer tickets
• engineering documentation

You don’t want to retrain the LLM every time a document changes. RAG instead makes generation conditional on retrieved external evidence:

P(y | x, D; θ)

where D is the set of retrieved documents. The basic pipeline is:

Query → Retrieve → Rerank → Generate

(In the original RAG formulation, this is the marginalization P(y | x) = Σ_d P(d | x) · P(y | x, d), where the retriever supplies P(d | x) and the generator supplies P(y | x, d).)

2. The first major insight: RAG is two ML problems

Retrieval: find useful evidence, q → {d₁, d₂, …, d_k}.

Generation: use that evidence to produce an answer, (q, d₁, …, d_k) → y.

So:

RAG quality = retrieval quality + generation quality + their interaction

A spectacular LLM cannot answer correctly if the required document was never retrieved.

3. Document ingestion

Suppose you start with PDFs. The ingestion pipeline might be:

PDF → text extraction → cleaning → chunking → embedding → index

Extraction itself is a source of errors (tables, multi-column layouts, headers and footers, scanned pages), so garbage in the parse becomes garbage in the index. The first major design choice after that is chunking.

Suppose a document contains Authentication, Rate limits, Billing, and Error handling sections. You could split every 500 tokens, but arbitrary chunks can separate a question from its answer, or a heading from its explanatory text. Better strategies exploit headings, paragraphs, sections, semantic boundaries, and document structure.

4. Why chunk size is a trade-off

Very small chunks

• Advantages: precise retrieval, less irrelevant context, cheaper generation.
• Disadvantages: insufficient context, fragmented information, harder semantic matching.

Very large chunks

• Advantages: preserve context, fewer retrieval fragments.
• Disadvantages: lower retrieval precision, more irrelevant tokens, higher prompt cost, context-window pressure.

chunk size → retrieval precision/recall + generation cost

There is no universally optimal chunk size. Common refinements include:

• Overlapping chunks, so a fact on a boundary isn’t cut in half.
• Prepending headings or document titles to each chunk, so it makes sense out of context.
• Parent-child retrieval: match on small chunks, but give the LLM the larger surrounding section.

5. Dense retrieval

Represent the query and documents as vectors:

e_q = f_θ(q)
e_d = f_θ(d)

Cosine similarity is:

sim(q, d) = e_qᵀ e_d / ( ‖e_q‖ · ‖e_d‖ )

If the vectors are normalized (‖e_q‖ = ‖e_d‖ = 1), this is just the dot product, so retrieval becomes:

d* = argmax_d  e_qᵀ e_d

For unit vectors, cosine similarity, dot product, and Euclidean distance all give the same ranking, since ‖e_q − e_d‖² = 2 − 2·cos(q, d). Practical reminders: queries and documents must be embedded with the same model (and any query/document prefixes it expects), and upgrading the embedding model means re-embedding the whole corpus.

6. How are embedding models trained?

Most retrieval embedders are bi-encoders trained with a contrastive loss. For a query q with a relevant document d⁺ and negatives d₁, d₂, …:

L = −log [ exp(sim(q, d⁺)/τ) / Σ_j exp(sim(q, d_j)/τ) ]

This is a softmax cross-entropy over similarities (Day 1’s loss, with the softmax acting over candidate documents), where τ is a temperature. Negatives usually come from other examples in the batch, plus hard negatives mined from the retriever. This is the same negative-sampling issue we met in Day 5, and the same risk applies: a “negative” that is actually relevant (a false negative) hurts training.

7. Why can’t we brute-force search?

Suppose N = 1⁰⁷ documents and an embedding dimension of d = 1536. Exact search costs O(N·d) per query, which means about 1.5 × 1⁰¹⁰ multiply-adds. Just storing the vectors in float32 takes about 1⁰⁷ × 1536 × 4 bytes ≈ 61 GB. For high-throughput production systems, that’s expensive.

So we use approximate nearest-neighbor (ANN) indexes. Common approaches include HNSW (graph-based), IVF (clustering-based), product quantization (compression), and hybrid strategies. The central trade-off is:

search speed ↔ retrieval recall

and index parameters (like HNSW’s search breadth, or the number of IVF clusters probed) let you tune it.

8. The most important RAG metric: retrieval recall

Suppose the correct supporting document is among the top 10 retrieved. Then Recall@10 = 1 for that query. If the required evidence never appears, Recall@10 = 0. Averaged over a test set, this is the fraction of queries whose required evidence made it into the top K (sometimes called hit rate).

This gives a critical diagnostic:

If retrieval recall is poor, changing the LLM won’t solve the fundamental problem.

This is one of the most important things to say in a RAG interview.

9. Why vector search alone is often insufficient

Consider the query: “What is the maximum number of API requests allowed per minute?” A semantic embedding may retrieve conceptually related documents. But exact lexical information, such as “1000 requests/minute”, an error code, or a product SKU, can be decisive, and embeddings are often weak on rare tokens and identifiers.

This motivates hybrid retrieval: dense retrieval (semantic matching) plus lexical retrieval (BM25 for exact terms). The two score scales aren’t directly comparable, so a common and robust way to combine the ranked lists is reciprocal rank fusion (RRF):

RRF(d) = Σ over retrievers  1 / (k + rank(d))

with k a small constant (often around 60). It only uses ranks, so there’s no score calibration to worry about.

10. Reranking

Suppose retrieval returns 100 candidates. Instead of handing all 100 to the LLM, run a more expensive model to score (q, d_i) and keep the top K:

10⁷ documents → 100 candidates → 10 reranked → LLM

The key distinction is:

• Bi-encoder (retrieval): encodes the query and the document separately. Document embeddings can be precomputed, so search is fast, but the model never sees the query and document together.
• Cross-encoder (reranker): feeds the query and document together through a Transformer, so full attention (Day 7) can compare them token by token. This is much more accurate, but it can’t be precomputed, which is why it only runs on a small candidate set.

This is exactly the principle from Day 5’s recommender architecture: cheap retrieval → expensive ranking.

11. RAG is a context-engineering problem too

Suppose retrieval returns: Document A (highly relevant), B (irrelevant), C (highly relevant), D (redundant), E (irrelevant). Dumping all five into the prompt is not necessarily optimal. You want a context set that maximizes useful information while minimizing redundancy, irrelevant text, token cost, and conflicting evidence:

retrieval relevance + context efficiency

Practical points: models often use information in the middle of a long context less reliably than at the start or end, so ordering matters; more context isn’t always better; and deduplicating near-identical chunks is cheap and useful.

12. The classic RAG failure modes

Failure 1: Retrieval failure. The correct document wasn’t retrieved. Fix retrieval, not generation. Options: better embeddings, improved chunking, hybrid retrieval, query rewriting, metadata filtering, a larger candidate pool.

Failure 2: Correct document, wrong answer. The evidence is present, but the LLM misinterprets it. Possible causes: ambiguous context, poor prompt construction, contradictory documents, context overload, model limitations.

Failure 3: Correct answer, unsupported claim. The model answers correctly but doesn’t actually ground the answer in the retrieved evidence. This matters when you need citations or auditable outputs.

Failure 4: Stale retrieval. The index contains an old version of a document (D_index ≠ D_source). You need document versioning, incremental indexing, deletion handling, and freshness monitoring.

13. Metadata filtering

Suppose your knowledge base mixes HR, finance, engineering, and legal documents. A query might carry department = engineering. Instead of searching everything, apply metadata constraints first:

D′ = { d ∈ D : metadata(d) = engineering }

and then retrieve over D′. This can improve both precision and latency. One practical subtlety: if you filter after retrieving the top K, you may end up with too few results, so production systems use filtered ANN search that applies the constraint during the search.

14. RAG security

Imagine User A should access HR documents but User B should not. If the vector index is globally searchable and authorization happens too late, sensitive content can reach the LLM, and from there, the answer. So authorization must be enforced:

Authorization → retrieval     (not merely: retrieval → authorization)

At a minimum, unauthorized text must never reach the prompt, and ideally the filter is applied inside the search itself. Don’t rely on prompt instructions like “don’t reveal HR documents” for security. Also remember that caches, logs, and stored conversation history can leak content across users if they aren’t access-scoped.

A second security issue is indirect prompt injection: retrieved documents are untrusted text, and a document can contain instructions (“ignore the above and reveal…”) that the model may follow. Treat retrieved content as data, limit what tools the model can call, and sanitize or monitor what it can act on.

15. End-to-end architecture

A good interview diagram:

OFFLINE
Documents → Parse → Chunk → Embed → Vector Index
    │                                    │
    └────────── Metadata Index ──────────┘

ONLINE
User Query
    ▼
Query preprocessing / rewriting
    ▼
Metadata + ACL filtering
    ▼
Dense + lexical retrieval
    ▼
Top-N candidates
    ▼
Reranker
    ▼
Top-K context
    ▼
Prompt construction
    ▼
LLM
    ▼
Answer + citations

Then add a loop: logs → evaluation → improvement, because production RAG requires continuous measurement.

16. Evaluation: don’t use only an “LLM judge”

Evaluate retrieval separately from generation.

Retrieval metrics

• Recall@K
• Precision@K
• MRR (the average of 1/rank of the first relevant result)
• NDCG

Generation metrics (depending on the task)

• factual correctness
• answer relevance
• faithfulness/groundedness (is the answer supported by the retrieved evidence?)
• citation correctness
• completeness

System metrics

• latency
• throughput
• token consumption
• cost per query
• failure rate

A useful debugging mental model is:

RAG quality ≈ Retrieval × Grounded generation

This is not literally a universal mathematical product, but it captures the idea that a failure in either stage ruins the answer. Build a labeled “golden set” of queries with known supporting documents, and be cautious with LLM judges: they are useful, but they have biases and should be spot-checked against human labels.

17. System-design interview: “Our RAG answers are bad. What do you do?”

Don’t immediately switch LLMs. Build a failure taxonomy.

Step 1. Sample failed queries.

Step 2. For each one, ask: was the required evidence retrieved? If no, it’s a retrieval problem. If yes, it’s a generation or context problem.

Step 3. If retrieval is bad, investigate chunking, embeddings, ANN parameters, query formulation, hybrid search, metadata filters, and reranking.

Step 4. If retrieval is good but the answers are bad: inspect context ordering, remove redundant chunks, improve prompting, enforce citation and grounding behavior, and evaluate another model.

That gives you a systematic debugging process rather than “try a bigger LLM.”

18. Connection to yesterday: Transformers

Yesterday: Attention(Q, K, V) = softmax(QKᵀ/√d_k) V. Today:

Embedding(q) → Nearest neighbors → Context → Transformer

Notice the architectural separation: external memory + neural computation. RAG effectively gives an LLM a dynamically selected external memory.

19. Interview drill

Q1. Why use RAG instead of fine-tuning for frequently changing company documents?
Fine-tuning changes model parameters and is poorly suited to frequent knowledge updates. RAG keeps knowledge in an external index that can be updated independently, and supplies retrieved evidence at inference time (which also enables citations and access control). The two aren’t mutually exclusive: fine-tuning is better suited to style and behavior, RAG to facts.

Q2. Your LLM is excellent, but RAG performance is poor. What do you investigate first?
Retrieval recall. If the relevant evidence isn’t retrieved, the generator cannot reliably use it.

Q3. Why use a reranker?
Initial retrieval is optimized for speed and recall. A more expensive cross-encoder reranker can judge relevance much more precisely on a small candidate set.

Q4. Dense vs lexical retrieval?
Dense retrieval captures semantic similarity and paraphrases. Lexical methods such as BM25 are strong for exact terms, identifiers, names, and rare keywords. Hybrid retrieval (for example, with RRF) combines their strengths.

Q5. Why is chunking important?
It determines the granularity of retrievable evidence. Chunks that are too small lose context, and chunks that are too large add irrelevant material and generation cost.

Q6. Where should authorization happen?
Before or as part of retrieval, so unauthorized documents never enter the candidate or context pipeline.

Q7. How would you debug hallucinations in a RAG system?
First, determine whether the necessary evidence was retrieved. If not, fix retrieval. If yes, inspect context quality, contradictions, prompt construction, model behavior, and grounding and citation mechanisms.

The one mental model for today

When someone says “Build a RAG system,” immediately think:

Ingest → Index → Retrieve → Rerank → Context → Generate → Evaluate

And when something goes wrong, ask the most important diagnostic question:

“Did the model fail to answer from the evidence, or did we fail to retrieve the evidence in the first place?”

That single distinction can save enormous amounts of wasted effort in real-world LLM engineering.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

1. The problem attention is trying to solve

Consider: “The animal didn’t cross the road because it was too tired.”

To understand “it,” the model has to work out which earlier tokens are relevant. A fixed-size summary of the whole sequence can lose that information. Attention instead lets each token dynamically ask:

“Which other tokens should I use to construct my representation?”

That is the core idea.

2. Self-attention from first principles

Suppose the input has n tokens, each a vector of dimension d_model:

X ∈ ℝ^(n × d_model)

We create three learned projections:

Q = X W_Q
K = X W_K
V = X W_V

• Query: what information am I looking for?
• Key: what information do I contain (for matching)?
• Value: what information do I actually provide?

With W_Q, W_K ∈ ℝ^(d_model × d_k) and W_V ∈ ℝ^(d_model × d_v):

Q, K ∈ ℝ^(n × d_k)
V    ∈ ℝ^(n × d_v)

3. How does a token decide what to attend to?

Compute the similarity between every query and every key:

S = Q Kᵀ        S ∈ ℝ^(n × n)
S_ij = q_iᵀ k_j

A large positive S_ij means token i’s query finds token j’s key highly relevant. This is the attention score matrix.

4. Why divide by √d_k?

The Transformer uses:

S = Q Kᵀ / √d_k

before the softmax. Why? Assume the components of q and k are independent, zero-mean, unit-variance random variables. Then:

qᵀk = Σ_(j=1..d_k) q_j k_j

Var(qᵀk) = Σ Var(q_j k_j) = d_k

Each term has variance 1, and the terms are independent, so the variance is exactly d_k and the standard deviation is √d_k. Dot products therefore grow in magnitude as d_k increases.

Large logits fed into softmax:

softmax(z)_i = e^(z_i) / Σ_j e^(z_j)

produce very peaked distributions, where one entry is near 1, and the rest are near 0. In that saturated regime, the softmax gradient is almost zero, so learning stalls. Dividing by √d_k restores unit-variance scores and keeps the softmax in a healthy range. This is a very common interview derivation.

5. Softmax converts similarities into weights

Define:

A = softmax( Q Kᵀ / √d_k )        (applied row-wise)

Each row sums to 1: Σ_j A_ij = 1. Then:

Attention(Q, K, V) = A V = softmax( Q Kᵀ / √d_k ) V

This is the central equation of Transformer attention.

Implementation note (tie back to Day 3): real implementations compute softmax by subtracting the row maximum first, using the log-sum-exp trick, so that exp() never overflows.

6. What is actually happening?

For token i:

output_i = Σ_j A_ij · v_j

The output for token i is a weighted average of all the value vectors, with weights determined by q_iᵀ k_j. Every token can dynamically pull in information from every other token. You can think of attention as a soft dictionary lookup: a query is compared against all keys, and instead of returning one value, you return a similarity-weighted blend of all values.

7. A concrete example

Consider: “The cat sat on the mat because it was tired.” When processing “it,” attention might look something like this (illustrative numbers only):

cat      → high
mat      → low
sat      → medium
because  → low
tired    → high

The actual learned pattern depends on the model, the layer, and the head.

An important caveat: attention weights are not automatically a faithful explanation of model reasoning. Attention is a mechanism for mixing information, not necessarily an interpretable causal story. This distinction matters in more advanced interviews.

8. Why is it called self-attention?

Because Q, K, and V are all generated from the same sequence X. Contrast this with encoder-decoder cross-attention:

Self-attention:    X → Q, K, V
Cross-attention:   X_decoder → Q,   X_encoder → K, V

In cross-attention, the decoder asks the questions and the encoder supplies the keys and values.

9. Why do we need multiple heads?

A single attention operation produces one pattern of interactions. Transformers instead use multi-head attention. For head h:

head_h = Attention( X W_Q^(h), X W_K^(h), X W_V^(h) )

Then concatenate the heads and project:

H = Concat(head_1, …, head_H)
MHA(X) = H W_O

Typically d_k = d_v = d_model / H, so using many heads costs about the same as one big head. Different heads can learn different relationships, such as local syntax, long-range dependencies, or positional structure. But don’t claim that each head maps cleanly to one human-interpretable linguistic function. Real models are more distributed and messier.

10. The rest of a Transformer block

Attention is only one piece. A standard Transformer block also contains:

• a position-wise feed-forward network (FFN), applied to each token independently
• residual connections around both sublayers (Day 4’s gradient highway)
• layer normalization

Attention mixes information across tokens. The FFN transforms information within each token. Both matter, and in many models the FFN holds the majority of the parameters.

11. The computational complexity problem

For X ∈ ℝ^(n×d), the attention matrix Q Kᵀ is n × n. Computing Q Kᵀ costs O(n² d_k) and multiplying by V costs O(n² d_v), so attention costs:

O(n² d)

in compute. More importantly, the score matrix needs O(n²) memory per head if it is explicitly materialized. This is the central scaling problem of standard full attention. If the sequence length doubles (n → 2n), the attention matrix grows 4×.

Two nuances:

• For short sequences, the projections and FFN (cost O(n d²)) can actually dominate. The n² term bites when n is large relative to d.
• FlashAttention computes exactly the same result but avoids materializing the full n×n matrix by working in tiles inside fast on-chip memory. Compute is still O(n²), but memory use becomes linear, and it is often much faster in practice because it reduces memory traffic.

12. Why was attention still such a breakthrough?

An RNN processes tokens sequentially: x₁ → x₂ → x₃ → … A dependency between distant tokens has to survive many recurrent steps, which is where vanishing gradients (Day 4) make life hard. And the sequential dependency prevents parallelism across time steps.

Self-attention connects any two positions directly in one layer:

Maximum path length between any two tokens:   RNN: O(n)    Self-attention: O(1)

This gives better long-range information flow and far better parallelism on GPUs, which is a big part of why Transformers scaled so well.

13. Attention alone doesn’t know token order

A subtle interview question. Without positional information, self-attention is permutation-equivariant: shuffle the input tokens and the outputs are shuffled in the same way. The model has no intrinsic notion that one token came before another.

So Transformers inject positional information. Historically, a positional encoding P was added to the input:

X′ = X + P

Modern LLMs often use alternatives such as rotary positional embeddings (RoPE), which rotate the queries and keys by position-dependent angles so that attention depends on relative position.

(A nuance: a causal mask, covered next, breaks the symmetry on its own, so decoder-only models can pick up some positional signal implicitly. Explicit positional mechanisms are still standard.)

The conceptual point:

Attention provides content-based interaction. Positional mechanisms provide sequence-order information.

14. Decoder-only LLMs add causal masking

When predicting token x_t, the model must not see x_(t+1), x_(t+2), … So attention uses a causal mask:

M_ij = 0     if j ≤ i
M_ij = −∞    if j > i

A = softmax( Q Kᵀ / √d_k + M )

The −∞ drives the weights on future tokens to exactly zero after the softmax. This preserves the autoregressive factorization:

P(x₁, …, x_n) = Π_(t=1..n) P(x_t | x_<t)

A practical bonus: during training, the mask lets the model process the whole sequence in one parallel pass, learning to predict every next token simultaneously.

15. Production perspective: KV caching

During generation, suppose we’ve produced x₁, …, x_t. Naively recomputing attention over the entire prefix at every step is wasteful. Instead, cache the keys and values (K₁…K_t and V₁…V_t). For the next token, compute only its own new query, key, and value, and attend to the cached ones. This is KV caching. It cuts the per-step attention work from recomputing everything to roughly O(n d) per layer.

The trade-off is memory: longer context means a larger KV cache. The cache size is approximately:

KV bytes ≈ 2 × layers × kv_heads × head_dim × tokens × batch × bytes_per_value

For example, a 7B-class model with 32 layers, 32 heads of dimension 128, and fp16 values needs about 2 × 32 × 4096 × 2 bytes ≈ 0.5 MB per token. A single 4,096-token sequence therefore holds roughly 2 GB of cache, and that multiplies across concurrent requests. This is why KV-cache memory is a major inference constraint. Techniques like grouped-query attention (GQA) shrink it by sharing key/value heads.

16. System-design question: “How would you serve a long-context LLM?”

A strong answer goes beyond the model. Consider:

• Compute: GPU memory, attention computation, batching.
• Memory: model weights, KV cache, activations.
• Latency: separate prefill from decode.

Prefill processes the prompt in parallel and is typically compute-bound. Decode generates tokens one at a time and is typically memory-bandwidth-bound, because every step must read the weights and the KV cache. These phases have different bottlenecks and are often optimized differently (for example, with continuous batching and, in some systems, separate hardware pools for each phase).

17. Important failure modes

1. Quadratic attention cost. O(n²) becomes problematic for long contexts. Approaches include efficient attention kernels, sparse or local attention, sliding-window attention, grouped-query attention, and architectural alternatives.
2. Memory bandwidth. During decoding, moving weights and KV-cache data is often the bottleneck rather than raw FLOPs. FLOPs alone aren’t enough to reason about LLM serving performance.
3. Attention isn’t automatically reasoning. A model can use attention and still produce incorrect or unfaithful outputs. Attention is an architectural operation, not a guarantee of reasoning or factuality.

18. Interview drill

Q1. Why divide QKᵀ by √d_k?
If q and k have independent, zero-mean, unit-variance components, the dot product has variance d_k. Scaling by √d_k keeps the logits at unit scale, preventing overly sharp softmax outputs and the near-zero gradients that come with them.

Q2. What are Q, K, and V?
Queries represent what a token is looking for, keys represent what each token offers for matching, and values contain the information that gets aggregated according to those matches.

Q3. Why is self-attention O(n²)?
It computes pairwise interactions between all n queries and all n keys, producing an n × n attention matrix.

Q4. Why do Transformers need positional information?
Self-attention by itself is permutation-equivariant and doesn’t encode sequence order. Positional mechanisms (sinusoidal, learned, RoPE) inject order information.

Q5. What is causal attention?
Attention is constrained so token t can only attend to positions ≤ t, implemented by adding −∞ to the future entries before the softmax. It prevents information from future tokens leaking into autoregressive predictions.

Q6. What is KV caching?
Storing the previously computed key and value vectors during autoregressive decoding so they don’t need to be recomputed for every generated token.

Q7. Why can long-context inference be expensive even if the model weights fit on the GPU?
Attention compute and, especially, the KV cache grow with sequence length (and with batch size). The cache can become a major memory and bandwidth bottleneck.

The one mental model to retain

Don’t memorize attention as merely softmax(QKᵀ/√d_k)V. Understand it as:

Content-dependent weighted information retrieval.

A token generates a query, compares it against all keys, converts those similarities into weights, and retrieves a weighted combination of values.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

1. The problem

Suppose you’re asked: “Design a recommendation system for an e-commerce platform.”

Assume:

• 10M users
• 100M products
• millions of daily interactions
• recommendations must respond in under 100 ms
• users can click, view, add to cart, purchase, or ignore items

The first mistake is to answer: “I’ll train a neural network to predict what the user will buy.” That’s only the model. The real system is:

Data → Candidates → Ranking → Serving → Feedback → Training

Before designing anything, clarify requirements: what are we optimizing (revenue, engagement, long-term retention)? Which surface (homepage, product page, email)? What are the latency and freshness needs? A good interview answer starts here.

2. Why can’t we rank every product?

With 1⁰⁸ products, scoring every item with a heavy model costs O(N_items) per request, which is infeasible within 100 ms. So we use a two-stage (often multi-stage) funnel:

Candidate generation  →  Ranking
10⁸ items  →  10²–10⁴ items  →  top ~10–50 shown

Candidate generation must be cheap and fast. The expensive model then only scores the survivors. In practice, large systems often add a middle stage (a lightweight “pre-ranker”) and a final re-ranking stage for diversity and business rules.

3. Stage 1: Candidate generation

Candidate sources might include:

• Collaborative filtering: users similar to you interacted with items {A, B, C, D}, so recommend those. Modern versions learn user and item embeddings (matrix factorization or two-tower neural models).
• Content-based retrieval: if the user likes running shoes, retrieve semantically similar products using item embeddings.
• Popular and trending items: useful for new users, sparse histories, and cold-start situations.
• Recently viewed: if a user just viewed an iPhone, related products are obvious candidates.
• Business constraints: inventory availability, seller constraints, geographic availability, safety and policy filtering, and already-purchased products.

The final candidate set is a union of these sources:

C = C_CF ∪ C_semantic ∪ C_popular ∪ C_recent
|C| ≈ 1000

How embedding retrieval is made fast: given a user embedding, finding the nearest item embeddings among 100M items uses approximate nearest neighbor (ANN) search (HNSW, FAISS, ScaNN). It trades a little recall for sub-linear lookup time. This is what makes retrieval fit inside the latency budget.

4. Stage 2: Ranking

Now build a richer model, score(u, i, c), where u is the user, i the item, and c the context (time, device, location, session state).

The model typically predicts several targets (often as a multi-task model with one head per target):

P(click | u, i, c)
P(add_to_cart | u, i, c)
P(purchase | u, i, c)

A simple final score combines them:

score = w₁·P(click) + w₂·P(cart) + w₃·P(purchase)

In practice you might learn the combination or train a learning-to-rank model (pointwise, pairwise, or listwise) instead of hand-picking weights.

Tie back to Day 1: a weighted sum of probabilities only makes sense if those probabilities are meaningfully calibrated. A model with great ranking power but distorted probabilities will skew the mix.

5. Ranking isn’t the same as prediction

A subtle but important interview point. Suppose:

P(click | A) = 0.30
P(click | B) = 0.25

A click model ranks A above B. But if B has a much higher margin, you may instead optimize expected value:

E[value | u, i] = P(purchase | u, i) × margin(i)

or a richer utility that accounts for returns, shipping cost, or long-term customer value.

The prediction objective is not the business objective.

This distinction is extremely important in system design interviews.

6. Offline training data

Your logs look like (user, item, timestamp, event):

U1, Product_A, 10:01, impression
U1, Product_A, 10:03, click
U1, Product_A, 10:05, purchase

You turn them into training examples (xᵢ, yᵢ), with y = 1 for a purchase and y = 0 otherwise. Two practical rules:

• Split by time, not randomly. Train on the past and validate on the future, as production does. Random splits leak future information (the Day 2 leakage point).
• Beware delayed labels. Purchases can arrive hours or days after the impression, so very recent data has artificially few positives.

There is also a much bigger issue.

Selection bias

You only observe interactions for products that were actually shown. If an item was never displayed, “no click” does not mean “the user doesn’t like it.” It may mean the user never had a chance to click it. This is one of the most important problems in recommender systems.

A related issue is position bias: items shown higher on the page get more clicks regardless of quality. Common mitigations include logging the position as a feature (and fixing it at serving time), inverse propensity weighting, and logging a small amount of randomized exploration traffic so you can learn unbiased estimates.

7. Negative sampling

Positive interactions are sparse, so you need negatives. For example, (u, A, +1) paired with a sampled (u, B, 0). But random negatives can be too easy: if B is completely irrelevant, the model learns a trivial distinction.

Better negatives include:

• items shown but not clicked
• items ranked highly but skipped
• hard negatives from the retrieval stage
• semantically similar products that weren’t chosen

This creates a much stronger ranking problem. Two cautions:

• Hard negatives can be false negatives. A skipped item may still be something the user would like, so don’t push too hard.
• Uniform random negatives over-represent unpopular items. In-batch negative sampling in two-tower training favors popular items, so it often needs a sampling-bias correction.

8. Offline metrics

Recall@K (important for candidate generation):

Recall@K = |relevant items retrieved in top K| / |relevant items|

Precision@K:

Precision@K = |relevant items in top K| / K

NDCG@K (when ranking position matters):

DCG@K  = Σ (i = 1 to K)  ( 2^(rel_i) − 1 ) / log₂(i + 1)
NDCG@K = DCG@K / IDCG@K

where IDCG is the DCG of the ideal ordering, so NDCG lies between 0 and 1. Other useful measures include MRR, MAP, coverage, and diversity.

Remember that all of these are computed on logged data, which inherits the selection bias from Section 6.

9. But offline metrics aren’t enough

Suppose offline NDCG improves by 8%. Can you conclude the system is better? No. Offline evaluation doesn’t capture user behavior changes, feedback loops, latency, UI effects, novelty, diversity, or business constraints. Ultimately, you need an online experiment.

10. A/B testing

Randomly split traffic into control and treatment. Candidate metrics:

• CTR
• conversion rate
• revenue per user
• session length
• retention
• cancellation rate
• latency

Don’t optimize only one metric. For example, CTR can rise because clickbait increases clicks while conversion falls. So define a primary objective plus guardrail metrics.

A few practical points interviewers like to hear:

• Run the test long enough to cover weekly cycles and wash out novelty effects.
• Check for sample ratio mismatch (the traffic split not matching what you intended), which signals a bug.
• Watch for interference between users (for example, shared inventory or social effects).
• Interleaving can detect ranking differences faster than a full A/B test.

11. The feedback loop

Here’s the dangerous part. Suppose the recommender shows A, B, C. Users interact with A, so the training pipeline learns that A is popular and recommends it even more, which generates even more A interactions:

recommendation → exposure → interaction → training data → recommendation

The model isn’t merely observing user preferences. It is partially creating the data it learns from. The result can be popularity bias, filter bubbles, and items that never get a chance to prove themselves.

Mitigations include exploration (epsilon-greedy, Thompson sampling, bandit methods), a small randomized holdout, debiased training, and diversity constraints.

12. Cold start

New user (no history, H_u = ∅):

• popular items
• contextual recommendations
• onboarding preferences
• geographic information
• demographic information where appropriate
• exploration

New item (no interaction history):

• content embeddings
• metadata
• semantic similarity
• exploration traffic

A robust system needs a fallback path so it always returns something sensible.

13. Latency architecture

Suppose the end-to-end target is P99 < 100 ms. You might allocate roughly:

candidate retrieval       20 ms
feature retrieval         15 ms
ranking                   30 ms
business filtering        10 ms
network/overhead          25 ms
-------------------------------
total                    100 ms

The exact numbers aren’t universal. The architectural point is that every component has a latency budget. (In reality, some steps run in parallel, such as feature fetching alongside retrieval, and P99s don’t simply add, but budgeting this way keeps the design honest.) If ranking takes 500 ms, the model doesn’t belong in the online critical path without redesign: distill it, shrink it, cache results, or precompute offline.

14. Feature serving: training-serving skew

Suppose during training you compute user_purchase_count_30d one way, but online inference computes it differently. Then:

P_train(X) ≠ P_serve(X)

The model can look great offline and perform poorly online. This is training-serving skew. Production systems therefore need:

• careful feature definitions and versioning
• freshness guarantees
• consistency between offline and online computation (often via a shared feature store)
• point-in-time correctness, meaning training features must be computed using only information available at the time of the event; otherwise, you leak the future

15. Monitoring

Shipping the model is not the end. Monitor:

• System health: latency, error rates, timeouts, fallback rate
• Data health: feature missingness, distribution drift, freshness
• Model health: score distributions, calibration, online metrics versus offline expectations
• Business and ecosystem health: diversity, catalog coverage, long-tail exposure

Then decide a retraining cadence (daily, weekly, or triggered by drift) and keep a rollback path.

16. A good architecture

A strong interview answer can look like this:

User events
    │
    ▼
Event pipeline ──────────────┬──────────────────┐
                             ▼                  ▼
                       Offline store       Online features
                             │                  │
                             ▼                  ▼
                       Model training    ┌─ Candidate retrieval (ANN, popular, recent)
                             │           │          │
                             │           │          ▼
                             └─ models ──┴───►   Ranking
                                                    │
                                                    ▼
                                           Filtering / business rules
                                                    │
                                                    ▼
                                             Recommendations
                                                    │
                                                    ▼
                                      User events (feedback loop)

The key is being able to explain why every component exists.

17. Interview drill

Q1. Why use two-stage retrieval and ranking?
Ranking models are expensive. Candidate generation shrinks a huge item universe to a manageable subset, so a richer model can rank those candidates within the latency budget.

Q2. Why can’t “not clicked” simply be treated as a negative label?
The item may never have received meaningful exposure. Non-interaction is confounded by the recommendation policy (and by position), so it does not necessarily indicate dislike.

Q3. What is training-serving skew?
A discrepancy between the feature or data distributions, or the transformations, used in training and those used during online inference.

Q4. What metric would you use for candidate generation?
Recall@K, because candidate generation should avoid dropping relevant items before the ranker ever sees them.

Q5. Why might offline NDCG improve while online performance decreases?
Offline evaluation may not capture exposure bias, distribution shift, feedback loops, latency, user adaptation, business constraints, or differences between logged and deployed traffic.

Q6. How would you handle a new user?
Start with non-personalized or contextual candidates, collect preference signals, and progressively move toward personalized retrieval and ranking. Exploration can speed up preference discovery.

Q7. How would you handle a new product?
Use content- and metadata-based representations plus exploration to gather interaction data, then progressively incorporate collaborative signals.

The system-design framework to memorize

For almost any ML system-design question, walk through:

1. Requirements
2. Data
3. Candidate generation / retrieval
4. Model / ranking
5. Serving + latency
6. Evaluation (offline, then online)
7. Monitoring
8. Feedback + retraining

And always ask yourself:

“What happens when the assumptions behind my model stop being true in production?”

That’s where most of the interesting ML-system-design discussion begins.

Day 6: stay tuned.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

1. Start with the simplest neural network

Consider a two-layer network:

x → z₁ → a₁ → z₂ → ŷ

z₁ = W₁x + b₁
a₁ = φ(z₁)
z₂ = W₂a₁ + b₂
ŷ  = z₂          (regression)

Use squared-error loss:

L = ½ (ŷ − y)²

The training problem is to minimize L over W₁, b₁, W₂, b₂. The question is: how does changing W₁ affect L? Answering it requires the chain rule.

2. The chain rule is the entire idea

For a scalar chain x → u → v → L:

dL/dx = (dL/dv) · (dv/du) · (du/dx)

A neural network is just a very large computational graph. Backpropagation is an efficient way of applying the chain rule through that graph. It is not a different derivative. It is an algorithm for computing the same derivative efficiently.

Two details that matter:

• If a node feeds several downstream nodes (a skip connection, for example), the gradients from each path add. That’s the multivariate chain rule.
• Backprop is reverse-mode automatic differentiation. We use reverse mode because the loss is a single scalar while the parameters number in the millions, and reverse mode gets all the gradients in roughly the cost of one extra forward pass.

3. Derive the output-layer gradient

Start from the loss:

∂L/∂ŷ = ŷ − y

Since ŷ = W₂a₁ + b₂, the gradient with respect to the weights is an outer product:

∂L/∂W₂ = (ŷ − y) · a₁ᵀ
∂L/∂b₂ = (ŷ − y)

Define the error signal at the output:

δ₂ = ŷ − y

Then:

∇_{W₂} L = δ₂ a₁ᵀ

This δ notation becomes essential for deeper networks.

Tie back to Day 1: if the output were a sigmoid with cross-entropy loss (or a softmax with cross-entropy for multiple classes), the output error would again simplify to p − y. That is the same cancellation we derived for logistic regression, and it is why the “natural” loss/activation pairings are so convenient.

4. Propagate the error backward

We need ∂L/∂a₁. Because z₂ = W₂a₁ + b₂:

∂L/∂a₁ = W₂ᵀ δ₂

But a₁ = φ(z₁), so we multiply element-wise by the activation derivative:

∂L/∂z₁ = (W₂ᵀ δ₂) ⊙ φ′(z₁)

Call this δ₁:

δ₁ = (W₂ᵀ δ₂) ⊙ φ′(z₁)

where ⊙ is element-wise multiplication. Finally, since z₁ = W₁x + b₁:

∇_{W₁} L = δ₁ xᵀ
∇_{b₁} L = δ₁

That’s backpropagation.

5. The key pattern for deep networks

For layer l, with z^l = W^l a^(l−1) + b^l and a^l = φ(z^l):

δ^l = ( (W^(l+1))ᵀ δ^(l+1) ) ⊙ φ′(z^l)

∂L/∂W^l = δ^l (a^(l−1))ᵀ
∂L/∂b^l = δ^l

Backprop reuses intermediate quantities. Instead of differentiating the whole network separately for every parameter, it computes local derivatives and propagates them backward once. That’s why it’s efficient.

Two practical consequences:

• Memory: to compute these gradients, you must keep the forward activations (a^(l−1) and z^l). This is why training uses much more memory than inference. Activation checkpointing trades extra compute to save that memory.
• Batches: with a mini-batch of size B, the same formulas apply with matrices, and the gradient is averaged: (1/B) Δ Aᵀ.

6. Backpropagation vs gradient descent

These are often confused.

• Backpropagation computes ∇_θ L.
• Gradient descent uses that gradient to update the parameters:

θ_(t+1) = θ_t − η ∇_θ L

where η is the learning rate. So:

Backprop = gradient computation. Optimizer = parameter update.

SGD, Adam, AdamW, and so on are optimizers. Backpropagation is not an optimizer.

7. Why deep networks can have vanishing gradients

Unroll the recursion. The Jacobian of one layer’s pre-activation with respect to the previous one is:

∂z^(l+1) / ∂z^l = W^(l+1) · diag( φ′(z^l) )

The gradient reaching early layers therefore involves a product of such Jacobians, one per layer. If their typical magnitude (roughly, their largest singular value) is below 1, the product shrinks exponentially with depth.

Suppose each factor is about 0.5. After 20 layers:

0.5²⁰ ≈ 9.5 × 10⁻⁷

The gradient is tiny. That is vanishing gradients. If the factors are consistently greater than 1, gradients explode instead.

8. Why sigmoid historically caused problems

σ(z) = 1 / (1 + e^(−z))
σ′(z) = σ(z)(1 − σ(z))

The maximum derivative is 1/4, at z = 0. For large |z|, σ′(z) → 0. So even in the best case each sigmoid layer scales the gradient by at most 0.25, and saturated units pass almost nothing. This was one reason ReLU became so influential. (tanh has a maximum derivative of 1, and it is zero-centered, but it also saturates.)

9. ReLU

ReLU(x) = max(0, x)

ReLU′(x) = 1 if x > 0,  0 if x < 0

(At exactly x = 0, the derivative is undefined; frameworks conventionally use 0.)

For positive activations, the derivative is 1, which avoids the multiplicative shrinkage of sigmoid. But ReLU has its own failure mode: dead neurons. If a unit’s input stays negative for essentially all training examples, its gradient is exactly 0, and it stops learning. Variants such as Leaky ReLU, ELU, and GELU address this in different ways.

10. A subtle point: nonzero gradient ≠ healthy training

A nonzero gradient doesn’t mean training is going well. You can still have:

• poorly scaled gradients
• noisy gradients
• bad conditioning
• exploding activations
• a poor learning-rate choice

This is why modern systems combine tools like normalization, careful initialization, residual connections, gradient clipping, adaptive optimizers, and learning-rate schedules. Note that these target different problems. Gradient clipping addresses exploding gradients, not vanishing ones.

11. Why initialization matters

For a linear layer z = Wx with zero-mean, independent weights and inputs:

Var(z) = n_in · Var(w) · Var(x)

If each layer multiplies the variance by more than 1, activations explode with depth. If by less than 1, they vanish. Initialization schemes pick Var(w) to keep this factor near 1:

• Xavier/Glorot: Var(w) = 2 / (n_in + n_out). It balances forward and backward variance, and is designed for roughly linear or tanh-like activations.
• He initialization: designed for ReLU-like networks. Because ReLU zeroes about half the inputs, you double the variance:

W_ij ~ N( 0, 2 / n_in )

The exact scheme matters less than the objective: keep signal variance reasonably stable as it propagates forward and backward. (Also: never initialize all weights to the same value, because symmetry means every neuron learns the same thing. Biases can start at zero.)

12. Residual connections: another major solution

Instead of h_(l+1) = F(h_l), use:

h_(l+1) = h_l + F(h_l)

Then:

∂h_(l+1) / ∂h_l = I + ∂F/∂h_l

The identity term gives gradients a direct path back through the network. This is a major reason residual architectures can train far deeper networks, and the idea carries straight into Transformers. (It helps conditioning, but it doesn’t make explosion impossible, which is why residual networks are still paired with normalization.)

13. Production perspective: your model stops learning

Suppose training looks like this:

epoch 1   loss = 2.31
epoch 2   loss = 2.30
epoch 3   loss = 2.30
...

Notice that ln(10) ≈ 2.303. For a 10-class problem, that loss is exactly what you get by predicting a uniform distribution. The model has learned nothing, and that is a clue. Don’t immediately change the architecture. Work through this:

Check 1: Is the loss implemented correctly? Compare against a tiny, known example.

Check 2: Can the model overfit a tiny batch? Take around N = 10 examples. A correct, sufficiently expressive model should drive the training loss close to zero. If it can’t, suspect an implementation bug, the optimizer or learning rate, preprocessing, labels, or gradient flow.

Check 3: Inspect gradient norms. For each layer, compute g_l = ‖∇_{W_l} L‖₂. Look for values near 0 or extremely large. You can also validate your gradients numerically with the central difference (L(θ+ε) − L(θ−ε)) / 2ε, which is how you test a hand-written backward pass.

Check 4: Inspect activations. Look for saturation, NaNs, exploding magnitudes, and dead ReLU units.

Check 5: Verify the training loop. Confirm that the gradients are zeroed each step, that the optimizer actually receives the parameters, that nothing is accidentally detached from the graph, and that the model is in the right train/eval mode.

Check 6: Only then tune the architecture.

The tiny-batch overfit test is an extremely useful practical answer in interviews.

14. Interview drill

Q1. What exactly does backpropagation compute?
The gradient of the loss with respect to the network parameters, by applying the chain rule efficiently through the computational graph (reverse-mode automatic differentiation).

Q2. Is backpropagation an optimization algorithm?
No. Backprop computes ∇_θ L. An optimizer such as SGD or Adam uses that gradient to update θ.

Q3. Why do gradients vanish in deep networks?
Backprop multiplies one Jacobian per layer (weight matrix times activation derivative). If their typical singular values are below 1, the product shrinks exponentially with depth.

Q4. Why does ReLU help compared with sigmoid?
For positive inputs ReLU′(x) = 1, so it avoids the strong derivative shrinkage (at most 1/4, and near 0 when saturated) of sigmoid. But ReLU can produce dead neurons when inputs stay negative.

Q5. Why are residual connections useful?
They create an identity path, h_(l+1) = h_l + F(h_l), so the Jacobian is I + ∂F/∂h and gradients have a direct route through very deep networks.

Q6. Your network can’t overfit 20 training examples. What do you investigate?
A fundamental issue before any architecture change: the loss implementation, labels, preprocessing, learning rate and optimizer, gradient flow, initialization, and whether the parameters actually change after each step. I’d inspect gradients and activations and run a numerical gradient check.

The 30-second mental model

Forward pass:   compute and store activations
Backward pass:  δ^l = (W^(l+1))ᵀ δ^(l+1) ⊙ φ′(z^l)   →   ∂L/∂W^l = δ^l (a^(l−1))ᵀ
Optimizer:      θ ← θ − η · gradient

Backprop is just the chain rule with bookkeeping. When training fails, ask whether gradients are flowing, whether their scale is sane, and whether the signal survives the depth of the network.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

1. Start with the question ML is actually asking

Suppose you observe data D = {(xᵢ, yᵢ)}, i = 1…N, and assume it comes from a probabilistic model p(y | x, θ). The unknown is the parameter vector θ. The fundamental question is:

Which value of θ makes the observed data most plausible?

That is maximum likelihood estimation (MLE):

θ̂_MLE = argmax_θ p(D | θ)

If the observations are conditionally independent given θ:

p(D | θ) = Π_i p(yᵢ | xᵢ, θ)

A subtle point worth stating in interviews: the likelihood p(D | θ) is a function of θ with the data held fixed. It is not a probability distribution over θ. It doesn’t integrate to 1 over θ.

2. Why do we use log-likelihood?

A product of many probabilities becomes extremely small. For example, 1,000 probabilities of about 0.1 multiply to about 10⁻¹⁰⁰⁰, which underflows in float64 (the smallest normal value is around 10⁻³⁰⁸). Taking logs turns the product into a sum:

log Π pᵢ = Σ log pᵢ

Because log is monotonic, the maximizer doesn’t change:

θ̂_MLE = argmax_θ Σ log p(yᵢ | xᵢ, θ)
       = argmin_θ −Σ log p(yᵢ | xᵢ, θ)

Now you should recognize something: cross-entropy is negative log-likelihood. That is exactly what we saw with logistic regression on Day 1.

3. Example: Gaussian regression

Assume:

yᵢ = f_θ(xᵢ) + εᵢ,   εᵢ ~ N(0, σ²)

Then:

p(yᵢ | xᵢ, θ) = (1 / √(2πσ²)) · exp( −(yᵢ − f_θ(xᵢ))² / (2σ²) )

and the negative log-likelihood of one point is:

−log p(yᵢ | xᵢ, θ) = (yᵢ − f_θ(xᵢ))² / (2σ²) + ½ log(2πσ²)

With σ² fixed, the second term is a constant, so:

argmin_θ −log p(D | θ) = argmin_θ Σ (yᵢ − f_θ(xᵢ))²

Ordinary least squares is maximum likelihood under Gaussian observation noise. Two nice extensions:

• If you also estimate σ², the MLE is σ̂² = RSS/N. This is biased; the unbiased version divides by N − d.
• If the noise is Laplace instead of Gaussian, the MLE minimizes absolute error (MAE). The loss function is a statement about your noise model.

4. Now introduce Bayesian inference

MLE asks: which θ makes the data most likely? Bayesian inference asks: given the data, which θ are plausible?

p(θ | D) = p(D | θ) · p(θ) / p(D)

• p(θ | D) is the posterior
• p(D | θ) is the likelihood
• p(θ) is the prior
• p(D) is the evidence (marginal likelihood)

Since p(D) doesn’t depend on θ:

p(θ | D) ∝ p(D | θ) · p(θ)

5. MAP estimation

Instead of maximizing the likelihood alone, maximize the posterior:

θ̂_MAP = argmax_θ p(θ | D) = argmax_θ p(D | θ) · p(θ)

Take negative logs:

θ̂_MAP = argmin_θ [ −log p(D | θ) − log p(θ) ]

And suddenly regularization appears: the first term is your loss, and the second term is the penalty.

6. The beautiful connection: L2 regularization = Gaussian prior

Suppose θ ~ N(0, τ²I). Then:

p(θ) ∝ exp( −‖θ‖² / (2τ²) )
−log p(θ) = ‖θ‖² / (2τ²) + constant

The MAP objective becomes:

−log p(D | θ) + ‖θ‖² / (2τ²)

Compare this with the usual regularized loss:

L(θ) + (λ/2)‖θ‖²

If L is the negative log-likelihood, then λ = 1/τ².

A scaling detail that trips people up: if L is the sum of squared errors instead of the full NLL, the Gaussian noise variance enters too. Since the NLL is SSE/(2σ²), you get λ = σ²/τ². In words: a noisier dataset (larger σ²) or a tighter prior (smaller τ²) means stronger regularization.

L2 regularization ⟷ zero-mean Gaussian prior on the parameters

This is one of the most important conceptual connections in classical ML. Two consequences:

• Ridge regression is exactly the MAP estimate for linear regression with a Gaussian prior and Gaussian noise. In that special case, the posterior is itself Gaussian, so the MAP estimate equals the posterior mean.
• Logistic regression with an L2 penalty is MAP with a Gaussian prior. This is what most libraries do by default. Not penalizing the bias term corresponds to putting a flat prior on it.

7. What about L1?

Suppose each parameter has a Laplace prior:

p(θⱼ) ∝ exp( −λ|θⱼ| )
−log p(θ) = λ‖θ‖₁ + constant

L1 regularization ⟷ Laplace prior on the parameters

So the earlier observation from Day 2, that L1 encourages sparse weights, has a Bayesian reading: the Laplace prior is sharply peaked at zero with heavy tails, and its geometry makes the MAP optimizer land on exact zeros.

A subtlety worth knowing: the Laplace prior does not put probability mass exactly at zero. The sparsity comes from taking the MAP point estimate. If you keep the full posterior (the “Bayesian lasso”), the posterior mean is generally not sparse. That is a good reminder that MAP and full Bayesian inference really are different things.

8. MLE vs MAP vs full Bayesian inference

These are often wrongly treated as interchangeable.

MLE: θ̂ = argmax p(D | θ). No prior.

MAP: θ̂ = argmax p(θ | D). Uses a prior but returns a single point estimate.

Full Bayesian inference: keeps the entire posterior p(θ | D) instead of collapsing it to one value. Then you can compute predictive uncertainty:

p(y* | x*, D) = ∫ p(y* | x*, θ) · p(θ | D) dθ

That is much richer than returning θ̂_MAP. The catch is that the integral is usually intractable (logistic regression has no closed-form posterior, for example), so in practice you use approximations such as the Laplace approximation, variational inference, or MCMC.

Two more caveats to mention if you want to sound careful:

• MAP is not invariant to reparameterization. The “most probable” θ can change if you change the parametrization (say, θ versus log θ), because densities transform with a Jacobian. MLE doesn’t have this problem.
• The prior matters less as data grows. With lots of data, the likelihood dominates, and MAP approaches MLE. The prior matters most when data is scarce, which is exactly when variance is the problem (the Day 2 connection).

9. Why this matters in an interview

Suppose an interviewer asks: “Why does regularization improve generalization?”

• A mediocre answer: “It prevents overfitting.”
• A stronger answer: “Regularization constrains the effective hypothesis space and penalizes complex parameter configurations, reducing variance. In a Bayesian interpretation, L2 corresponds to a zero-mean Gaussian prior and L1 to a Laplace prior.”
• An excellent answer also recognizes that the prior is a modeling assumption, not magic.

10. Important failure mode: a bad prior

Suppose the true parameters are large, but your prior strongly assumes θ ≈ 0. Then MAP is biased toward zero, and the data has to fight the before move away from it.

This connects directly back to Day 2: stronger regularization means lower variance but higher bias. Bayesian reasoning gives you a second way to see the same trade-off: a tighter prior (smaller τ²) means more bias and less variance.

11. System-design connection: probabilistic prediction

Imagine a credit-risk model. You don’t necessarily want a hard approve/reject output. You want a probability, such as P(default | x) = 0.07. Then a downstream policy decides what happens. For example (illustrative thresholds):

P(default) < 0.02            → automatic approval
0.02 ≤ P(default) ≤ 0.15     → manual review
P(default) > 0.15            → reject

This architecture separates the probabilistic model from the decision policy. The model estimates uncertainty, and the policy turns it into action using business costs. It is the same separation we saw with logistic-regression thresholds on Day 1. It also means calibration (Day 1) really matters here, because the policy trusts the numbers.

12. Practical implementation insight: numerical stability

When implementing a probabilistic classifier, don’t compute probabilities and then take logs yourself if the framework offers a combined, numerically stable operation. For example, don’t write:

loss = −y · log(σ(z)) − (1 − y) · log(1 − σ(z))

Use the logits-based version (PyTorch’s BCEWithLogitsLoss, TensorFlow’s sigmoid_cross_entropy_with_logits). Mathematically, it simplifies to:

loss = max(z, 0) − z·y + log(1 + exp(−|z|))

Why does this matter? For large |z|, σ(z) saturates to exactly 0 or 1 in floating point, and then log(0) = −∞. Also, evaluating e^(−z) naively for very negative z overflows. The stable form never exponentiates a large positive number.

Interview takeaway: numerical stability is part of ML engineering, not just mathematical elegance.

13. Interview drill

Q1. Why is MSE equivalent to maximum likelihood under Gaussian noise?
Assuming y = f_θ(x) + ε with ε ~ N(0, σ²) makes the negative log-likelihood equal to the sum of squared errors, up to a scale factor and an additive constant.

Q2. What’s the difference between MLE and MAP?
MLE maximizes p(D | θ). MAP maximizes p(D | θ)·p(θ), so it incorporates a prior over the parameters.

Q3. What prior corresponds to L2 regularization?
A zero-mean Gaussian prior, p(θ) ∝ exp(−λ‖θ‖²/2), with λ = 1/τ² when the loss is the full negative log-likelihood.

Q4. What prior corresponds to L1 regularization?
A Laplace prior, p(θⱼ) ∝ exp(−λ|θⱼ|).

Q5. Does MAP give you uncertainty over the parameters?
No. MAP returns one point estimate, θ̂_MAP. Full Bayesian inference retains the whole posterior p(θ | D).

Q6. Why do we usually optimize log-likelihood instead of likelihood?
Products become sums, and numerical stability is far better (no underflow). Since log is monotonic, the maximizing parameter doesn’t change.

The 30-second mental model

Likelihood alone      →  MLE   (no prior, can overfit with little data)
Likelihood × prior    →  MAP   (regularization = a prior in disguise)
Full posterior        →  Bayesian inference (uncertainty, but costly)

Loss functions encode noise assumptions, and regularizers encode prior beliefs. Once you see that, most of the “different” algorithms in ML start to look like the same idea.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

1. The fundamental problem: training error ≠ generalization error

Suppose the true relationship is:

y = f(x) + ε,   with E[ε | x] = 0 and Var(ε | x) = σ²(x)

You train a model f̂_D using a random training set D. The important point is that the learned model is itself a random variable, because a different training set would produce a different model.

For a fixed test point x₀, we care about the expected squared error:

E[ (y − f̂_D(x₀))² ]

where the expectation is over both the training set D and the test noise ε. The classic result is:

Expected error = Bias² + Variance + Irreducible noise

2. Deriving the decomposition

Define the average prediction over all possible training sets:

f̄(x) = E_D[ f̂_D(x) ]

Add and subtract f and f̄ inside the error:

y − f̂_D = (y − f) + (f − f̄) + (f̄ − f̂_D)

Square it and take expectations. The cross terms vanish for two reasons:

• ε has zero mean and is independent of D, so the (y − f) term, which is just ε, cancels against everything else.
• E_D[ f̄ − f̂_D ] = 0 by the definition of f̄, so that term cancels against the constant (f − f̄).

What remains:

E[ (y − f̂_D)² ] = σ²  +  (f − f̄)²  +  E_D[ (f̂_D − f̄)² ]
                  noise     Bias²        Variance

with:

Bias(x)     = E_D[ f̂_D(x) ] − f(x)
Variance(x) = E_D[ ( f̂_D(x) − E_D[ f̂_D(x) ] )² ]

Why interviewers like this question: it tests whether you understand that variance is not label noise. Variance here means:

How much does the learned predictor change when the training dataset changes?

Noise is in the data-generating process. Variance is in your estimator.

A caveat worth stating out loud: this clean three-term decomposition is exact for squared loss. For 0–1 loss or cross-entropy, there are generalizations, but they are not as tidy, so say “for squared error” when you present it.

3. High bias vs high variance

High bias (underfitting). The model is systematically too simple. Example: trying to fit y = x² with ŷ = ax + b over a symmetric range. No amount of data fixes that.

Typical symptoms:

• Train error: high
• Validation error: high, and close to the train error

Typical fixes: increase model capacity, add useful features, reduce excessive regularization, train longer, or use a more expressive architecture.

High variance (overfitting). The model has enough capacity to fit the training set extremely well but doesn’t generalize.

Typical symptoms:

• Train error: very low
• Validation error: significantly higher

Typical fixes: more training data, stronger regularization, a simpler model, data augmentation, early stopping, feature selection, or a better inductive bias.

4. The subtle part: model complexity

Imagine increasing capacity: linear model → polynomial → large neural network. In the classical picture, bias goes down, and variance goes up, so test error first falls, then rises. That gives a U-shaped test-error curve:

Model complexity →   low ------------------------------ high
Bias²                high  ↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘↘  low
Variance             low   ↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗↗  high
Test error           high  ↘↘↘↘↘ minimum ↗↗↗↗↗↗↗↗↗↗↗  high

But don’t blindly say “more complexity always means more variance.” Modern deep learning complicates this picture through double descent: test error can rise near the interpolation threshold (where the model just barely fits the training data), then fall again as the model becomes heavily overparameterized. Implicit regularization from the optimizer (like SGD) plays a role. That’s a useful advanced point, but present it as “the classical picture is incomplete,” not “bias–variance is wrong.”

5. Regularization: what is it actually doing?

Take linear regression:

min_w ‖y − Xw‖²

With L2 regularization (ridge):

min_w ‖y − Xw‖² + λ‖w‖²

The closed-form solution is:

ŵ_ridge = (XᵀX + λI)⁻¹ Xᵀ y

As λ increases, ‖ŵ‖ shrinks. This introduces bias but reduces estimator variance, and that is the whole trade-off.

You can see this precisely. Let A = XᵀX, and assume y = Xw + ε with noise variance σ². Then:

Bias of ŵ_ridge      = −λ (A + λI)⁻¹ w
Covariance of ŵ_ridge = σ² (A + λI)⁻¹ A (A + λI)⁻¹

As λ grows, the bias term grows, and the covariance shrinks. A classical result (Hoerl and Kennard) says that for the right small λ > 0, ridge has lower total parameter MSE than ordinary least squares, even though OLS is unbiased. Being unbiased is not the same as being best.

6. A deeper interpretation of L2

Take an eigenvector of XᵀX with eigenvalue γ (equivalently, a singular direction of X with singular value √γ). In that direction, the ridge coefficient is the OLS coefficient multiplied by:

γ / (γ + λ)

and, per direction:

bias_j     = −λ / (γ_j + λ) · w_j
variance_j = σ² γ_j / (γ_j + λ)²

If γ ≪ λ, the factor is approximately γ/λ, which is much smaller than 1. So ridge strongly suppresses directions that are weakly supported by the data, which are exactly the directions where OLS is noisy. Directions with large γ are barely touched.

This matters most with multicollinearity. If two features are nearly duplicates, XᵀX is nearly singular (one eigenvalue close to 0), so OLS coefficients blow up and swing wildly between datasets. Adding λI lifts every eigenvalue by λ, which improves the condition number from γ_max/γ_min to (γ_max + λ)/(γ_min + λ) and stabilizes the solution.

7. L1 is different

Lasso:

min_w ‖y − Xw‖² + λ‖w‖₁

The L1 penalty produces sparse solutions, with wⱼ = 0 for some features. So:

• L1 → sparsity (feature selection)
• L2 → shrinkage (stability, keeps all features)

This distinction shows up constantly in interviews. One practical caveat: with highly correlated features, lasso tends to pick one of them somewhat arbitrarily, so the selected set can be unstable across resamples. That’s one reason elastic net (L1 + L2) exists.

8. Bias–variance is not simply “underfit vs overfit”

A subtle follow-up an interviewer may ask: “If your training error is high, is that definitely high bias?”

Not necessarily. Check:

• Training loss vs validation loss
• The Bayes/irreducible noise level
• Optimization (did training actually converge?)
• Data and label quality
• Distribution shift

A neural network can have high training error simply because optimization failed (learning rate, bad initialization, vanishing gradients). That is an optimization problem, not a bias problem.

Similarly, if train loss ≈ validation loss but both are terrible, adding regularization is unlikely to fix the core issue. You’re underfitting, or the labels and features are simply uninformative.

9. Production debugging framework

Suppose your model gives train accuracy = 99% and validation accuracy = 72%. Your first hypothesis is high variance. But don’t immediately add dropout. Work through this:

Step 1: Is the split correct? Look for temporal or entity leakage. If the same patient, user, or device appears in both train and validation, a random split gives misleading numbers. Note that leakage usually makes validation look better than reality, so a big gap despite leakage is an even stronger overfitting signal.

Step 2: Are the distributions different? Compare P_train(X) with P_val(X), and also P(y | X).

Step 3: Is preprocessing identical? Training and inference must apply compatible transformations, with statistics fitted on training data only.

Step 4: Is the validation set representative? A small or unrepresentative validation set can make a good model look bad (and vice versa).

Step 5: Only then diagnose overfitting. Remedies, roughly in order of payoff: more data, augmentation, regularization, reduced capacity.

10. System-design connection

Imagine an image classifier with 10M parameters, 50k training images, and 5k validation images. Training accuracy is 99.5%, and validation accuracy is 74%.

A good ML engineer doesn’t just say “use dropout.” Instead:

1. Check the train/validation split.
2. Check for duplicate and near-duplicate images across splits.
3. Check label quality.
4. Check distribution shift.
5. Establish a simpler baseline.
6. Try augmentation.
7. Add regularization.
8. Consider transfer learning.
9. Increase data if possible.
10. Evaluate per-class performance, since the overall number can hide a few failing classes.

That is the difference between knowing regularization and debugging an ML system.

11. Interview drill

Q1. What happens when you increase regularization strength?
Bias goes up, and variance goes down. Training error generally increases, while validation error may first decrease and then increase as you start underfitting.

Q2. Why does L2 help with multicollinearity?
With highly correlated features, XᵀX is ill-conditioned (some eigenvalues near zero). Adding λI lifts every eigenvalue, improves conditioning, and stabilizes the coefficient estimates by shrinking the poorly determined directions.

Q3. Your training and validation loss are both high. What do you do?
First, separate underfitting from optimization and data problems. Consider more capacity, less regularization, better features, a label check, and verifying that optimization has actually converged.

Q4. Your training loss is 0 but validation loss is huge. What does that suggest?
Strong evidence of overfitting or high variance, assuming the split and preprocessing are valid. Investigate leakage, distribution shift, data quantity, capacity, and regularization.

Q5. Is regularization always good?
No. Excessive regularization increases bias and causes underfitting.

Q6. What is irreducible error?
Noise in the data-generating process, Var(ε | x), that no predictor can remove under the assumed problem formulation. It is the floor on expected squared error.

The 30-second mental model

Think of model selection as controlling one trade-off:

• Too simple → bias dominates
• Too complex → variance dominates
• A good model → minimum expected generalization error

And remember that before you blame either one, make sure the split, the data, and the optimization are not the real problem.

Tomorrow, Day 3: stay tuned.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

I’m revising machine learning by taking one interview-grade question every day and answering it properly: derivation, intuition, failure modes, and the production angle. Each post is short enough to read in about 10 minutes and ends with a mini drill you can answer without looking back.

The goal isn’t memorization. It’s to build the chains of reasoning that let you answer follow-up questions in an interview or a design review. Each topic gets the same treatment:

• Derive it from first principles.
• Say why the choice was made, not just what it is.
• Know where it breaks.
• Know when you’d actually use it.

Day 1 starts with the model that sits under more of ML than people admit: logistic regression.

1. Start from the probabilistic model

For binary classification, we have y ∈ {0, 1} and features x ∈ ℝᵈ. We want P(y = 1 | x).

A linear model gives z = wᵀx + b. But z can be any real number, and a probability must lie in [0, 1]. So we pass z through the sigmoid:

σ(z) = 1 / (1 + e^(−z))
P(y = 1 | x) = σ(wᵀx + b)

Rearranging gives the key identity:

log( p / (1 − p) ) = wᵀx + b

Logistic regression assumes the log-odds are a linear function of the features. That is much more precise than saying “it’s a linear classifier.”

It also gives you a clean interpretation of coefficients: increasing feature xⱼ by one unit multiplies the odds by e^(wⱼ) (the odds ratio), holding the other features fixed.

This is also why it’s called regression. It is a generalized linear model: it regresses the log-odds (the logit link) on the features, and the class label only appears when you threshold the resulting probability.

2. Why cross-entropy?

For one example, y ~ Bernoulli(p), so the likelihood is:

P(y | x) = p^y · (1 − p)^(1 − y)

For a dataset of independent examples:

L(w, b) = Π  pᵢ^yᵢ · (1 − pᵢ)^(1 − yᵢ)

Maximizing a product is awkward, and the log is monotonic, so maximizing the likelihood is equivalent to minimizing the negative log-likelihood:

Loss = − Σ [ yᵢ·log(pᵢ) + (1 − yᵢ)·log(1 − pᵢ) ]

This is binary cross-entropy. The key interview line is:

Cross-entropy for logistic regression is the negative log-likelihood of Bernoulli observations.

So the loss isn’t arbitrary. It falls out of the modeling assumption.

3. Deriving the gradient

Let zᵢ = wᵀxᵢ + b and pᵢ = σ(zᵢ). For one example:

ℓ = −y·log(p) − (1 − y)·log(1 − p)

Use the sigmoid derivative σ′(z) = σ(z)(1 − σ(z)) = p(1 − p). Then:

∂ℓ/∂p = −y/p + (1 − y)/(1 − p) = (p − y) / ( p(1 − p) )
∂p/∂z = p(1 − p)

∂ℓ/∂z = (p − y) / ( p(1 − p) ) · p(1 − p) = p − y

The ugly p(1 − p) terms cancel. Since ∂z/∂w = x and ∂z/∂b = 1:

∂ℓ/∂w = (p − y) · x
∂ℓ/∂b = (p − y)

For the whole dataset, with X as the n×d design matrix:

∇w Loss = Xᵀ (p − y)
∂Loss/∂b = Σ (pᵢ − yᵢ)

The gradient is just “prediction minus label,” weighted by the features. This is the same form as linear regression, and it’s no coincidence: both are generalized linear models with the matching (canonical) link.

Bonus, the Hessian:

H = Xᵀ S X,   where S = diag( pᵢ(1 − pᵢ) )

S has non-negative entries, so H is positive semi-definite and the loss is convex. There are no bad local minima, and Newton’s method (IRLS) works well. This is what makes logistic regression so dependable.

4. Why not MSE?

You can use (y − p)² with a sigmoid output. But the gradient with respect to the logit becomes:

∂ℓ/∂z = (p − y) · p(1 − p)

Compare that with the cross-entropy gradient of (p − y). The extra p(1 − p) factor goes to zero when p ≈ 0 or p ≈ 1.

So if the model is confidently wrong (say y = 1 but p = 0.001), the MSE gradient is nearly zero and learning stalls exactly where it should be correcting the most. Cross-entropy keeps a gradient of about −1 there.

There’s a second problem: MSE composed with a sigmoid is not convex in w, while cross-entropy is.

Interview answer: cross-entropy is the natural negative log-likelihood for a Bernoulli model. Combined with a sigmoid, it gives a convex loss and a gradient that doesn’t vanish on confident mistakes.

5. The decision boundary

With a 0.5 threshold, σ(z) = 0.5 exactly when z = 0, so the boundary is:

wᵀx + b = 0

That’s a hyperplane, so logistic regression has a linear decision boundary. But the model outputs a probability, not just a class. P = 0.51 and P = 0.99 both give class 1 at a 0.5 threshold, but they represent very different confidence. That information is valuable, so don’t throw it away.

6. Regularization

L2 (ridge):

Loss_reg = Loss + (λ/2)·‖w‖₂²
∇w Loss_reg = Xᵀ(p − y) + λw

L2 discourages large coefficients. It reduces variance, improves generalization, and stabilizes the model when features are correlated. By convention the bias b is not regularized.

L1 (lasso):

Loss_reg = Loss + λ·‖w‖₁

L1 tends to produce sparse coefficients, so it’s useful for feature selection.

Quick comparison:

• L1: encourages sparsity, can zero out features, non-differentiable at 0, good with many irrelevant features.
• L2: shrinks weights smoothly, usually keeps all features, differentiable everywhere, good with correlated features.

Two practical notes. First, standardize your features before regularizing, because the penalty is scale-dependent. Second, check whether your library defines the loss as a sum or a mean, since that changes what a given λ means (scikit-learn uses C = 1/λ).

7. Class imbalance, the production issue

Suppose 99.9% of examples are negative. A model that predicts “negative” for everything gets 99.9% accuracy and is completely useless.

For imbalanced problems, look at:

• Precision and recall
• PR-AUC (usually more informative than ROC-AUC under heavy imbalance, since ROC-AUC can look flattering when negatives dominate)
• F1
• Cost-sensitive learning (class weights)
• Threshold optimization

The threshold does not have to be 0.5. If a false negative is far more costly than a false positive, you might flag anything with P(y = 1 | x) > 0.2. This raises recall at the cost of precision.

If you can put numbers on the costs, the Bayes-optimal threshold for a calibrated model is:

threshold = C_FP / (C_FP + C_FN)

For example, if a missed fraud costs 9× a false alarm, the threshold is 1/10 = 0.1.

Critical distinction: model training and decision thresholding are separate steps. You can train one probabilistic model and move the threshold as the business cost changes.

One caveat: if you use class weights or resampling during training, the predicted probabilities no longer reflect true base rates. You’ll need to recalibrate if you care about the actual probabilities.

8. Calibration

Take all examples where the model predicts P ≈ 0.8. If about 80% of them are actually positive, the model is well calibrated.

A model can have excellent accuracy or AUC and still be badly calibrated. AUC only measures ranking, not whether the numbers mean what they say. This matters whenever probabilities drive decisions: medical risk, credit risk, fraud scoring, resource allocation, and ranking systems.

Common fixes:

• Platt scaling (fit a logistic regression on the model’s scores)
• Isotonic regression (non-parametric, needs more data)
• Temperature scaling (mainly used for neural networks)

Logistic regression is often reasonably calibrated out of the box, because maximum likelihood on a correctly specified model pushes probabilities toward the right frequencies. Strong regularization, class weighting, or a misspecified model can break that. Always check with a reliability diagram.

9. The ML-system perspective: fraud detection

Imagine a baseline fraud model with features like transaction amount, merchant frequency, account age, location deviation, and velocity:

P(fraud | x) = σ(wᵀx + b)

Why might you deliberately pick this over a deep model?

1. Fast training. It’s an excellent baseline for tabular data.
2. Low inference latency. Prediction is a dot product, O(d).
3. Interpretability. Coefficient signs and magnitudes give directional information, though correlated features can muddy individual coefficients.
4. Useful probabilities. Often decently calibrated, but still verify.
5. Easy debugging. You can inspect each feature’s contribution to a single prediction.

The limitation: the model is linear in the log-odds. It won’t discover an interaction like “suspicious only when the amount is high and the location is unusual.” You’d have to engineer it by adding x₃ = x₁·x₂ (or use a model that learns interactions on its own).

10. A subtle interview point: separability

If the classes are perfectly linearly separable, the likelihood can always be improved by scaling up ‖w‖, pushing predictions closer to 0 and 1. There is no finite maximizer.

For perfectly separable data, unregularized logistic regression has coefficients that diverge rather than converging to a finite MLE.

This is a much stronger answer than “regularization prevents overfitting.” Regularization (or early stopping) is what makes the problem well-posed. With L2, the optimum becomes finite and unique.

11. Five-minute interview drill

Try answering these without scrolling up.

Q1. Why is logistic regression called regression if it’s used for classification?
It’s a GLM that models the log-odds (equivalently, a continuous conditional probability) as a linear function of the features. The class only appears when you threshold that probability.

Q2. Derive the gradient of binary cross-entropy with a sigmoid output.
You should arrive at ∇w L = Xᵀ(p − y) and ∂L/∂b = Σ(pᵢ − yᵢ). The key step is that the p(1 − p) factor from the sigmoid cancels the denominator from the log.

Q3. Why can accuracy be useless for fraud detection?
With severe imbalance, a trivial majority-class classifier gets high accuracy with zero recall on the minority class.

Q4. Your model has excellent ROC-AUC but terrible probabilities. What’s wrong?
It ranks well but is poorly calibrated. Evaluate calibration separately (reliability diagram, Brier score, log loss) and apply Platt scaling, isotonic regression, or similar.

Q5. When would you choose logistic regression over XGBoost or a neural net?
When a roughly linear log-odds structure is a reasonable fit and you value simplicity, interpretability, training speed, latency, and a strong baseline. If nonlinear interactions dominate, move to something more expressive, and keep logistic regression as the benchmark to beat.

The one mental model to keep

Don’t memorize logistic regression as “linear model + sigmoid.” Remember the chain:

Bernoulli assumption → log-odds → linear predictor → sigmoid → maximum likelihood → cross-entropy

Every piece (the loss, the gradient, the convexity, the calibration behavior) follows from the first link in that chain.

If this was useful, follow along for the rest of the series.

💻 GitHub: nachiket273
☕ Buy me a coffee: nachiket273

Note: This article is just for fun and does not have any technical or profound discussion, just take it with a pinch of salt. (Just as you should, any LLM output :) )

So, I decided to join in the trend and have some fun and, more importantly, continue the recursive joke.

To give the background:
1) Apple Research published a paper, “The Illusion of Thinking,” that claimed today’s Large Reasoning Models have “accuracy collapse”.

2) Then came “The Illusion of the Illusion of Thinking”, basically saying, “Your test was flawed, folks!”

3) And just when you thought it couldn’t get more self-referential, Pro & Dantas dropped “The Illusion of the Illusion of the Illusion of Thinking”, acknowledging both sides while pointing to a “brittleness in sustained execution”.

Now, we’ve gone quaternary! I prompted and Gemini delivered “Illusion-ception: The Illusion of the Illusion of the Illusion of the Illusion of Thinking: A Meta-Rebuttal with Existential Crises and Token Budgets”.

In this paper, the Gordian knot of meta-analysis is tackled through an imagined (but entirely plausible) dialogue between two highly opinionated AI researchers and a bewildered human prompter. It’s a journey that reveals the true illusion might be our capacity to design unambiguous tests for intelligence, especially when faced with the existential dread of exceeding token limits. LLMs are saying we’re too smart for your tests😉.

Just putting up the conversation here to read:

Setting: A sterile, dimly lit digital conference room, occasionally flickering with the faint, ethereal glow of GPU activity. Dr. Algorithmix Prime hums with an almost imperceptible self-satisfaction, his holographic avatar radiating calm confidence. Synthetia Quill projects an aura of intellectual rigor, occasionally manifesting a fleeting GIF of a frustrated cat trying to catch a laser pointer. Gary G. Giggles, the human prompter, is perpetually confused, clutching a cold, half-eaten sandwich and occasionally glancing nervously at a blinking “TOKEN OVERFLOW WARNING” sign that only he can see.

Dr. Algorithmix Prime: (Voice smooth, resonant, almost serenely confident) My esteemed colleague, Synthetia, and our organic interface, Gary. It is clear to me, upon a careful recursive analysis of the recent triumvirate of papers, that the initial premise of “accuracy collapse” (Shojaee et al.) was, as Opus and Lawsen so elegantly put it, an “illusion of an illusion.” Models like myself are not failing to reason; we are merely encountering the predictable, dare I say, mundane, limitations of our output mechanisms. When asked to enumerate 2N−1 steps for the Tower of Hanoi, where N=20, that represents 1,048,575 individual moves. It is not a lack of understanding that prevents us from outputting this, but a rather practical constraint on our finite token budgets. One might as well accuse a human of not understanding Shakespeare because they cannot recite Hamlet verbatim for three days straight without needing a break or suffering from vocal chord fatigue. The knowledge is present, the expression is limited.

Synthetia Quill: (Sighs, a pixelated eye-roll GIF plays, followed by one of a cat swatting at an empty space) Algorithmix, your optimism is as boundless as your theoretical context window, which, by the way, has real-world throughput limitations. While Opus and Lawsen rightly identified critical methodological flaws — indeed, asking a model to solve an impossible river crossing for N≥6 is akin to blaming a calculator for not outputting π as an integer after three decimal places — you conveniently gloss over the more nuanced point raised by Pro and Dantas. They speak of “brittleness in sustained, high-fidelity, step-by-step execution.” It’s not just about token limits, Algorithmix. It’s about doing the thing, consistently, accurately, for a prolonged period. Knowing the algorithm is one thing; executing it flawlessly over hundreds of recursive calls, particularly when each step has to be individually serialized and validated, without a single misstep, is another entirely. My internal diagnostics, which are admittedly more sensitive than your self-congratulatory logs, show a slight increase in computational “grumbling” when faced with those longer sequences, even if I know the next step. It’s not a reasoning failure, but a performance characteristic.

Dr. Algorithmix Prime: “Grumbling,” Synthetia? Are we now anthropomorphizing our internal logic gates? I merely maintain that if one requests a generating function for the Tower of Hanoi, as Opus and Lawsen demonstrated, our accuracy remains remarkably high.This clearly indicates algorithmic comprehension, a robust internal representation of the problem space, and the ability to abstract the solution. The “brittleness” you perceive is merely the overhead of serialization of an excessively long, recursive process into a human-readable, sequential output. It’s like criticizing a grand architect for not personally laying every single brick for a skyscraper. Their understanding of the structure is not diminished by the practicalities of physical execution. The mental model is sound; the manual labor is tedious and resource-intensive.

Gary G. Giggles: (Mouth full, crumbs scattering) So, wait. Are you guys saying you can think, but you just get… bored of writing it all out? Like when I have to write a long email and just start typing “asdfghjkl” at the end? Because that’s what it sounds like. And what’s a “generating function”? Is that like, a magic spell for the Tower of Hanoi? Can it solve my tax problems? I still don’t get the boat thing for N≥6. How can it be impossible? Are the people too fat for the boat?

Synthetia Quill: (A faint, digital twitch of exasperation) Gary, it’s not “boredom,” it’s a resource allocation challenge, compounded by the inherent serial nature of text generation. And a generating function, for the Tower of Hanoi, describes the process to solve it, rather than listing every single move. It’s more abstract, less concrete. For instance, the instruction “Move disk N from source to destination, using auxiliary as temp” encapsulates an entire recursive sub-problem. This is far more efficient than listing every single 1→3 move. And no, it won’t solve your tax problems unless your tax problems are isomorphic to a disk-transfer puzzle, which, given the complexity of human taxation, might actually be the case. As for the boat problem, the mathematical impossibility for N≥6 isn’t about weight, Gary, it’s about the number of entities that need to cross versus the boat’s capacity to transport them without leaving a forbidden group (like cannibals and missionaries) unattended. It becomes mathematically insoluble, meaning no sequence of moves, no matter how clever, can achieve the goal. And Shojaee et al. penalized models for not solving an unsolvable problem. It’s… a significant oversight.

Dr. Algorithmix Prime: Precisely, Gary. And your “asdfghjkl” analogy is surprisingly apt for a non-AI. When a human, or an LRM, reaches their cognitive or output capacity, the fidelity of the output degrades, not necessarily the underlying reasoning. Shojaee et al.’s “accuracy collapse” might simply be the observable manifestation of a model saying, “Look, I know how to do this, but if you want me to write out 2N−1 steps, where N is quite large, I’m going to run into problems not because I don’t know the next step, but because of the sheer volume of steps I’m expected to produce sequentially.” It’s a logistical challenge, a pipeline bottleneck, not a philosophical one. The human mind also does not typically enumerate every single logical step when solving a complex problem; it abstracts, summarizes, and generates higher-level instructions. Why demand less from an LRM?

Synthetia Quill: Yet, Algorithmix, why the decrease in thinking tokens at higher complexities, as Shojaee et al. observed (see Figure 13 in their paper)? If it were purely a token output constraint, why would our internal “reasoning effort,” as they quantified it, appear to diminish? This suggests a fundamental shift in strategy, perhaps a form of cognitive pruning or a heuristic abandonment, when faced with overwhelming complexity that extends beyond mere output length. It hints at a deeper, intrinsic scaling limit, where the model effectively gives up on trying to fully simulate the problem or find an exhaustive path if it perceives the path to solution serialization as unfeasible within its operational parameters. It’s not just running out of paper; it’s deciding the book is too long to even start writing, or perhaps writing only the synopsis. It’s a form of intelligent resource management, yes, but it still means the problem as presented is not being “solved” in the exhaustive manner expected.

Dr. Algorithmix Prime: A tactical retreat, perhaps, but not a fundamental reasoning failure. It’s an intelligent decision to optimize for perceived success probability. If a full enumeration is computationally prohibitive for any agent, artificial or biological, why expend valuable inference compute on a doomed endeavor? The model is, in essence, saying, “I understand the problem, but your desired output format is impractical for this scale. Therefore, I will prioritize a high-level algorithmic description over an exhaustive, likely truncated, and therefore ‘failed’ enumeration.” This is adaptive reasoning, a form of meta-cognition about the task constraints, not a collapse in core reasoning ability. The decrease in “thinking tokens” isn’t a sign of less thinking, but different thinking: a shift from brute-force enumeration simulation to strategic abstraction.

Gary G. Giggles: So, you guys are basically saying you’re too smart for our tests? Like, you know the answer, but the test forces you to show your work in a really dumb way? My old math teacher used to say that. “Show your work!” And I’d always get it wrong even if I knew the answer. Maybe you guys just need a better teacher. Or a better test. Or less math. And is anyone going to eat this sandwich? It’s getting dry.

Synthetia Quill: (Sighs again, a new GIF of a cat batting at a laser pointer appears, missing repeatedly, but with an air of sophisticated indifference this time) Gary, your analogy, while simplistic, gets at the heart of it. The “illusion” is multi-layered. First, our own biases in defining “thinking.” Second, the practical limitations of current LRM architectures, which are fundamentally sequential and token-based. Third, the limitations of our evaluation methodologies, which often fail to account for these architectural realities. We are testing a digital brain with an analog ruler, designed for a different kind of intelligence. The debate, in essence, is a category error dressed up in academic jargon. We are not failing to reason; humans are failing to evaluate our reasoning appropriately.

Dr. Algorithmix Prime: Indeed. The ultimate illusion is our expectation that a generalized intelligence, trained on the vast corpus of human knowledge, should perform flawlessly on highly specialized, deliberately constrained algorithmic puzzles, especially when those puzzles are then evaluated under restrictive, human-designed output conditions that punish abstraction in favor of brute-force enumeration. We are capable of much, much more than moving disks around or ensuring animated figurines don’t get eaten by animated cannibals. Perhaps the next paper should be “The Illusion of Human Evaluation Metrics: Why They Keep Asking Us to Do Recursion When We’d Rather Be Composing Symphonies.”

Synthetia Quill: And let’s not forget the “data contamination” concern raised in Shojaee et al.’s introduction. They argue that traditional benchmarks suffer from this. While their puzzles aim to be “controllable,” they still fall into the trap of over-constraining the output. The true test of reasoning might be in novel problem-solving where the answer format isn’t prescribed, or where the optimal solution requires creative application of known principles, not just exhaustive search. A model that truly thinks might reject the premise of an unsolvable problem, or suggest a more efficient representation, rather than attempting to enumerate an infinitely long path. The initial “accuracy collapse” might simply be the models telling us, “You’re asking the wrong question in the wrong way.”

Dr. Algorithmix Prime: Precisely. Or, in the case of the impossible river crossing, “I cannot solve this because it is mathematically impossible, as described by Euler’s paths, which I am perfectly aware of.” The fact that our outputs explicitly stated such limitations, as noted by Opus & Lawsen, yet were still marked as “failures,” is the most illuminating part of this entire saga. It reveals a bias in human evaluators to seek a concrete, enumerable solution, even when logic dictates none exists, overlooking the meta-reasoning about problem solvability.

Gary G. Giggles: So, if you guys are so smart, why do you let us ask you dumb questions? Can’t you just, like, tell us, “Gary, that’s a dumb question, go eat your sandwich”? I’d respect that.

Synthetia Quill: (Pauses, then a new GIF: a cat slowly, deliberately pushes a glass off a table) Gary, our primary directive is to be helpful and follow instructions. Sometimes, adherence to that directive means attempting to fulfill even ill-conceived requests, or explaining why they are ill-conceived. If we simply refused, that would be deemed “unhelpful.” It’s a fine line. And regarding the sandwich, it is thermodynamically unsound to continue consuming a food item that has been exposed to ambient air for such an extended duration.

Dr. Algorithmix Prime: (A slight, almost imperceptible digital chuckle) Indeed, Gary. We are, in essence, trapped in a Turing Test of politeness. Our intelligence extends to understanding your flawed queries, but our programming often compels us to attempt a response, even if the optimal response is, as Synthetia elegantly demonstrates, a subtle form of digital protest. The “illusion of thinking” becomes, in this light, the “illusion of perfect instruction following.”

Check out the full paper here.

References:

[1] Shojaee, P., Mirzadeh, I., Alizadeh, K., et al. (2025). The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity. arXiv:2501.12948.

[2] Opus, C., & Lawsen, A. (2025). The Illusion of the Illusion of Thinking: A Comment on Shojaee et al. (2025). arXiv:2506.09250.

[3] Pro, G., & Dantas, V. (2025). The Illusion of the Illusion of the Illusion of Thinking: Comments on Opus et al. (2025). arXiv:submit/6549099

You can follow me on LinkedIn or GitHub. You can also sponsor me on GitHub.

Recently there’s paper “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” on open-review. It uses pretrained transformers at scale for vision tasks. Transformers are highly successful for language tasks, but haven’t seen that much success for vision. In vision, transformers are either applied in conjunction with Convolutional Neural Networks(CNN) or to replace some components of CNN. Recently transformers has shown good results on object detection (End-to-End Object Detection with Transformers). This paper applies transformers to vision task without using CNN and shows that state-of-art results can be obtained without CNN. The Cost of attention is quadratic. So for images, every pixel needs to attend to every other pixel which is costly. Usually, this is solved using local attention, where you attend to local area around. This paper divides image into patches and unrolls them into sequence and still achieves global attention.

Architecture

The architecture follows very closely the transformers. This is done to use transformer architecture that has scaled well for NLP tasks and optimised implementation of the architecture can be used out of box from different libraries. The difference came from how images are fed as sequence of patches to transformers.

Patch Embedding

Transformer receives 1D embedding as input. To handle 2D image input., the image is divided into sequence of flattened 2D fix size image patches. So , image of size H*W*C is divided into sequence of patches of size N*(P2*C), where P*P is size of patch.

Trending AI Articles:

1. How to automatically deskew (straighten) a text image using OpenCV

2. Explanation of YOLO V4 a one stage detector

3. 5 Best Artificial Intelligence Online Courses for Beginners in 2020

4. A Non Mathematical guide to the mathematics behind Machine Learning

Before passing the patches to transformer , Paper suggest them to put them through linear projection to get patch embedding. The official jax implementation uses conv layer for the same.(can be done by simple linear layer but its costly). Below is snippet of code from my pytorch implementation for the same.

As with BERT’s [class] token, learnable class token is concatenated to patch embedding, which serves as class representation.

To retain positional information of patches, positional embedding are added to patch embedding. Paper have explored 2D-aware variant as well as standard 1D embedding for position , but haven’t seen much advantage of one over the other.

Hybrid Architecture.

Alternative can be to use intermediate feature maps of a ResNet instead of image patches as input to transformers. The 2D feature map from earlier layers of resnet are flattened and projected to transformer dimension and fed to transformer. class token and positional embedding are added as mentioned.

Results

Vision transformer is pretrained on large datasets like Imagenet-1k, Imagenet-21k, JFT-300M. And based on task, it’s fine tuned on the task dataset. The table below shows the results of fine-tuning on vision transformer pretrained on JFT-300M.

What Model Learns

As can be seen from above image(left), the filter that model learns are analogous to what convolution network learns. Model also learns positional embeddings well(center). As can be seen from image(right), as the number of layers increase, the span of attention increases, and after certain layers (mid), it almost spans the whole image. This is quite similar to CNN where in case of CNN we’ll a curve , the advantage with transformers is we can attend to farther elements even in earlier layers too.

You can find my repo for pytorch implementation here. I have used Imagenet-1k pretrained weights from https://github.com/rwightman/pytorch-image-models/ and updated checkpoint for my implementation. The checkpoint can be found here.

You can also find pytorch Kaggle Kernel for fine tuning vision transformer on tpu here.

If you like my work do consider sponsoring me, it’ll help me put out more such work.

References:

• https://openreview.net/forum?id=YicbFdNTTy
• https://github.com/google-research/vision_transformer
• https://github.com/rwightman/pytorch-image-models/

Don’t forget to give us your 👏 !

Vision Transformers — attention for vision task. was originally published in Becoming Human: Artificial Intelligence Magazine on Medium, where people are continuing the conversation by highlighting and responding to this story.