Autograd

What is Autograd?

Autograd = Automatic Differentiation. It’s the engine that computes gradients for you automatically — no manual calculus required.

When you train a neural network, you need to know: “how much does the loss change if I tweak this weight?” That’s a gradient. Autograd computes all of them for you.

Why Autograd?

Without it, you’d have to derive and code every gradient by hand. For a network with millions of parameters — that’s impossible. Autograd handles it all, no matter how complex your model.

Computation Graph

Every calculation is recorded as a graph — nodes are values, edges are operations. This graph is what Autograd walks backwards through to compute gradients.

Get Abdul Awal Nadim’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Let’s say you compute: L = (x * w + b)²

Each node knows its inputs. During backprop, Autograd walks this graph right to left, applying the chain rule at each step.

Basic Idea: Forward + Backward

Forward pass:   compute the output (loss)
Backward pass:  compute gradients using chain rule

The chain rule says: if L = f(g(x)), then dL/dx = (dL/df) × (df/dg) × (dg/dx). Autograd applies this automatically at every node in the graph.

Real Example: PyTorch

import torch

x = torch.tensor(2.0, requires_grad=True)
w = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)

# Forward pass
out = (x * w + b) ** 2   # = (6 + 1)² = 49

# Backward pass (compute all gradients)
out.backward()

print(x.grad)   # dL/dx = 2*(x*w+b) * w = 2*7*3 = 42.0
print(w.grad)   # dL/dw = 2*(x*w+b) * x = 2*7*2 = 28.0
print(b.grad)   # dL/db = 2*(x*w+b) * 1 = 2*7*1 = 14.0

Let’s visualize how those gradients flow backward:

The gradient starts at 1 at the loss node, then multiplies through each operation on the way back. That’s the chain rule in action.

Autograd With Multiple Values (Tensors)

In practice, you work with tensors (arrays), not single numbers. Autograd works exactly the same way — just across all elements simultaneously.

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
w = torch.tensor([0.5, 1.0, 1.5], requires_grad=True)

loss = (x * w).sum()   # dot product → scalar

loss.backward()

print(x.grad)   # tensor([0.5, 1.0, 1.5])  ← = w
print(w.grad)   # tensor([1.0, 2.0, 3.0])  ← = x

Key rule: .backward() only works directly on a scalar. For non-scalar outputs, pass a gradient tensor of the same shape as the argument.

How to Clear Gradients

This is one of the most common mistakes for beginners. Gradients accumulate — they add up across .backward() calls. You must clear them manually before each training step.

optimizer.zero_grad()   # ← always do this first!
loss = model(x)
loss.backward()
optimizer.step()

Summary

Here’s everything in one place:

The one-line mental model:

Autograd watches every operation you do on a tensor, builds a graph of them, then walks that graph backwards — multiplying local derivatives together (chain rule) — to give you ∂Loss/∂weight for every single parameter. You just call .backward().

That’s the engine behind all of deep learning training.