Automatic Differentiation for Beginners

If you’ve ever trained a neural network, optimized a function, or dabbled in scientific computing, you’ve probably heard the term “automatic differentiation” (often shortened to autodiff or AD). It’s the magic behind frameworks like PyTorch, TensorFlow, and JAX. But what is it, really?

In this post, we’ll build up the concept from scratch, with small examples you can follow by hand. No prior knowledge of machine learning is required, just some familiarity with basic calculus.

Why are derivatives needed?

Derivatives tell us how a function changes. If we have a function f(x), its derivative f′(x) tells us the slope at any point, which is how much the output changes when we increase the input slightly.
This is useful because many problems in science and engineering boil down to optimization: finding the input that minimizes (or maximizes) some function.
A classic example is training a neural network, where the goal is to find weights that minimize prediction error. The main algorithm used for this purpose is gradient descent, which requires computing derivatives of the error function with respect to every individual weight, sometimes numbering in the millions.
So the question becomes, how do we compute derivatives efficiently and accurately, especially for complicated functions?

Methods for Computing Derivatives

There are three commonly used approaches, and understanding their trade-offs is key to understanding the advantages of automatic differentiation.

1. Symbolic Differentiation
This is the method typically taught in calculus classes. Given f₁(x) = x²sin(x), you apply rules (product rule, chain rule) and get :
f'₁(x) = 2x sin(x) + x²cos(x)

Tools like Mathematica and SymPy do this automatically. The problem is that for complex functions, especially those defined by code with loops and branches, symbolic expressions can grow rapidly in size. A short program may produce a derivative formula that spans many pages, a phenomenon known as expression swell.
Consider what happens when a function is defined by a loop. Take the logistic sigmoid f₁(x) = 1 / (1 + exp(-x)) and compose it with itself n times, something a recurrent network or any iterative algorithm does naturally:

def f(x, n):
    for _ in range(n):
        x = 1 / (1 + exp(-x))
    return x

The program is O(n) in length. But every time we compose f with itself, the chain rule fires and the symbolic system must substitue the full expression for f into itself. At depth 2, the derivative is f'₂(x) = f'₁(f₁(x)).f'₁(x) , which already expands to a sizeable fraction:

At depth 3, the entire expression is substituted into itself again. The size of the formula does not grow linearly with n, instead, it grows roughly exponentially, because each new level wraps the previous expression inside it. A loop with 10 iterations and only a few characters of code can produce a derivative formula containing millions of symbols. This is the phenomenon known as expression swell, the formula that the symbolic system must manipulate becomes so large that it is effectively unmanageable, even though the underlying computation is simple.

2. Numerical Differentiation
This uses the definition of a derivative directly:

To see it in action, take f(x) = x² at x = 3. The exact derivative is f’(3) = 6. with h = 0.001, we have:
f’(3) ≈ [f(3.001) - f(3)] / 0.001 = [9.006001 - 9] / 0.001 = 6.001

For small values of h, this method works, but two issues generally arise. First, there is a trade-off in precision. If h is chosen too large, the approximation becomes inaccurate; if it is too small, floating-point errors become significant. Second, if the function has n inputs, computing the full gradient requires n+1 function evaluations. For a small function with three inputs, this is trivial, but for a neural network with one million parameters, it would require one million plus one evaluations per gradient. This quickly becomes unworkable.

3. Automatic Differentiation
Autodiff is the best of both worlds: it’s exact (like symbolic) and efficient (unlike numerical). The key idea is that any computation, no matter how complicated, is built up from simple operations (addition, multiplication, sin , exp , and so on). The derivatives of these elementary operations are known. By applying the chain rule in a systematic manner during the computation, it is possible to compute exact derivatives together with the function value.
This captures the essence of automatic differentiation. We now unpack it in more detail.

The chain rule:
If y = f(g(x)) , then:

In words, to get the derivative of a composition, multiply the derivatives of the pieces.
For example, if y = sin(x²), we can think of it as y = sin(u) where u = x².
Then:

This rule generalizes beautifully to chains of many operations, which is exactly what computer programs are.

Computational graphs
A program’s computation can be represented as a computational graph, where nodes are operations and edges represent data flow.
Consider the function:

we can break this into small steps:

1. a = x + y
2. b = sin(x)
3. f = a ⋅ b

This can be represented as a computational graph as shown below:

The graph has inputs x and y feeding into intermediate nodes a and b, which feed into the output f . Autodiff works by traversing this graph and applying the chain rule at each node.

There are two directions you can go: forward and backward. Both give the exact same answer, but they differ in efficiency depending on the shape of your problem.

Forward-mode Autodiff

Forward mode is the more intuitive of the two. We walk through the graph from inputs to outputs, carrying along both a value and a derivative at each step.
Let’s compute f(x , y) = (x + y) ∙ sin(x) and it’s derivative with respect to x at the point x = 2, y = 3.

We start by tagging our inputs with derivatives. Since we’re differentiating with respect to x, we say x˙ = 1 (the derivate of x with respect to itself) and y˙ = 0 (the derivative of y with respect to x).

Now we walk through:

Step 1: a = x + y = 2 + 3 = 5
Derivative: a˙ = x˙ + y˙ = 1 + 0 = 1.

Get Bwhiz’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Step 2: b = sin(x) = sin(2) ≈ 0.909
Derivative: b˙ = cos(x) ∙ x˙ = cos(2) ∙ 1 ≈ -0.416.

Step 3: f = a ∙ b ≈ 5 ∙ 0.909 ≈ 4.546

The derivative using the product rule : f˙ = a˙ ∙ b + a ∙ b˙ ≈ 1 ∙ 0.909 + 5 ∙ (-0.416) ≈ -1.171.
So ẟf(2,3) / ẟx ≈ -1.171.

Recall that a program can be viewed as a computational graph, where each node is an operation and edges represent data flow. In forward-mode automatic differentiation, we augment this graph so that every value also carries information about how it changes with respect to the input. At each node, we compute both the value and its sensitivity to a chosen input simultaneously. This is known as the dual number approach, where each quantity is extended with an additional component that tracks its derivative.

The key point is that this is still just a single forward pass through the graph. However, to compute derivatives with respect to a different input (for example, y instead of x), we must repeat the entire computation with different initial sensitivities, such as x˙ = 0 and y˙ = 1. In general, forward mode requires one pass per input variable. This is manageable for functions with a small number of inputs, but becomes impractical for high-dimensional problems such as neural networks with millions of parameters.

Reverse-mode Autodiff

Now, reverse mode flips the direction. Instead of asking “how does each intermediate variable change as x changes?”, we ask “how does the output change as each intermediate variable changes?”

Given that we start at the output f , we compute for each node v in the graph the quantity ẟf/ẟv called the adjoint. It is seen that the adjoint of f is 1, since f changes 1-for-1 with itself. We then recursively propagate these adjoint values backward through the graph.

Let’s revisit the same example: f(x , y) = (x + y) ∙ sin(x) at x = 2, y = 3.

In the forward pass we just compute values, we dont do derivatives yet:

• a = x + y = 5
• b = sin(x) ≈ 0.909
• f = a ∙ b ≈ 4.546

For the backward pass, we propagate adjoints from output back to inputs.

Starting at the output. The output node is f itself, so it changes one-for-one. so the adjoin of f is 1 (ẟf/ẟf). This is our starting point.

At the node f = a ∙ b, we have two parents, a and b. We want to know how f depends on each of them, which we get from the partial derivatives of this operation:

Now computing the adjoint for both a and b :

These tell us: “if a changes by a tiny amount, f changes by 0.909 times that amount,” and similarly for b.

For the b = sin(x) node, we know the adjoint of b = 5. This means a unit change in b produces a 5-unit change in f . Now we want to know how much f changes when x changes.
The local derivative at this node ∂b/∂x = cos(x). At x = 2, that’s cos(2) ≈ -0.416. This means a unit change in x changes b by -0.416.

To chain the information above together: a unit change in x changes b by -0.416 and a unit change in b changes f by 5. So a unit change in x (through this specific path) changes f by:

This gives us the contribution of b-path to the adjoin of x. Now we dont write the adjoint of x yet because x is used in two places in this graph (it feeds into a and into b), and we need to account for both before we’re done.

Visiting the a = x + y node. The local partials are both 1 (the derivative of x + y with respect to either variable is just 1). So:

Finally we sum the contributions. This is critical for reverse mode: when a variable is used in multiple places, sum all the contributions to get its adjoint. This is a direct consequence of the multivariable chain rule, if x affects f through several paths, the total effect is the sum of the effects along each path.

x was used in two places (in a and b), so:

y was used in only place (in a), so:

Notice that the adjoint of x is -1.171 and it matches the ∂f/∂x we got from the forward mode earlier. Both modes are computing the same mathematical derivative, they just traverse the graph in opposite directions.

In a single backward pass, we got the derivatives with respect to both inputs. Forward mode would have needed two passes, one per input. With two inputs the difference is trivial, but with a million inputs (the typical neural network), reverse mode is a million times cheaper.
Although it is worth mentioning that there’s a downside to reverse mode, because we need all the forward-pass values during the backward pass, we have to store every immediate. This is why training large neural networks is memory-hungry. Forward mode doesn’t have this issue, it computes values and derivatives together and can discard them as it goes.

Conclusion

Automatic differentiation is one of those ideas that seems mysterious at first, but becomes almost obvious once you see the mechanics. Its core insights are:

• Every computation can be decomposed into simple operations whose derivatives are known.
• The chain rule allows us to combine these local derivatives into derivatives of the overall computation.
• By propagating derivatives through the computational graph, either forward from inputs or backward from outputs, we can compute exact gradients efficiently.

Taken together, these ideas show that automatic differentiation is not mysterious at all. It is simply the systematic application of the chain rule across a sequence of elementary operations. What once seemed like a complex trick reduces to careful bookkeeping of derivatives, carried out in a way that scales to even the largest computations.