Backpropagation Made Simple: Understanding Neural Network Calculations

Basics of Neural Network Part 2— Forward Pass, Backward Pass and Chain Rule

Ever wondered how neural networks “learn” from data? It all comes down to two key steps: the forward pass, where the network computes outputs, and the backward pass, where it calculates gradients to improve its predictions. Understanding these steps is crucial for grasping how deep learning models optimize themselves — and it’s not as complicated as it sounds. In this post, we break it down with simple examples and clear explanations.

The Setup

We’re trying to compute a function: J = 3(a+bc)

where a, b and c are input values. To compute J, you naturally break it into smaller sub-steps (intermediate results):

• u = bc
• v = a+uv
• J = 3v

Each step depends on the previous one — this chain of computations is what we represent using a computation graph. The computation graph:

• Organizes your computation into small, differentiable pieces.
• Clearly shows dependencies between variables.
• Lets you do both forward propagation (compute output)
  and backward propagation (compute gradients) efficiently.

Example Calculation

To make this tangible, let’s plug in some numbers: a=5, b=3, and c=2. The table below shows how the values flow through our three steps to produce a final answer.

✅ Final output: J = 33

This left-to-right sequence of calculations — starting with inputs and ending with the final output — is the forward pass.

Forward Pass

The forward pass means: Compute the output J from the inputs (a, b, c).

Get Indigo Shadow’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In a neural network, this corresponds to:

• computing activations layer by layer
• until we get the final prediction or cost value J.

Backward Pass

Once we have J, we often want to compute gradients — that is, “How much does J change if we change a, b, or c (and consequently u and v)?”. This is represented as:

That’s what backpropagation does: It moves right-to-left through the computation graph, applying the chain rule to compute derivatives.

Step 1. Compute dJ/dv

From the equation J = 3v: If v increases a little, J increases 3 times as much.

In Python code, we might store this as:

dv = 3  # shorthand for dJ/dv

Step 2. Compute dJ/da

a appears in the equation v = a + u. So increasing a increases v equally —

and since we already know that

we can apply the chain rule:

So:

da = 3  # shorthand for dJ/da

Step 3. Compute dJ/du

u also appears in v = a + u, so similarly:

and again, using the chain rule:

So:

du = 3  # shorthand for dJ/du

Step 4. Compute dJ/db

Now, b affects u, because u = bc. So:

Then, using the chain rule again:

So:

db = 6  # shorthand for dJ/db

Step 5. Compute dJ/dc

Similarly, c affects u through u = bc:

So:

dc = 9  # shorthand for dJ/dc

🧠 Why this matters

This small example is exactly how backpropagation works in a neural network:

• Each node performs a simple computation.
• During the forward pass, you compute all the outputs.
• During the backward pass, you use the chain rule to compute derivatives efficiently, starting from the output and moving backward.

This is why frameworks like TensorFlow and PyTorch can automatically compute gradients — they build and traverse a computation graph like this under the hood.

By following the forward and backward passes through a simple computation graph, you can see exactly how neural networks compute outputs and gradients step by step. The backward pass, guided by the chain rule, allows the network to efficiently learn by updating its parameters. Once you understand these principles, the “magic” behind deep learning starts to feel a lot more logical — and far less mysterious.