Gradient descent -Part1

How does a machine learn from its mistakes using gradient descent ?

Why gradient descent at all?

Picture a machine with one knob. You feed it a 2. It is supposed to output a 10. The knob is at 3, and the machine multiplies, so it outputs 6. The expected value is 10. but it returned 6. The output needs to climb by 4.

The part that trips people: you know the right answer, but not the right knob setting. Those are different things. “The output should be 10” is given. “The knob should be at 5” is what you have to find. The answer key talks about the output. It says nothing about the knob.

You also cannot look inside. A real network has millions of knobs wired to each other, and you have no map. All you can do is turn a knob a little and see whether the output got better or worse.

A useful picture: you are on a hillside in thick fog. You want the valley floor. You cannot see the valley, the hill, or three feet ahead. You can feel which way the ground tilts under your feet.

So you take a small step downhill. Feel again. Step again. Repeat.

You reach the bottom not because you knew where it was, but because “downhill” was available at every point.

Gradient descent is that walk, done with arithmetic. The tilt is the gradient. Everything else is detail.

What would you do instead?

The alternatives are why this looks less like a clever trick and more like the only door out of the room.

Option 1: guess randomly. Set the knob, check the error, keep the change if it helped. One knob: you might stumble into something decent. A thousand knobs: you will never hit a good combination in the lifetime of the universe. Random guessing dies as soon as you have more than a couple of knobs.

Option 2: try every value. Sweep from −1000 to +1000 in steps of 0.001 and keep the best. Two million tries; a computer does that instantly. Solved for one knob. Every new knob multiplies the work:

A small network has thousands of knobs. A language model has hundreds of billions. Trying combinations is not slow. It is not a thing that can happen.

Option 3: solve it with algebra. For the toy machine you can. “Find the knob where the error is zero” is one line and gives w=5w = 5w=5 exactly.

Stack a few layers with nonlinear functions between them and “solve for the knobs” becomes billions of tangled equations with no closed form.

What is left is the fog. You cannot see the landscape, but you can always measure the slope where you are standing. That measurement is cheap, exact, and — this is the crucial part — its cost does not explode as you add knobs. One pass through the network gives the slope for every knob at once. A hundred billion of them, in one go.

That is the trade. Give up the global view. Keep the local slope. Take a lot of steps.

The formula

In words, before symbols:

new setting  =  old setting  −  (step size × slope)

That’s it. That’s gradient descent. In symbols:

Four pieces:

Three things to notice, because they’re where the cleverness hides.

The minus sign does the thinking. You never decide whether to add or subtract. The slope comes out signed, and subtracting it always moves downhill:

• Positive slope: increasing the weight raises the loss, so we decrease the weight. ✓
• Negative slope: increasing the weight lowers the loss, so we increase the weight. ✓
• (Here weight means the model that stays inside that black box)

Same line of code, both directions, no if statement. That's the trick.

The step is not the learning rate. The step is the learning rate times the slope. On a steep slope the step is large. Near the bottom the slope shrinks, so the steps shrink with it. Gradient descent brakes on its own. Nobody programs that in. It falls out of multiplying by the slope.

The learning rate is temperament. The ground tells you how steep it is. That part is not yours. The learning rate is how much stride you give per unit of steepness.

Send three hikers down the same hill, feeling the same ground:

🐢 The timid one (η too small) The timid one uses a learning rate that is too small. Even where the ground plunges away, they inch forward. They will reach the bottom — sometime next week.

🚶 The sensible one (η about right) The sensible one uses a learning rate that is about right. They stride out where it is steep and shorten up as it levels off. Down in a dozen steps.

🏃 The reckless one (η too big) The reckless one uses a learning rate that is too large. They feel a steep patch and leap. They sail clean over the valley and land higher up the far side than where they started. They feel an even steeper patch there. They leap again, harder. They are now going up the mountain, at speed, forever

Nobody handed them different information. The hill was the same for all three, and all three read the slope correctly. The only difference is how aggressively each one converts “steep” into “stride.”

The gradient can be computed perfectly and training can still fail. Learning rate is not smarter information about the loss surface. It is only how aggressively you turn “this way is downhill” into a change in the weights. Too small, and you crawl. About right, and you descend. Too large, and you overshoot the valley and diverge. Same hill. Same slope. Different temperament.

The slope is measured, not guessed. This isn’t a heuristic or an estimate. Calculus gives the exact tilt at the exact point you’re standing. The only guesswork in the entire system is where you started and how big

Watching it actually run

Let’s make the fog concrete. Here’s our error, as a function of the knob:

L(w)=(w−4)²

Its slope at any point is 2(w−4). We’ll use a step size of η=0.3.

You can see the valley is at w = 4. The algorithm cannot. It only ever gets to ask one question: what's the slope right here?

Starting too low, at w = 1

The Knob is converging at 4.

Starting too high, at w = 10

Same formula. Nothing changed. We just dropped it somewhere else on the hill.

Both runs land on 4, from opposite directions, with identical code. One walked up, one walked down. Neither was told which way to go, and neither was told where 4 was. The slope handled it.

Look at the error columns: 9 → 1.44 → 0.23 → 0.037, and 36 → 5.76 → 0.92 → 0.147. Both collapsing fast, and both decelerating as they arrive, exactly as promised.

Here’s the whole thing, runnable:

w, eta = 1.0, 0.3            # try 10.0 too

for step in range(6):
    error = (w - 4) ** 2
    slope = 2 * (w - 4)
    print(f"step {step}:  w={w:7.4f}   error={error:8.4f}   slope={slope:+7.3f}")
    w = w - eta * slope      # ← the entire algorithm

Every neural network ever trained is that last line, wearing a lot of scaffolding.

Learning rate : The one thing you actually choose

The slope is handed to you. The step size η is yours, and it’s the difference between training working and training exploding. Same starting point, same problem, six different η:

That last row is the one to burn into memory. The slope was correct on every single step. The math was perfect. The steps were just too big, so each one flew past the valley and landed further up the opposite side than where it started — and then did it again, worse. This is what people mean when a training run “goes to NaN.” Nothing was wrong except one number that a human picked.

Get Anandha L Ranganathan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

That’s our reckless hiker, in a table.

If you have ever overcorrected a car, you already know this failure.

You drift a little to the right, so you yank the wheel left — too hard. Now you are pointed at the other lane, so you yank right, even harder. Every fix is bigger than the mistake that caused it. Half a minute later you are in a ditch.

Your reading of the road was not the problem. Each time, you knew which way you were off. The problem was how hard you turned the wheel.

Engineers call that hardness gain. Too much gain and a correcting system starts to swing, then blows up.

The learning rate is that gain. That is all it is.

And if one special value of it happens to land on the exact answer in a single step, that is luck for that one curved bowl. Change the problem and the same number can explode.

There is no learning rate that works for every job. That is why people tune it. That is why most training runs start with bigger steps and finish with smaller ones: bold early, careful near the bottom.

Two knobs, and where the chain rule comes in

One knob was a warm-up. Real models have more, and the moment you have two, something new is needed.

Give our neuron a second knob — a bias that just shifts the output up or down:

y=w⋅x+b

With input x = 2, target t = 10, starting at w = 3, b = 0.

Why one slope isn’t enough any more

We need to know how the error responds to w, and separately how it responds to b. Two knobs, two slopes. And here's the wrinkle:

Neither knob touches the error directly. The knobs make a prediction. The prediction makes the error. Nothing connects w to L except by going through y first:

    w ──┐
        ├──▶  y_hat  ──▶  L (y-y_hat)
    b ──┘   (prediction)  (error)

So “how does the error respond to w?" can't be answered in one move. It has to be answered in two hops, and that's precisely what the chain rule is for.

The chain rule, in one sentence

To get the slope across a chain, multiply the slopes of each link.

That’s it. If the error is twice as sensitive to the prediction, and the prediction is three times as sensitive to the knob, then the error is six times as sensitive to the knob. Sensitivities multiply as they pass down a chain.

How to picture it

Think of gears. Turn a small gear by one tooth; the gear it drives turns by three; the gear that one drives turns by two. Turn the first gear one tooth and the last one moves 3 × 2 = 6. You never needed a direct connection between first and last — you just multiplied along the linkage.

Or think of it as a message passed backwards. The error announces how upset it is, and that message travels back up the wires, getting scaled by each link it crosses:

w  ◀────────  y  ◀────────  L
        "×2"          "−8"
     (a 1 change     (raising y by 1
      in w moves      drops the error
      y by 2)         by 8)

Start at the error. Walk backwards. Multiply as you go. By the time you reach a knob, you have its slope: −8 × 2 = −16.

In derivative, this can be written as. We need to find ∂L/∂w but we can’t find, but using chaing rule in derivate we identify them.

That backwards walk has a name. It’s backpropagation. It isn’t a separate idea from the chain rule — it’s the chain rule, applied to a whole network, organised so that no work gets repeated.

Both knobs, in one pass

The beautiful part: the backwards message −8 gets computed once, then reused by every knob feeding into that point. Each knob just scales it by its own local link:

• w is connected through the input, so: −8 × 2 = −16
• b is connected directly, so: −8 × 1 = −8

Now update both, same formula, same step size (η=0.01):

Work out the slopes. Starting at w = 3, b = 0:

prediction   y = w × 2 + b  =  3 × 2 + 0  =  6
we wanted    t = 10         →  4 short

slope on the prediction  =  2 × (6 − 10)  =  −8      ← computed once
      slope for w  =  −8 × 2  =  −16      (w is multiplied by the input 2)
      slope for b  =  −8 × 1  =   −8      (b is added on directly)

Turn each knob. Same line, twice, with a step size of 0.01:

w:   3 − 0.01 × (−16)  =  3 − (−0.16)  =  3 + 0.16  =  3.16
b:   0 − 0.01 × (−8)   =  0 − (−0.08)  =  0 + 0.08  =  0.08

Look closely at what just happened. The rule says subtract — always, no exceptions. But the slope is negative, and subtracting a negative adds. Minus times minus makes plus, and the knob goes up, without a single line of code ever deciding to increase it.

Had the prediction come out too high instead — say 14 — the slope would have been +16, and the identical line gives 3 − 0.16 = 2.84. Down. Same arithmetic, opposite direction, nobody choosing.

Do that four times and you get the table above. The w knob:

The turns shrink every step — 0.160, 0.144, 0.130, 0.117 — with nothing arranging that. As the prediction closes in on 10 the slope shrinks, and the step is the slope times 0.01. The self-braking from earlier, visible as a column of numbers.

Here it is in code:

x, t, eta = 2.0, 10.0, 0.01
w, b = 3.0, 0.0

for step in range(4):
    y     = w * x + b              # forward:  make a prediction
    error = (y - t) ** 2           # how wrong
    dL_dy = 2 * (y - t)            # backward: one message...
    dL_dw = dL_dy * x              #   ...scaled for w
    dL_db = dL_dy * 1              #   ...scaled for b
    w = w - eta * dL_dw            # update, same line as before
    b = b - eta * dL_db            # ...once per knob

Two details worth pointing out:

w moves faster than b. Look at the columns — w travels 3.000 → 3.434 while b only goes 0.000 → 0.217, exactly twice as slow. That's the input x = 2 doing it: w gets multiplied by the input on its way to the prediction, so it has twice the leverage, so it gets twice the slope, so it takes twice the step. Knobs with more influence get moved more. Automatically.

Two knobs beat one. With only w, the error would have gone 16 → 13.54. With both, it goes 16 → 12.96. More knobs, more ways down, faster descent.

Picturing two knobs at once

With one knob, the error was a curve and you walked left or right along it.

With two, it’s a surface — a bowl, with w running east-west and b running north-south, and height being the error. You're a marble on the inside of that bowl.

The two slopes are the tilt in each direction. Put them together and you get an arrow pointing straight downhill on the surface — and that arrow is what “the gradient” really means. Not a number: a direction in knob-space.

Add a third knob and the bowl becomes something in 4D that nobody can picture, which is fine, because the arithmetic never needed you to picture it. The gradient is just a list of slopes, one per knob, and the update is applied to each one independently:

for every knob i:
    knob[i] = knob[i] - eta * slope_for_knob_i

One knob or a hundred billion — that loop is unchanged.