Gradient Descent: How AI Improves Step by Step

An AI model makes a prediction. The prediction is wrong. We calculate how wrong it is using a loss function. But then comes a more important question — What should the model change so that the next prediction is better?

This is where gradient descent comes. Gradient descent is one of the fundamental mathematical ideas behind machine learning. It gives an AI model a systematic way to move from wrong prediction to adjustment of parameters to make it slightly less wrong. And it repeats this process thousands, millions or sometimes billions of times.

That repetitive improvement is what we call training.

A Real-Life Problem - Finding the Best Price

We run a small food delivery business. We sell a meal at ₹200. At this price, profit is not great. We try ₹220. Profit improves. We try ₹240. Profit improves again. We try ₹260. Sale falls and Profit falls. So somewhere between ₹240 and ₹260 there may be a better price. We are trying to solve an optimization problem. Mathematically, we could imagine a function — Loss = f(price) where loss represents how far our current pricing decision is from the best possible pricing decision. Our goal is simple — Find the price where loss is minimum. Machine learning faces essentially the same problem. Instead of adjusting one price, a neural network may be adjusting millions or billions of parameters. Gradient descent tells the model which direction to move.

In this previous article we discussed loss functions -

https://medium.com/@piyas.de/loss-functions-how-ai-knows-about-the-wrongness-of-the-prediction-9881f6b5960c

Suppose a model predicts — Actual value = 10 and Prediction = 7
The prediction is wrong by 3.

One possible loss function is squared error — Loss = (Prediction — Actual)² Therefore — Loss = (7- 10)² = (-3)² = 9. So we know the model is wrong. But knowing that the loss is 9 is not enough. The model needs to know — Which parameter should change? and In which direction? and By how much?

That is the job of gradient descent.

Start With One Parameter

Suppose our model is extremely simple — Prediction = weight × input.

Let input = 2, actual answer = 10, Initially weight = 3 and the model predicts — prediction = 3 × 2 = 6. The correct answer is 10.

Using squared error — Loss = (6–- 10)² = 16. So our weight of 3 is not ideal. If the weight were 5, Prediction = 5 × 2 = 10 and Loss = 0

Therefore, mathematically, the learning problem becomes — How do we move the weight from 3 toward 5?

Turn the Problem Into a Function

Our model is Prediction = wx where w = weight, x = input.

Loss is L = (wx — y)² where y = correct answer using x = 2 and y = 10 we obtain L(w) = (2w — 10)². So now loss depends entirely on `w`. Let us calculate several values.

Here Loss decreases as the weight approaches 5. Then it increases again. If we plotted it, we would get a bowl-shaped curve. The bottom of the bowl represents the minimum loss. Our job is to reach that bottom.

How Does the Model Know Which Way Is Down?

While standing on the side of a hill in thick fog, we cannot see the valley. We can feel the slope underneath our feet. If the ground slopes downward toward our right, we move right. If it slopes downward toward our left, we move left. We repeat like -

Measure slope
→ move downhill
→ measure slope again
→ move downhill again

Eventually we reach a low point. Gradient descent works almost like this. Here the gradient tells us the slope.

The Mathematics of the Slope

Our loss function is L(w) = (2w — 10)² We expand it L(w) = 4w² — 40w + 100
We differentiate with respect to `w`: dL/dw = 8w — 40

This derivative tells us how loss changes when the weight changes. At
w = 3, we get — dL/dw = 8(3) — 40 = 24 –40 = -16
So the gradient is negative. That tells us something important. Increasing `w` will decrease the loss. So we need to move the weight upward. Now suppose w = 7, Then dL/dw = 8(7) — 40 = 56 –40 = 16

Now the gradient is positive. Therefore we should decrease the weight. Notice the property of the derivative. On one side of the minimum,
gradient < 0 and on the other gradient > 0 At the minimum —
gradient = 0.

In our example — 8w — 40 = 0=> 8w = 40=> w = 5 is the weight that produces zero loss.

The Gradient Descent Equation

Gradient descent uses the following update rule

new weight = old weight — learning rate × gradient

Mathematically w_new = w_old — η × dL/dw — The Greek letter `η`, pronounced eta, represents the learning rate. This equation contains the central mechanism of machine learning. Let us break it down.

Current parameter is w_old Where the model is now. Gradient is dL/dw. The direction and steepness of the loss function. Learning rate is η — How large a step the model takes.

Now Updated parameter is w_new Where the model moves next.

Let us Perform Gradient Descent Manually

Start with w = 3. We already calculated gradient = -16. If learning rate = 0.1
Then new weight = 3 + –0.1 × (-16) => = 3 + 1.6 => = 4.6

Our model immediately moves from w = 3 to w = 4.6. Let us calculate the new prediction. prediction = 4.6 × 2 = 9.2. Loss becomes (9.2–10)² = 0.64
Previously Loss = 16 now becomes Loss = 0.64

One mathematical step produced a dramatic improvement. At w = 4.6 gradient becomes 8(4.6) — 40 = 36.8 — 40 = -3.2. Updating again —
w_new = 4.6–0.1(-3.2) = 4.92 and Prediction is 4.92 × 2 = 9.84
Here Loss is (9.84 — 10)² = 0.0256.

So we have moved from Loss 16 -> Loss 0.64 -> Loss 0.0256 and The model is learning. One more point here. The first update was large 3 → 4.6. The next update was smaller 4.6 → 4.92. Because the gradient is getting smaller.

Far from the minimum, the slope is steep.

Near the minimum, the slope becomes shallow.

That naturally causes gradient descent to slow down as it approaches a good solution. This is one of the elegant mathematical properties of optimization.

Why Do We Subtract the Gradient?

The gradient points in the direction where the function increases fastest. We want loss to decrease. So we move in the opposite direction. Hence
parameter = parameter — learning_rate × gradient. If the gradient is positive — parameter decreases and if the gradient is negative — parameter increases. Either way, the update tries to move to accuracy.

The Learning Rate Is Extremely Important

Now consider the learning rate. If ,during training, Learning rate will be too small and the model learns very slowly. When learning rate is reasonable, The data may be like -

3.0
4.6
4.92
4.984
…
It converges efficiently.

For learning rate too large, the data may be -

3
6.2
1.1
9.0
-3.8
…

The parameter may oscillate or even diverge. So the learning rate controls the speed and stability of learning.

A Statistical Interpretation

So far we have used one observation — x = 2, y = 10
Real machine-learning systems do not usually learn from one observation.

If we have -

We may build a model — Sales prediction = w × Advertising Spend. Each row produces its own error.

For observation `i` — error_i = prediction_i — actual_i. Instead of optimizing one error, we usually optimize an average. For example, Mean Squared Error — MSE = (1/n) × Σ(prediction_i — actual_i)² . Gradient descent therefore tries to find parameters that reduce the average error across the dataset. This is where optimization meets statistics. The model is not trying to perfectly memorize one observation. It is trying to find parameters that statistically explain many observations well.

Why This Matters for Generalization

Suppose a student memorizes the answers to 100 mathematics questions. That does not necessarily mean the student understands mathematics. Similarly, a model that perfectly reproduces its training examples may not perform well on new examples. Training therefore becomes a statistical balancing problem. We want parameters that reduce expected error not just on known data, but also on unseen data.

Conceptually, we want to minimize — Expected Loss = E[L(Y, Ŷ)], where `E` represents expectation. But the true population distribution is unknown. We therefore estimate it using training data. This is why datasets, sampling, variance and statistical generalization become so important in machine learning.

Batch Gradient Descent

We have one million training examples. One approach is -

1. calculate predictions for all one million examples
2. calculate their losses
3. calculate the average gradient
4. update the model

This is called batch gradient descent. The advantage is that the gradient estimate uses the entire dataset. The disadvantage is computational cost.Modern AI datasets can contain enormous amounts of information. Waiting for the entire dataset before every update would be inefficient.

Stochastic Gradient Descent

At the opposite extreme, we could select one training example. Calculate its loss. Calculate its gradient. Update the model immediately. This is Stochastic Gradient Descent, or SGD. The word stochastic essentially tells us that randomness is involved. Instead of calculating the exact gradient across the whole dataset, we use a randomly selected sample to estimate it.

Statistically, this introduces noise.

If the true gradient is approximately 0.42, individual observations might produce

0.61
0.19
0.55
0.33
The estimates fluctuate. But over many observations, they can still provide useful information about the general direction.

Mini-Batch Gradient Descent

Neural networks normally use a compromise. Instead of 1 observation
or entire dataset we select a small batch.

Get Piyas De’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For example -

32 examples
64 examples
128 examples
256 examples

For each batch -

Forward pass
→ calculate predictions
→ calculate loss
→ calculate gradients
→ update parameters

Then the next batch arrives. This is mini-batch gradient descent. It gives us a balance between computational efficiency and statistical stability.

From One Weight to Millions of Parameters

Our example contains one weight w A neural network contains many. Suppose there are three — w₁, w₂, w₃. The loss becomes a function
L(w₁, w₂, w₃). We calculate a partial derivative for every parameter — `
∂L/∂w₁, ∂L/∂w₂, ∂L/∂w₃. Together these form the gradient vector — ∇L
For example ∇L = [0.7, -0.2, 1.1]

That tells us:

w₁ should decrease
w₂ should increase
w₃ should decrease

The update becomes θ_new = θ_old — η∇L where `θ` represents all model parameters. The simple one-dimensional downhill walk has now become a walk through a multidimensional mathematical landscape.

The Landscape of a Neural Network

A neural network may contain millions or billions of parameters. Therefore its loss function is not a simple two-dimensional curve. It is better imagined as an extremely high-dimensional surface. Each combination of parameters corresponds to a different position in this landscape. The height represents loss.

Training means searching this enormous mathematical space for parameter combinations that produce sufficiently low loss. We cannot visualize such a surface directly. But the underlying idea remains the same:

Calculate slope
→ take a step
→ recalculate
→ take another step

Where Does the Gradient Come From in a Neural Network?

A neural network performs many mathematical operations

inputs
→ weighted sums
→ activation functions
→ more weighted sums
→ probabilities
→ loss

The final loss depends indirectly on every earlier weight. Calculus gives us the chain rule to trace that relationship backward. This process is called backpropagation. So here -

Forward pass:
parameters → prediction → loss

Backward pass:
loss → gradients for every parameter

Optimizer:
gradients → updated parameters

Gradient descent and backpropagation are closely related, but they are not identical. Backpropagation calculates the gradients. Gradient descent uses those gradients to change the parameters.

A Python Experiment

Let us run one example -

weight = 3.0
x = 2
y = 10
learning_rate = 0.1
for step in range(10):
prediction = weight * x
loss = (prediction - y) ** 2
gradient = 2 * (prediction - y) * x
weight = weight - learning_rate * gradient
print(
 step,
 round(weight, 4),
 round(loss, 4)
 )

We should see the weight move toward 5 while the loss approaches to 0. Here this line performs the model prediction — prediction = weight * x. This calculates squared-error loss — loss = (prediction — y) ** 2
The derivative of (wx — y)² with respect to `w` is 2(wx — y)x. Therefore
gradient = 2 * (prediction — y) * x and finally gradient descent performs the parameter update weight = weight — learning_rate * gradient

The program simply repeats mathematics.

The Learning Process

Try this variation -

weight = 3.0
x = 2
y = 10
learning_rate = 0.1
for step in range(8):
prediction = weight * x
 loss = (prediction - y) ** 2
 gradient = 2 * (prediction - y) * x
print(
 f"Step {step}: "
 f"weight={weight:.4f}, "
 f"prediction={prediction:.4f}, "
 f"loss={loss:.6f}, "
 f"gradient={gradient:.4f}"
 )
weight -= learning_rate * gradient

As training progresses -

weight → 5
prediction → 10
loss → 0
gradient → 0

Those four movements summarize the optimization process.

Experiment With the Learning Rate

Now change — learning_rate = 0.1 to learning_rate = 0.01. Training becomes slower. With learning_rate = 0.3 and learning_rate = 0.5 training will be faster. This experiment teaches something important. AI training is not only about finding a gradient.

The size of the update matters also.

How Does This Connect to Generative AI?

Imagine training a language model. The model receives — The capital of France is … Let us say its predicted probabilities are

Paris 0.31
London 0.26
Berlin 0.19
Rome 0.12
Other 0.12

The correct next token is Paris. The loss function tells the model how poor its probability distribution was. Backpropagation calculates how each parameter contributed to that loss. The optimizer then updates those parameters. After many training examples, the probability distribution may become

Paris 0.82
London 0.06
Berlin 0.04
Rome 0.03
Other 0.05

This improvement did happen because — parameters were repeatedly updated from statistical patterns in training data. Gradient-based optimization gradually shaped the model’s internal representation.

One Training Example Barely Changes a Large Language Model

This is an important point. A generative model does not normally become intelligent because of one dramatic update. Its learning is distributed across enormous numbers of small parameter adjustments. Individual updates may be extremely small. Collectively they create complicated statistical behavior. This is one reason the title of this article matters — AI improves step by step.

Gradient Descent Is Really an Error-Correction Loop

We can now describe machine learning as a feedback system.

Input -> Model -> Prediction -> Compare with truth -> Loss ->
Gradient -> Parameter update -> New model -> Repeat

The model is continuously being told “Given your current parameters, your predictions produce this much error. Here is a direction that should reduce it.” Training repeats this correction process.

The Mathematics Behind “Learning”

When people say “The AI learned.” — the phrase can make the process sound complex. At a simplified mathematical level, learning means —
θ(t+1) = θ(t) — η∇L(θ(t))

In words — Take the current parameters, calculate how the loss changes with them, and move the parameters slightly in the direction that should reduce the loss.

Repeat it enough times across enough representative data, and a model can learn complicated statistical relationships.

Statistics Tells Us What We Are Optimizing

Calculus tells us — How should the parameters move?

Statistics tells us — What does good performance actually mean across data?

Here we are interested in minimizing something closer to Expected future error. Training data provides a sample from the underlying population. The loss calculated on that sample becomes an estimate of how well the model captures the broader pattern. Gradient descent then optimizes that estimate. This is why mathematics, statistics and machine learning cannot be separated.

One Simple Mental Model

Whenever we hear the phrase gradient descent, we may remember this story.

It is Mathematically -

Measure error
→ calculate slope
→ move parameters
→ measure error again

That repeated process is one of the foundations of modern artificial intelligence. So Loss Function answered about the wrongness of AI. In current topic — after discovering the error — AI calculates gradients and changes its parameters.

So our learning chain now looks like this:

Data
↓
Prediction
↓
Loss
↓
Gradient
↓
Parameter Update
↓
Better Prediction

Here we have moved one step closer to understanding how neural networks actually train.

Final Thought

AI training can look magical when viewed from the outside.

Billions of parameters change. Language models improve. Images become more realistic. Predictions become more accurate.

But beneath all of this is an understandable idea -

Make a prediction.
Measure the mistake.
Calculate which direction reduces the mistake.
Take a small step.
Repeat.

A single gradient-descent step is simple. Millions of mathematically coordinated steps can produce something extraordinarily powerful.

And that is how AI improves — step by step.