Machine Learning Foundations & Gradient Descent

Machine Learning is a way for computers to learn from data and make predictions or decisions without being directly programmed. For example, if a child looks at many pictures of dogs and cats, they learn to tell the difference. In the same way, a computer can look at lots of pictures, text, or other data and learn to understand or recognize things on its own.

Machine learning can be described by the following image:

At first, computers/machines are given many inputs and their corresponding output. Then, computers learn on their own the relation between the input and output. The relation may be expressed as a function. For example, if the inputs are 1,2,3,4,.... and the corresponding outputs are 2,4,6,8,..... then a computer can understand that the output is twice the input. So the relation is y=2x. This is machine learning, where the computer receives the input x and output y. Then, it finds the relation f(x).

Types of Machine Learning:

ML can be classified into 3 types.

1. Supervised Machine Learning is when a model is trained using data that already has the correct answers (called labels). For example, imagine we have 200 images, 100 of dogs and 100 of cats. Each image is labeled so the computer knows which ones are cats and which ones are dogs. By looking at these labeled images, the computer learns the patterns that make a cat look like a cat and a dog look like a dog. Once trained, the model can then look at a new image and guess whether it’s a cat or a dog. This process is called supervised learning because the model learns with the help of correct answers.
2. Unsupervised Machine Learning is when a model is trained using data that does not have any labels. For example, imagine we have 200 images of animals, but we don’t tell the computer which ones are cats or dogs. The computer looks at all the images and tries to find patterns or similarities by itself. It might group similar images — for example, all cat images in one group and all dog images in another — even though it doesn’t know what a “cat” or “dog” is. This process is called unsupervised learning because the model learns without any labeled answers.
3. Reinforcement Learning is a type of machine learning where a model learns by interacting with an environment and getting rewards or punishments based on its actions. For example, imagine teaching a robot to play a game. At first, it doesn’t know what to do, so it tries different moves. If it makes a good move, it gets a reward (like points). If it makes a bad move, it loses points or gets nothing. Over time, the robot learns which actions lead to better rewards and which ones to avoid. This is called reinforcement learning because the model is trained by rewarding good behavior and correcting mistakes through trial and error.

Now, Supervised Machine Learning is generally divided into two main types.

1. Classification is used in machine learning when the goal is to predict categories or labels. For example, if we give a computer images of fruits labeled as “apple,” “banana,” or “orange,” it learns to recognize which fruit is in each image. After training, when we give it a new fruit image, it can tell whether it’s an apple, banana, or orange. So, classification helps the computer choose the correct class for new data.
2. Regression is a type of machine learning where the goal is to predict a continuous value. For example, imagine we have data about houses, including their size, number of rooms, and location, along with their prices. The computer looks at this data and learns the relationship between the house features and its price. After training, when we give the computer details of a new house, it can predict its price based on the patterns it learned. So, regression helps the computer predict numerical values, such as house prices, temperatures, or sales numbers.

Unsupervised machine learning can be categorized into four main types: clustering, association, anomaly detection, and dimensionality reduction.

1. Clustering is a type of unsupervised learning where the goal is to group similar data together. For example, imagine you have a collection of different-colored marbles, but you don’t know which color corresponds to which marble. Clustering can help the computer group the marbles into clusters like “red marbles,” “blue marbles,” and “green marbles,” just by looking at their colors. The computer doesn’t need to be told which color is which — it can figure it out on its own. So, clustering helps the computer group similar items together.
2. Dimensionality Reduction is a technique used to reduce the number of features in a dataset while keeping the most important information. For example, imagine you have a dataset with hundreds of features (like age, height, weight, income, etc.). Dimensionality reduction can help the computer find the most important features and discard the rest, making the data simpler and easier to work with. So, dimensionality reduction helps the computer simplify complex data by keeping the most relevant details.
3. Association Rule Learning is a method in unsupervised learning where the goal is to find relationships between different items in large datasets. For example, in a store, association rule learning might discover that people who buy bread are also likely to buy butter. The computer learns these patterns from transaction data and can then suggest items to customers. So, association rule learning helps the computer find patterns of items that often appear together.
4. Anomaly Detection is used to identify unusual or unexpected data that doesn’t fit the normal pattern. For example, in a bank, anomaly detection can help find unusual transactions that might indicate fraud. The computer looks at past transactions and learns what normal transactions look like. When a new transaction comes in that is very different from the usual ones, the computer flags it as an anomaly. So, anomaly detection helps the computer spot outliers or unusual behavior in data.

So,
⭕Labeled data ▶ Supervised ML
⭕Labeled data, and we have to predict class ▶ Supervised Classification Algorithm
⭕Labeled data, and we have to predict a value ▶ Supervised Regression Algorithm
⭕Unlabeled data ▶ Unsupervised ML
⭕Unlabeled data, and we need to find patterns or groups ▶ Unsupervised Clustering Algorithm
⭕Unlabeled data, and we need to reduce complexity ▶ Unsupervised Dimensionality Reduction Algorithm
⭕Unlabeled data, and we need to find frequent patterns or associations ▶ Unsupervised Association Rule Learning
⭕Data with normal behavior, and we need to detect unusual data ▶ Anomaly Detection
⭕Data with rewards and penalties, and we need to learn the best action ▶ Reinforcement Learning

Terminologies:

We need to understand some common terminologies to progress further.

Dataset: A collection of data that is used for training and testing a model. It contains both features and targets.

Train Set: A portion of the dataset used to train the model. The model learns from this data to make predictions.

Test Set: A portion of the dataset used to test the model after training. It helps to check how well the model performs on new, unseen data.

Model: A mathematical system or algorithm that makes predictions based on data. It learns patterns from the training data and applies them to make predictions.

Feature: An individual measurable property or characteristic of the data used to make predictions. For example, in a dataset of houses, features could include “size” and “location.”

Target: The value that the model is trying to predict or classify. For example, in a house pricing model, the target would be the “price” of the house.

Predict: The process of using the trained model to make an estimate or decision about new, unseen data.

Overfitting: When a model learns too much from the training data, including noise and random fluctuations, leading to poor performance on new, unseen data.

Underfitting: When a model is too simple and fails to capture the patterns in the training data, resulting in poor performance on both the training and test data.

Hyperparameters: These are Settings that you choose before training the model (like learning rate, number of layers in a neural network) that help guide the training process.

Validation Set: A portion of the dataset used to evaluate the model during training, helping to adjust hyperparameters and prevent overfitting.

How Machine Learning Training is Done:

Suppose we have a dataset of house pricing like the following (for simplicity, we are taking a small dataset & assuming housing price depends only on the size of the house):

Simple dataset

Now, at first, we will look at the dataset and decide what we have to do. As the dataset is labeled and we have to predict a value, we have to use a Supervised Regression Algorithm. As Linear Regression is the simplest, let’s use it for this problem. Here, Size is the feature, and Price is the target. We will split the dataset into training set and test set. Generally, 80% is kept for training and 20% for testing. Here, we will keep the first 5 rows for training and the last row for testing.

Get Shadman Sobhan’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In Linear Regression, we want to find the best-fitting straight line that predicts the target (Price) from the input (Size). The line has the formula: y = mx + c, where m is the slope of the line — how much the price changes with size, c is the intercept — the price when size is zero.

We start by making a guess — random values for m and c. Using these, we predict the y values (prices) for each x (size) in our training data. But these predictions won’t be perfect at first. So, we measure how far off our predictions are using something called the Loss Function. A common one is Mean Squared Error (MSE):

Suppose we take m=2, c=4. The outputs will be 6, 14, 24, 34, and 44. The MSE will be 642.6. If we change m & c to 3, 6 it will be 9, 21, 36, 51, and 66. MSE will be 130.8. So, errors are reduced, and we are going in the right direction. In this way, various values of m & c can be checked to find the most optimum value that makes the MSE value the lowest. That is our desired value. In this case, ideally, m=4 & c=5. So y = 4x + 5 is the relation between input and output; this is our model. In machine learning, rather than taking random values to check whether the loss is lowest, we use an algorithm called Gradient Descent.

Differentiation to Find The Lowest Value:

Let’s say we have a curve defined by:y = 9x² +6x +50

You can think of y as the loss and x as a feature. Our goal is to find the value of x that minimizes the loss. The curve is as follows:

If f(x) is a function of x, then the value(s) of x where f(x) reaches its lowest point (minimum) will satisfy f′(x)=0. f′(x) is f(x) differentiated with respect to x.

f(x)= y = 9x² +6x +50. Then the derivative (i.e., slope or gradient) is: f'(x)= y1 = 18x+6. For the minimum value of y, 18x + 6 = 0 or x = -1/3. Now, if we the plot of f'(x) is :

If we use differentiation, we have to find the value of x where f’(x) is 0.
Let’s say the computer is at x = -3:
At x = -3, we calculate the derivative: f′(x)=18(−3)+6=−48
This negative value means the slope is going downhill, so we should move right to reduce the loss.
Next, try x = -8/3: f′(x)=18(−8/3)+6=−48+6=−42
We’re still negative but getting closer to 0, so we continue in the same direction.
Eventually, we reach x = -1/3, where f'(x) = 0, meaning we've hit the lowest point on the curve.

Without differentiation, we would have to guess and check many values of x to find the minimum. That’s time-consuming and inefficient.

Using differentiation or a method like gradient descent allows the computer to:
● Understand the direction to move (based on the sign of the derivative),
●Know how far to move (based on the size of the derivative),
●And eventually stop when the derivative becomes zero — indicating a minimum.

Applying Gradient Descent on The Given Problem:

Let’s get back to our housing price prediction problem. The loss function is :

Equation 1

To minimize the loss, we compute the gradients with respect to m and c:

Equation 2

Equation 3

The overall algorithm is:

When loss gradient is very low; parameters will also update very slowly.

We can do this by following this code:

import numpy as np
import matplotlib.pyplot as plt

# --------------------------
# 1. Data
# --------------------------
x = np.array([1, 5, 10, 15, 20])
y = np.array([9, 25, 45, 65, 85])

# --------------------------
# 2. Gradient Descent Setup
# --------------------------
def compute_loss(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

def compute_gradients(x, y, y_pred):
    n = len(x)
    dm = (-2 / n) * np.sum(x * (y - y_pred))
    dc = (-2 / n) * np.sum(y - y_pred)
    return dm, dc

def gradient_descent(x, y, lr=0.001, epochs=100):
    m, c = 0.0, 0.0
    loss_history = []
    m_history = []
    c_history = []

    for epoch in range(epochs):
        y_pred = m * x + c
        loss = compute_loss(y, y_pred)
        dm, dc = compute_gradients(x, y, y_pred)

        m -= lr * dm
        c -= lr * dc

        loss_history.append(loss)
        m_history.append(m)
        c_history.append(c)

    return m, c, loss_history, m_history, c_history

# --------------------------
# 3. Run Training
# --------------------------
learning_rate = 0.001
epochs = 1000000

m_final, c_final, loss_history, m_hist, c_hist = gradient_descent(x, y, lr=learning_rate, epochs=epochs)

print(f"Final Model: y = {m_final:.4f}x + {c_final:.4f}")
print(f"Final Loss: {loss_history[-1]:.4f}")

# --------------------------
# 4. Visualize
# --------------------------
fig, axs = plt.subplots(1, 2, figsize=(12, 5))

# Loss over epochs
axs[0].plot(loss_history, label="MSE Loss", color='purple')
axs[0].set_title("Loss over Epochs")
axs[0].set_xlabel("Epoch")
axs[0].set_ylabel("Loss")
axs[0].grid(True)
axs[0].legend()

# Fitted line
axs[1].scatter(x, y, color='blue', label="Training Data")
axs[1].plot(x, m_final * x + c_final, color='red', label=f"Fitted Line: y = {m_final:.2f}x + {c_final:.2f}")
axs[1].set_title("Linear Regression Fit")
axs[1].set_xlabel("x")
axs[1].set_ylabel("y")
axs[1].legend()
axs[1].grid(True)

plt.tight_layout()
plt.show()

The output is :

Final Model: y = 4.0000x + 5.0000
Final Loss: 0.0000

As we had expected y=4x+5

The importance of hyperparameter tuning can be found by using different hyperparameters:

Learning Rate: 0.001
Epochs: 1000
Final Model: y = 4.1737x + 2.4481
Final Loss: 2.0038

Learning Rate: 0.005
Epochs: 1000
Final Model: y = 4.0150x + 4.7802
Final Loss: 0.0149

Learning Rate: 0.01
Epochs: 1000
Final Model: y = -339805968568083239265530625675142850672520440490558353352887417748647011957850479438027999057677478562507719744654503043794126996610277047050911470147732262096938647145756338012452983147209293032388674801943746190294500279250271324698195527388053774864862995455035704233613156360911367509795971217252941824.0000x + -23123132324258653054329625847997072012724839172695715715363899452794460127210066287654213763824808121769583476882146702831768204182449040171184989909165227112406067941596356748523351743437113426720746461272807888735329140185756802038990722753688709912684731838050009188376018948004201929155626008030740480.0000
Final Loss: inf

Why Gradient Descent Rather Than Finding Lowest Loss:

As we don’t know the exact value, we have to reach the lowest point somehow to find the lowest loss, so we simply can’t find the lowest possible value of loss. Now, through differentiating, if we reach 0 or a very low value, then it becomes possible to find the lowest value. So, differentiation is preferred as there is a target value.
Now, suppose we start at -10 and the lowest value is at -1/3 where differentiated value 0. How will we find the optimum step size?
If we take step size 1 or 1/2, we can never reach 0. We have to take multiples of 1/3. Now, even if we do it, like we take 1/9, a multiple of 1/3, it will be a very long process to reach the lowest value.

Gradient descent for y = 9x² +6x +50

But in gradient descent, step size changes based on derivative value and sign, making it very versatile and able to reach the lowest value in an optimum way.

In reality, a dataset is much larger (100–10000s or more rows) and contains many features. So, machine learning models always try to find the relation that has the lowest value of error and predicts the best.

In this article, I tried to explain the basics of machine learning along with terminologies and a simple example of how a model is trained.