1. Introduction
2. Feature Engineering
3. Model Evaluation
4. Cross Validation
5. Residual Analysis
6. Conclusion

Introduction

Polynomial Regression represents a sophisticated approach to modelling the relationship between one or more independent variables and a dependent variable, wherein this relationship is expressed as an nth degree polynomial. This technique is particularly useful for capturing the nuances of curvilinear relationships that linear regression may not adequately represent.

The core methodology for fitting Polynomial Regression models is the method of least squares. This method, grounded in the principles of the Gauss-Markov Theorem, aims to minimise the sum of the squares of the differences between observed and predicted values. By doing so, it ensures the estimation of coefficients that bring about the best possible fit to the data under the constraints of the theorem.

What sets Polynomial Regression apart as a special case of Linear Regression is its ability to model data with a curvilinear relationship through the use of polynomial equations. This adaptability allows it to capture complex patterns and trends in the data, making it a powerful tool for predictive modelling where the relationship between variables is not strictly linear.

Polynomial Model

Polynomial example from 1 to 5 degrees

lets, take an example of quadratic polynomial model:

Quadratic model

convex, when a > 0 and concave when a < 0.

Convex

Concave

Polynomial Model Assumption

Assumptions underlying Polynomial Regression elucidate the conditions necessary for its effective application. These assumptions provide a framework for understanding how the dependent variable’s behaviour is influenced by one or more independent variables. Key assumptions include:

1. Additive Relationship: The model presumes that the dependent variable’s variations can be accounted for through a linear or curvilinear additive relationship with a set of k independent variables (Xi, where i = 1 to k). This suggests that the impact of each independent variable on the dependent variable is distinct and can be summed to predict the dependent variable’s behaviour.
2. Nature of Relationship: It is assumed that the connection between the dependent variable and each independent variable can either be linear or take the form of a curvilinear (specifically, polynomial) relationship. This allows the model to capture both simple and more complex patterns in the data.
3. Independence of Variables: A fundamental assumption is that the independent variables do not exhibit multicollinearity; they are independent of each other. This independence is crucial for the model’s ability to isolate the effect of each independent variable on the dependent variable without interference from correlations among the independent variables themselves.
4. Distribution of Errors: The errors, or residuals — the differences between observed and predicted values — are assumed to be independent, normally distributed with a mean of zero, and exhibit constant variance. This assumption, central to Ordinary Least Squares (OLS) regression, ensures that the model’s estimations are unbiased and efficient, conforming to the principles of the Gauss-Markov Theorem.

Linear Interpretation

Linear Interpretation

Linear Interpretation is achieve by fitting two line between two data points.

The formula for known points x1 and x2 and the value of the dependent function is

Formula

Matrix Representation of Polynomial Regression Equations

In polynomial regression, we aim to approximate the relationship between the independent variable x and the dependent variable y using a polynomial function of degree n, where n is the degree of the polynomial.

The general form of a polynomial function is:

general form of a polynomial function

matrix form

Matrix Notation :

• X represents the matrix of independent variables (also known as the design matrix).
• c represents the column vector of coefficients.
• Y represents the column vector of dependent variables.

Least Square Solution:

• In polynomial regression, we often have more equations (data points) than unknowns (coefficients). This leads to an overdetermined system.
• The least squares method aims to find the coefficients that minimize the sum of the squares of the differences between the observed and predicted values.
• The solution to the overdetermined system is given by the formula

Matrix Operations:

Why do we need Polynomial Regression?

When we develop a predictive model and observe its performance, we might find that it significantly underperforms. This discrepancy often becomes apparent when comparing the actual values to the predicted outcomes, especially if we notice that the actual data points form a curved pattern, while our prediction model — assuming it’s linear — generates a straight line that fails to approximate the central tendency of these points accurately. This misalignment signals the need for a more sophisticated approach, and that’s where polynomial regression comes into play.

Polynomial regression extends beyond the limitations of linear regression by accommodating the curved relationships between the independent (predictors) and dependent (outcome) variables. Unlike linear regression, which presupposes a straight-line relationship between variables, polynomial regression can model data with inherent curvature. This capability makes it a powerful alternative when linear models fall short, unable to capture the complex dynamics within the dataset.

The fundamental advantage of polynomial regression lies in its flexibility to fit a wide range of curvatures, thereby providing a more accurate representation of the data. This method does not require the relationship between variables to be linear, marking a critical distinction from linear regression. Consequently, polynomial regression is particularly valuable in scenarios where the data points deviate from a linear pattern, offering a clearer, more precise modeling of intricate relationships.

Feature Engineering

Loading libraries and our customer advertisement dataset.

import pandas as pd
import numpy as np
import seaborn as sns
import scipy.stats as stats
import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split

df = pd.read_csv('advertising.csv')
df[:5]

Pre Processing:

we are using same dataset, which we have used earlier in Linear regression model.

#feature selection
X = df['TV'].values.reshape(-1, 1)
X[:5]

#target
y = df['Sales'].values
y[:5]

# splitting the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Now, we have selected feature as TV and Target as Sales from our columns. similarly, we going to split data into train and test. where, 80 percent training and 20 percent testing with random state value of 42. It is used as a seed to the random number generator. This seed controls the shuffling applied to the data before it is split.

Model Selection

# Linear Regression
lr = LinearRegression()
lr.fit(X_train, y_train)

# predictions for plotting linear regression
X_train_range = np.arange(X_train.min(), 300, 1)[:, np.newaxis] 
y_linear = lr.predict(X_train_range)

We are going to create a line from the minimum value of the training set to 300, where 300 represents the maximum value on the X-axis scale. This means predictions will be made within the range from the minimum value of the training set to the maximum value of the X-axis scale.

pr_quad = LinearRegression()
pr_cubic = LinearRegression()
pr_quartic = LinearRegression()
pr_quintic = LinearRegression()

quadratic = PolynomialFeatures(degree=2)
cubic = PolynomialFeatures(degree=3)
quartic = PolynomialFeatures(degree=4)
quintic = PolynomialFeatures(degree=5)

X_train_quad = quadratic.fit_transform(X_train)
X_train_cubic = cubic.fit_transform(X_train)
X_train_quartic = quartic.fit_transform(X_train)
X_train_quintic = quintic.fit_transform(X_train)

X_test_quad = quadratic.transform(X_test)
X_test_cubic = cubic.transform(X_test)
X_test_quartic = quartic.transform(X_test)
X_test_quintic = quintic.transform(X_test)

pr_quad.fit(X_train_quad, y_train)
pr_cubic.fit(X_train_cubic, y_train)
pr_quartic.fit(X_train_quartic, y_train)
pr_quintic.fit(X_train_quintic, y_train)

y_quad_train = pr_quad.predict(X_train_quad)
y_cubic_train = pr_cubic.predict(X_train_cubic)
y_quartic_train = pr_quartic.predict(X_train_quartic)
y_quintic_train = pr_quintic.predict(X_train_quintic)

plt.figure(figsize=(10, 5))
plt.scatter(X_train, y_train, label="Training data points", color='black')
plt.scatter(X_test, y_test, label="Testing data points", color='red')

X_train_range = np.arange(X_train.min(), X_train.max(), 1)[:, np.newaxis]
y_quad_range = pr_quad.predict(quadratic.fit_transform(X_train_range))
y_cubic_range = pr_cubic.predict(cubic.fit_transform(X_train_range))
y_quartic_range = pr_quartic.predict(quartic.fit_transform(X_train_range))
y_quintic_range = pr_quintic.predict(quintic.fit_transform(X_train_range))

plt.plot(X_train_range, y_quad_range, label="Quadratic fit", color='red', lw=2)
plt.plot(X_train_range, y_cubic_range, label="Cubic fit", color='blue', lw=2)
plt.plot(X_train_range, y_quartic_range, label="Quartic fit", color='green', lw=2)
plt.plot(X_train_range, y_quintic_range, label="Quintic fit", color='orange', lw=2)

plt.xlabel("TV")
plt.ylabel("Sales")
plt.legend()
plt.title("Polynomial Regression")
plt.grid(True, ls='--', alpha=0.5)
plt.show()

Best fit line using polynomial regression

Quartic and quintic models may visually appear to provide a better fit to the data, we must be careful with overfitting. Further statistical analysis, including cross-validation and performance metrics (like R-squared, RMSE, or others), is necessary to choose the model that provides the best trade-off between bias and variance, thus ensuring both accuracy and generalisable.

Model Evaluation

from sklearn.metrics import r2_score, mean_squared_error, mean_absolute_error

Now, we going to perform post processing which mean how accurate and strong is our predictions.

r2_quad = r2_score(y_train, y_quad_train)
mse_quad = mean_squared_error(y_train, y_quad_train)
rmse_quad = np.sqrt(mse_quad)
mae_quad = mean_absolute_error(y_train, y_quad_train)

r2_cubic = r2_score(y_train, y_cubic_train)
mse_cubic = mean_squared_error(y_train, y_cubic_train)
rmse_cubic = np.sqrt(mse_cubic)
mae_cubic = mean_absolute_error(y_train, y_cubic_train)

r2_quartic = r2_score(y_train, y_quartic_train)
mse_quartic = mean_squared_error(y_train, y_quartic_train)
rmse_quartic = np.sqrt(mse_quartic)
mae_quartic = mean_absolute_error(y_train, y_quartic_train)

r2_quintic = r2_score(y_train, y_quintic_train)
mse_quintic = mean_squared_error(y_train, y_quintic_train)
rmse_quintic = np.sqrt(mse_quintic)
mae_quintic = mean_absolute_error(y_train, y_quintic_train)

print("Polynomial Regression Metrics:")
print("Quadratic:")
print(f"R-squared: {r2_quad:.4f}")
print(f"MSE: {mse_quad:.4f}")
print(f"RMSE: {rmse_quad:.4f}")
print(f"MAE: {mae_quad:.4f}")

print("\nCubic:")
print(f"R-squared: {r2_cubic:.4f}")
print(f"MSE: {mse_cubic:.4f}")
print(f"RMSE: {rmse_cubic:.4f}")
print(f"MAE: {mae_cubic:.4f}")

print("\nQuartic:")
print(f"R-squared: {r2_quartic:.4f}")
print(f"MSE: {mse_quartic:.4f}")
print(f"RMSE: {rmse_quartic:.4f}")
print(f"MAE: {mae_quartic:.4f}")

print("\nQuintic:")
print(f"R-squared: {r2_quintic:.4f}")
print(f"MSE: {mse_quintic:.4f}")
print(f"RMSE: {rmse_quintic:.4f}")
print(f"MAE: {mae_quintic:.4f}")

R Squared, Mean Square Error, Root Mean Square Error and Mean Absolute Error

• R-squared (Coefficient of Determination): Measures the proportion of the variance in the dependent variable that is predictable from the independent variable(s). Higher values indicate better fit. Quintic polynomial regression achieved the highest R-squared value of 0.8363, indicating it explains more variance in the data compared to lower-degree polynomials.
• Mean Squared Error (MSE): Represents the average of the squares of the errors between predicted and actual values. Lower values indicate better predictive performance. Quintic polynomial regression resulted in the lowest MSE (4.3864), suggesting it has the best predictive accuracy among the evaluated models.
• Root Mean Squared Error (RMSE): The square root of the MSE. It measures the average magnitude of the errors in the predicted values. Similar to MSE, lower values indicate better performance. Quintic polynomial regression also demonstrated the lowest RMSE (2.0944), reinforcing its superior predictive accuracy.
• Mean Absolute Error (MAE): Represents the average absolute difference between predicted and actual values. Lower values indicate better accuracy. Quintic polynomial regression had the lowest MAE (1.6592), suggesting it has the smallest average absolute deviation from the actual values, thus the most accurate predictions.
• Among the polynomial regression models evaluated, the quintic polynomial (degree 5) consistently outperforms the others in terms of R-squared, MSE, RMSE, and MAE. Therefore, for this dataset, the quintic polynomial regression model is the most suitable for prediction due to its superior accuracy and predictive power.

Cross Validation

The concept of k-fold cross-validation, where k is specifically set to 5, follows the same principles as general k-fold cross-validation, but with k fixed at 5.

1. Data Partitioning: The dataset is divided into five approximately equal-sized folds.
2. Iterative Training and Evaluation: The model is trained and evaluated five times. In each iteration, one of the five folds is used as the test set, and the remaining four folds are used as the training set.
3. Performance Metric Calculation: The performance metric (e.g., mean absolute error, root mean squared error) is computed for each iteration.
4. Aggregation of Results: The performance metrics from all five iterations are averaged to obtain a single estimation of model performance.

k=5 cross-validation offers a robust estimate of model performance by addressing variability in evaluation caused by random data partitioning. It ensures that each data point is used for both training and testing, reducing bias in the evaluation process.

Compared to leave-one-out cross-validation (LOOCV), which can be computationally expensive for large datasets, 5-fold cross-validation strikes a balance between computational cost and reliability of the estimation.

from tabulate import tabulate
from sklearn.model_selection import cross_val_score
from sklearn.metrics import make_scorer, r2_score

model_names = ["Quadratic", "Cubic", "Quartic", "Quintic"]
models = [pr_quad, pr_cubic, pr_quartic, pr_quintic]

results = []

for name, model in zip(model_names, models):
    cv_scores = cross_val_score(model, X_train, y_train, cv=5, scoring='r2')
    results.append([name] + list(cv_scores))

headers = ["Model", "CV 1", "CV 2", "CV 3", "CV 4", "CV 5"]
print(tabulate(results, headers=headers))

Cross Validation (5 Fold)

• Identical Scores Across Models: It’s very unusual to see the exact same cross-validation scores across different models and folds. This suggests that either the models are performing identically on the given dataset, which is highly improbable given the difference in complexity between the models, or there might be an error in the cross-validation process or data recording.
• Lack of Overfitting: Generally, as the complexity of the model increases (going from quadratic to quintic), we would expect to see overfitting, especially on small or noisy datasets. Overfitting would typically manifest as higher performance on the training set and lower performance on the validation set. However, since the scores are identical across all models, overfitting cannot be inferred from this data.
• Good Performance: The scores are quite high across all folds and models, which could indicate that all models are performing well. However, the uniformity of the scores suggests that you should revisit the cross-validation procedure to ensure that the data is not being leaked between the training and validation sets and that the models are being properly trained and validated.
• Possible Data Leakage or Error: The identical nature of the scores suggests that there might be data leakage, where information from the validation set is inadvertently being used to train the model, or there might be an error in the computation or reporting of the CV scores.

Residual Analysis

residuals_linear = y_train - lr.predict(X_train)
residuals_quad = y_train - y_quad_train
residuals_cubic = y_train - y_cubic_train
residuals_quartic = y_train - pr_quartic.predict(X_train_quartic)
residuals_quintic = y_train - pr_quintic.predict(X_train_quintic)

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(10, 10))
ax1.scatter(y_quad_train, residuals_quad, color='black', alpha=0.5)
ax1.set_xlabel('Fitted values')
ax1.set_ylabel('Residuals')
ax1.set_title('Residual Plot for Quadratic Regression')
ax1.axhline(y=0, color='red', linestyle='--')
ax1.grid(True, ls='--', alpha=0.5)

ax2.scatter(y_cubic_train, residuals_cubic, color='blue', alpha=0.5)
ax2.set_xlabel('Fitted values')
ax2.set_ylabel('Residuals')
ax2.set_title('Residual Plot for Cubic Regression')
ax2.axhline(y=0, color='red', linestyle='--')
ax2.grid(True, ls='--', alpha=0.5)

ax3.scatter(pr_quartic.predict(X_train_quartic), residuals_quartic, color='green', alpha=0.5)
ax3.set_xlabel('Fitted values')
ax3.set_ylabel('Residuals')
ax3.set_title('Residual Plot for Quartic Regression')
ax3.axhline(y=0, color='red', linestyle='--')
ax3.grid(True, ls='--', alpha=0.5)

ax4.scatter(pr_quintic.predict(X_train_quintic), residuals_quintic, color='orange', alpha=0.5)
ax4.set_xlabel('Fitted values')
ax4.set_ylabel('Residuals')
ax4.set_title('Residual Plot for Quintic Regression')
ax4.axhline(y=0, color='red', linestyle='--')
ax4.grid(True, ls='--', alpha=0.5)
plt.tight_layout()
plt.show()

Residual plot

Residual Plots

• Quadratic Regression: The residuals are not randomly dispersed around the horizontal axis, suggesting that a quadratic model may not be adequate for the data. Some curvature in the residual plot indicates that a higher-order polynomial might be necessary.
• Cubic Regression: Residuals appear more randomly dispersed than in the quadratic model, but there is still some pattern present, which could be a sign of remaining non-linearity in the data that the cubic term has not fully captured.
• Quartic Regression: The dispersion of residuals shows more randomness compared to the quadratic and cubic models, suggesting a better fit. However, there are some signs of potential outliers or influential points that deviate significantly from the rest.
• Quintic Regression: The residuals are dispersed, but there is a noticeable spread in the variance of residuals as the fitted values increase, which indicates heteroscedasticity. This might mean that even a fifth-degree polynomial does not fully capture the underlying relationship or that there are other issues at play, such as outliers.

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(10, 8))
sns.histplot(residuals_quad, kde=True, color='black', alpha=0.5, ax=ax1)
ax1.set_title('Histogram Plot of Residuals for Quadratic Regression')
ax1.set_xlabel('Residuals')
ax1.set_ylabel('Frequency')
ax1.grid(True, ls='--', alpha=0.5)

sns.histplot(residuals_cubic, kde=True, color='blue', alpha=0.5, ax=ax2)
ax2.set_title('Histogram Plot of Residuals for Cubic Regression')
ax2.set_xlabel('Residuals')
ax2.set_ylabel('Frequency')
ax2.grid(True, ls='--', alpha=0.5)

sns.histplot(residuals_quartic, kde=True, color='green', alpha=0.5, ax=ax3)
ax3.set_title('Histogram Plot of Residuals for Quartic Regression')
ax3.set_xlabel('Residuals')
ax3.set_ylabel('Frequency')
ax3.grid(True, ls='--', alpha=0.5)

sns.histplot(residuals_quintic, kde=True, color='orange', alpha=0.5, ax=ax4)
ax4.set_title('Histogram Plot of Residuals for Quintic Regression')
ax4.set_xlabel('Residuals')
ax4.set_ylabel('Frequency')
ax4.grid(True, ls='--', alpha=0.5)

plt.tight_layout()
plt.show()

Residual Histogram plot

Quadratic to Quintic Regressions: All box plots show a relatively symmetric distribution of residuals around the median. However, there are outliers present in all models, as evidenced by the points beyond the whiskers. The box plot for the quintic regression seems to indicate a larger spread of residuals compared to others.

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(8, 8))
stats.probplot(residuals_quad, dist="norm", plot=ax1)
ax1.set_title('Q-Q Plot of Residuals for Quadratic Regression')
ax1.set_xlabel('Theoretical Quantiles')
ax1.set_ylabel('Sample Quantiles')
ax1.grid(True, ls='--', alpha=0.5)

stats.probplot(residuals_cubic, dist="norm", plot=ax2)
ax2.set_title('Q-Q Plot of Residuals for Cubic Regression')
ax2.set_xlabel('Theoretical Quantiles')
ax2.set_ylabel('Sample Quantiles')
ax2.grid(True, ls='--', alpha=0.5)

stats.probplot(residuals_quartic, dist="norm", plot=ax3)
ax3.set_title('Q-Q Plot of Residuals for Quartic Regression')
ax3.set_xlabel('Theoretical Quantiles')
ax3.set_ylabel('Sample Quantiles')
ax3.grid(True, ls='--', alpha=0.5)

stats.probplot(residuals_quintic, dist="norm", plot=ax4)
ax4.set_title('Q-Q Plot of Residuals for Quintic Regression')
ax4.set_xlabel('Theoretical Quantiles')
ax4.set_ylabel('Sample Quantiles')
ax4.grid(True, ls='--', alpha=0.5)
plt.tight_layout()
plt.show()

Residual QQ plot

All Regressions: The Q-Q plots should ideally show all points falling on the diagonal line, which would indicate that the residuals are normally distributed. In all polynomial models, most points follow the line, but there are deviations at the ends, suggesting the presence of outliers or heavy tails in the distribution of residuals.

fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(8, 10))
sns.boxplot(y=residuals_quad, color='black', ax=ax1)
ax1.set_title('Box Plot of Residuals for Quadratic Regression')
ax1.set_xlabel('Residuals')
ax1.grid(True)

sns.boxplot(y=residuals_cubic, color='blue', ax=ax2)
ax2.set_title('Box Plot of Residuals for Cubic Regression')
ax2.set_xlabel('Residuals')
ax2.grid(True)

sns.boxplot(y=residuals_quartic, color='green', ax=ax3)
ax3.set_title('Box Plot of Residuals for Quartic Regression')
ax3.set_xlabel('Residuals')
ax3.grid(True)

sns.boxplot(y=residuals_quintic, color='orange', ax=ax4)
ax4.set_title('Box Plot of Residuals for Quintic Regression')
ax4.set_xlabel('Residuals')
ax4.grid(True)
plt.tight_layout()
plt.show()

Residual Bar plot

Quadratic to Quintic Regressions: All box plots show a relatively symmetric distribution of residuals around the median. However, there are outliers present in all models, as evidenced by the points beyond the whiskers. The box plot for the quintic regression seems to indicate a larger spread of residuals compared to others.

Conclusion and Recommendations

1. Heteroscedasticity: This is evident in the residual plots, particularly in the quintic regression. A potential fix could be to use weighted least squares or to transform the dependent variable to stabilise the variance of the residuals.
2. Outliers: Outliers and influential points are detected in all models. Investigating these points to determine if they are data entry errors or true outliers is important. If they are not errors, robust regression techniques might be more appropriate.
3. Model Complexity: As the polynomial degree increases, so does the complexity of the model. It is important to balance the goodness of fit with the model’s simplicity to avoid overfitting.

References

polynomial models

https://www.slideserve.com/ifama/polynomial-models

polynomial features

https://scikitlearn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html

1. Introduction
2. Theoretical Background
3. K-means Algorithm
4. Choosing the Number of Clusters
5. Evaluation Method
6. Conclusion

1. Introduction

In the vast sea of data that surrounds us, the ability to uncover patterns and groupings is not just valuable; it’s a necessity for making informed decisions. This is where the K-means algorithm, a stalwart of machine learning, shines by offering a simple yet powerful method for data clustering. Whether you’re a data scientist or just data-curious, understanding K-means can provide new insights from the data you encounter daily.

What is K-means?

At its core, the K-means algorithm is a method to automatically partition a dataset into distinct groups, or clusters, based on similarity. It’s a building block of unsupervised learning, tasked with finding structure in unlabeled data. By grouping data points into clusters where members are more similar to each other than to those in other clusters, K-means provides a clear view of your data’s natural divisions.

Definition of K-means Clustering

Clustering stands as a pivotal method in exploratory data analysis, offering insights into the underlying structure of datasets. Essentially, it involves grouping data points into subgroups, or clusters, based on their similarity. The aim is to ensure that members within a cluster share a high degree of resemblance, while those in different clusters exhibit distinct differences. This process hinges on selecting an appropriate similarity measure, like Euclidean or correlation-based distance, which can vary depending on the specific application.

Importance and Applications of K-means Clustering

The utility of clustering extends across various domains. For instance, in market segmentation, it helps identify customer groups with similar behaviors or attributes. In the group of image processing, it aids in segmenting and compressing images by grouping similar regions. Additionally, it’s instrumental in organising documents by topics, enhancing our understanding and management of large datasets.

2. Theoretical Background

The theoretical foundation of clustering lies in partitioning data into subsets. This partitioning is based on the concept of similarity, which is quantified through various metrics. The main goal is to maximise the similarity of objects within a cluster and minimise the similarity of objects between different clusters. The effectiveness of a clustering algorithm can largely depend on the context of the application, including the nature of the data and the specific requirements of the task at hand.

Principles of Clustering

Clustering relies on a few key ideas:

• Similarity Measurement: A most important step in clustering is defining a similarity metric, which quantifies how alike two data points are. This metric directly influences the formation of clusters.
• Feature Selection: Choosing the right set of features is essential, as it affects the similarity measurements. Irrelevant or noisy features can lead to poor clustering performance.
• Algorithm Selection: Different algorithms are suited to different types of data and clustering objectives. Choosing the right algorithm is key to obtaining meaningful clusters.
• Evaluation: Assessing the quality of the clusters formed is vital for verifying the effectiveness of the clustering process. This often involves metrics that measure the cohesion of clusters and the separation between them.

Similarity Measures

The choice of similarity measure is critical in clustering. In K-means, the similarity between data points is typically assessed using Euclidean distance, measuring the straight-line distance between points in space. However, other measures can be used depending on the nature of the data:

• Euclidean Distance: Ideal for continuous data, where the geometric distance is meaningful.
• Cosine Similarity: Used for text data or high-dimensional data, focusing on the angle between vectors rather than their magnitude.
• Manhattan Distance: Useful for ordinal or discretely-valued data, based on the sum of absolute differences.

Each similarity measure has its context where it performs best, and the choice of measure can significantly impact the results of clustering.

3. K-means Algorithm

Algorithm ensures that each piece of data is assigned to a single cluster, aiming to enhance the uniformity within clusters while maintaining clear distinctions between them. The essence of this algorithm is to optimise the allocation of data points to clusters in such a way that the cumulative squared distance from each point to the center of its cluster (the centroid) is minimised. A successful outcome is marked by high similarity among members of the same cluster and notable differences between separate clusters.

Executing the K-means Algorithm involves a series of steps:

1. Selection of Cluster Count: Deciding on the number of clusters, denoted as K.
2. Centroid Initialisation: The dataset is first mixed and then K data points are chosen at random as the initial centroids, ensuring no repetitions.
3. Iterative Optimisation: The algorithm repeatedly adjusts the assignments and centroids until the centroids stabilise, showing no further changes in their positions.
4. Distance Calculation: It computes the total squared distance between the data points and each centroid.
5. Cluster Assignment: Data points are affiliated with the nearest centroid, signifying their cluster membership.
6. Centroid Recalculation: The algorithm recalculates the position of each centroid by averaging the coordinates of all the points in the cluster, refining the cluster’s coherence.

Expectation-Maximisation

The K-means algorithm adopts the Expectation-Maximisation (E-M) strategy to partition data into clusters. This process alternates between two key steps: the expectation (E) step, which involves assigning data points to the nearest cluster, and the maximization (M) step, which computes the centroids of these clusters. Let’s get into the mathematical framework underpinning this approach.

Objective Function

The objective of K-means is encapsulated in the function J, defined as:

In this equation, Wik serves as an indicator variable that equals 1 if the data point Xi is allocated to cluster k, and 0 otherwise. The symbol μk denotes the centroid of the cluster that data point Xi is associated with.

E-step (Expectation Step)

During the E-step, the algorithm updates Wik by assigning each data point Xi to the cluster with centroid μk that minimises the squared Euclidean distance. ∥xi−μk∥².

M-step (Maximisation Step)

In the M-step, K-means recalculates the centroids μk for each cluster based on the new assignments from the E-step. The updated centroid is computed as the mean of all points assigned to the cluster:

This step ensures that the centroids accurately reflect the current makeup of their clusters, improving the algorithm’s partitioning with each iteration.

Standardisation of Data

• Why It’s Necessary: Clustering algorithms, including K-means, utilize distance metrics to determine the similarity between data points. When features in a dataset are on different scales (for example, age in years and income in thousands of dollars), it can distort the distance measurements, heavily biasing the algorithm towards the larger-scale features. This can lead to misleading clustering results.
• Recommended Approach: To counteract this issue, it’s recommended to standardize the data before clustering. Standardization involves adjusting the data such that each feature has a mean of zero and a standard deviation of one. This process ensures that all features contribute equally to the distance calculations, allowing for a fair comparison of similarities across different dimensions.

Initialisation Sensitivity

• The Challenge: The K-means algorithm begins with a random initialisation of centroids (the starting points of clusters). Due to its iterative nature and reliance on these initial centroids, K-means can converge to different local optima depending on the starting conditions. This means that different initialisations can lead to different clustering outcomes, some of which may be suboptimal.
• Mitigation Strategies: To address this variability and increase the chances of finding a more globally optimal solution, it’s advisable to run the K-means algorithm multiple times with different random initializations of centroids. After these runs, the outcome that yields the lowest sum of squared distances (a measure of how compact the clusters are) should be selected. This approach helps mitigate the impact of unfortunate initial placements and leverages the algorithm’s stochastic nature to explore a broader range of potential solutions.

Convergence and Within-Cluster Variation

• Observing Convergence: The convergence of the K-means algorithm is typically determined when there’s no significant change in the assignment of examples to clusters, indicating stability in the solution. This stability is often measured as minimal or no change in within-cluster variation, which implies that further iterations do not meaningfully alter the structure of the clusters.
• Practical Implication: While the convergence criterion is a useful indicator of algorithm stability, it’s essential to remember that convergence to a stable solution does not necessarily mean that the solution is the best or most meaningful clustering of the data. It merely indicates that the algorithm has found a locally optimal arrangement of data points into clusters based on the initial conditions and distance metrics used.

#Importing Libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

#creating KMeans
class K_Means:
    def __init__(self, k=3, tolerance=0.0001, max_iterations=500):
        # the number of clusters to form as well as the number of centroids to generate.
        self.k = k
        # the percentage change in the centroids before considering the algorithm has converged.
        self.tolerance = tolerance
        # the maximum number of times the algorithm will attempt to converge before stopping.
        self.max_iterations = max_iterations
# method to fit model inside dataset
    def fit(self, data):
        # initialise Centroids Randomly
        # empty dictonary to store centroids
        self.centroids = {}
        # for loop to iterate inside k
        for i in range(self.k):
            self.centroids[i] = data[i]
        # loop through max iterations times
        for i in range(self.max_iterations):
            # create empty classes for each centroid
            self.classes = {i: [] for i in range(self.k)}
            # assigning data points to nearest centroid
            for features in data:
                # euclidean distance between the current data point and each centroid,
                distances = [np.linalg.norm(features - self.centroids[centroid]) for centroid in self.centroids]
                classification = distances.index(min(distances))
                self.classes[classification].append(features)
            # previous centroids
            previous = dict(self.centroids)
            # update centroids as mean of assigned data points
            for classification in self.classes:
                self.centroids[classification] = np.mean(self.classes[classification], axis=0)
            # convergence
            isOptimal = True
            for centroid in self.centroids:
                original_centroid = previous[centroid]
                curr = self.centroids[centroid]
                if np.sum((curr - original_centroid) / original_centroid * 100.0) > self.tolerance:
                    isOptimal = False
            # break loop if converged
            if isOptimal:
                break
# method to predict
def prediction(self, data):
    # euclidean distance
    distances = [np.linalg.norm(data - self.centroids[centroid]) for centroid in self.centroids]
    classification = distances.index(min(distances))
    return classification
    
# generating dataset for testing
np.random.seed(42)
# creating data from random numbers
data = np.random.randn(500, 2)
# standard scale
data = (data - data.mean())/data.std()
print(data[:10])

# K_Means for the clustering
km = K_Means(3)
# fitting model
km.fit(data)
# checking centroids
km.centroids

4. Choosing the Number of Clusters

The Elbow Method

The Elbow Method is a heuristic used to determine the optimal number of clusters by plotting the within-cluster sum of squares (WCSS) against the number of clusters. The WCSS decreases as the number of clusters increases because the clusters are smaller and the data points are closer to the centroids. The goal is to identify the point where the decrease in WCSS begins to slow down, which is indicative of diminishing returns by increasing K. This point resembles an “elbow” in the plot, hence the name. The Elbow Method is intuitive and easy to implement but can sometimes be ambiguous if the “elbow” is not very clear.

The Silhouette Method

The Silhouette Method provides a more nuanced approach by measuring how similar an object is to its own cluster compared to other clusters. The silhouette score ranges from -1 to 1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. For each K, you calculate the average silhouette score of all objects. The optimal number of clusters corresponds to the K with the highest average silhouette score. This method not only helps in determining the number of clusters but also provides insight into the separation distance between the resulting clusters, offering a more detailed perspective on cluster cohesion and separation.

Other Methods for Determining the Optimal K

Besides the Elbow and Silhouette Methods, several other techniques can help in deciding the optimal number of clusters:

• Gap Statistic: Compares the total within intra-cluster variation for different values of K with their expected values under null reference distribution of the data. The gap statistic measures the gap between the logarithm of within-cluster dispersion and its expectation under a reference null distribution. The optimal K is the one that maximizes this gap.
• Davies-Bouldin Index: Aims to identify the number of clusters that minimize the average ‘similarity’ between clusters, where similarity is a measure that compares the distance between clusters with the size of the clusters themselves. A lower Davies-Bouldin Index indicates a better partitioning.
• Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC): Used in mixture-model based clustering, these criteria evaluate the model fit of the clustering solution while penalizing for the number of parameters to avoid overfitting. The optimal K is the one that minimizes the AIC or BIC.

def within_cluster_sum_of_squares(data, k):
    kmeans = K_Means(k)
    kmeans.fit(data)
    wcss = sum([np.sum([np.linalg.norm(x - kmeans.centroids[ci])**2 for x in kmeans.classes[ci]]) for ci in kmeans.classes])
    return wcss

Visualisation of Clusters

# clustering
def clustering(km, x1, x2):
    plt.figure(figsize=(10, 6))
    # total colors for plotting clusters
    colors = ["black", "green", "cyan", "orange", "blue", "magenta", "yellow", "white", "purple", "brown", "pink"]

    # iterate over each cluster
    for classification in km.classes:
        color = colors[classification]
        for features in km.classes[classification]:
            # plotting each data point in the cluster
            plt.scatter(features[0], features[1], color=color, s=30, alpha=0.5)

    # initialise centroid label counter
    centroid_label = 1

    # iterate over each centroid and plot them with number labels
    for centroid in km.centroids:
        # plotting centroid of cluster
        plt.scatter(km.centroids[centroid][0], km.centroids[centroid][1], s=300, marker="X", color='red')
        # text label at centroid center
        plt.text(km.centroids[centroid][0], km.centroids[centroid][1], str(centroid_label), ha='center', va='center', color='white', fontsize=12, fontweight='bold')
        # increment centroid label counter
        centroid_label += 1

    plt.grid(True, ls='--', alpha=0.2, color='grey')
    plt.title("K-Means Clustering")
    plt.xlabel(f"{x1}")
    plt.ylabel(f"{x2}")
    plt.show()

# calling function for plot
clustering(km, "feature1", "feature2")    # iterate over each cluster
    for classification in km.classes:
        color = colors[classification]
        for features in km.classes[classification]:
            # plotting each data point in the cluster
            plt.scatter(features[0], features[1], color=color, s=30, alpha=0.5)

Cluster

• Clusters: The data points are color-coded, likely to represent different clusters identified by the K-means algorithm. Typically, each color corresponds to one cluster. In this case, we have three distinct groups of data points represented by different colors (light blue, dark grey, and green).
• Centroids: The red numbers (1, 2, and 3) indicate the centroids of each cluster. These are the calculated mean positions of all the data points within a single cluster and are often considered the center of their respective clusters. The positioning of these centroids is crucial as the K-means algorithm assigns data points to clusters based on the shortest distance to these centroids.
• Data Distribution: The spread of the data points suggests the natural groupings within the dataset. From the plot, it seems that the light blue and green points are more mixed together, indicating that the separation between these two clusters might not be very clear. The dark grey points are more densely packed and appear to be well-separated from the other two clusters.
• Feature Axes: The feature1 axis (horizontal) and feature2 axis (vertical) are the two features based on which the clustering is done. These could be any quantitative aspects of the data, such as height, weight, income, etc., depending on the dataset being analyzed.
• Cluster Overlap: There appears to be some overlap between the clusters, especially between the clusters represented by the light blue and green points. This could suggest that the chosen features might not be entirely effective at separating the clusters distinctly, or it could reflect a natural overlap in the properties of the data points.

5. Evaluation Method

The Elbow Method is a powerful technique that helps us determine the optimal number of clusters (k) for K-means clustering. It does this by analyzing the Sum of Squared Distances (SSE) between data points and their respective cluster centroids. As we evaluate different values of k, we look for the point where the SSE curve begins to level off, resembling an elbow. This elbow represents the point beyond which increasing the number of clusters does not significantly improve the fitting of our model to the data. To put this method into practice, let’s consider the geyser dataset. By examining the SSE for various k values, we aim to identify where the curve transitions to a flatter slope, indicating the most appropriate number of clusters for our analysis.

def elbow(data):
    # K values range from 1 to 10
    k_values = range(1, 10)
    # formula for with-in cluster sum of squares
    wcss_scores = [within_cluster_sum_of_squares(data, k) for k in k_values]

    plt.figure(figsize=(8, 4))
    plt.plot(k_values, wcss_scores, '-o', color='red')
    plt.title('Elbow Method For Optimal k')
    plt.xlabel('Number of clusters k')
    plt.ylabel('Within-Cluster Sum of Squares')
    plt.xticks(k_values)
    plt.grid(True, ls='--', alpha=0.2, color='grey')
    plt.show()

elbow(data)

• X-axis (Number of clusters k): This represents the number of clusters used in the K-means algorithm. The graph shows values from 1 to 9 clusters.
• Y-axis (Within-cluster sum of squares): This axis shows the within-cluster sum of squares (WCSS), which is a measure of the compactness of the clusters. It is calculated as the sum of the squared distances between each data point and the centroid of its assigned cluster.
• Line: The line connects the WCSS for each number of clusters. As the number of clusters increases, the WCSS generally decreases because the clusters become smaller and tighter.
• Elbow Point: The elbow of the plot is the point where the WCSS begins to decrease at a slower rate as the number of clusters increases. This point is considered to be a good trade-off between the number of clusters and the cluster compactness.

Elbow seems to be around the point where k=3 or k=4. This suggests that increasing the number of clusters beyond 3 or 4 leads to diminishing returns in terms of decreasing the WCSS. Hence, the optimal number of clusters for this particular dataset might be 3 or 4 based on the Elbow.

Another Example

from sklearn.datasets import make_blobs
#features and target
X, y = make_blobs(n_samples=200, centers=4, random_state=42)
#standard scale 
X = (X- X.mean())/X.std()

# K_Means for the clustering
km = K_Means(4)
# fitting model
km.fit(X)

# checking centroids
km.centroids

K-Means Clustering Visualization

The scatter plot displaying the output of the K-means clustering on a dataset.

• Clusters: The different colors (light blue, green, grey, and orange) represent different clusters assigned by the K-means algorithm. The distribution of colors shows the grouping of the data points into four clusters.
• Centroids: The red numbers (1, 2, 3, and 4) mark the centroids of each cluster. These centroids are the mean position of all data points in the cluster and are used by the K-means algorithm to assign points to the nearest cluster.
• Feature Axes: The axes, feature1 and feature2, are the two features used for clustering. The clustering process places data points into clusters based on their similarity in this two-dimensional feature space.
• Cluster Characteristics: We observe that the clusters have varying densities and separations. For example, the orange cluster appears more spread out, whereas the grey cluster is more compact. The centroids are located centrally within each of their respective clusters, which is characteristic of well-separated clusters.

Elbow for optimal k

Elbow Method for Optimal k

Using the Elbow Method to identify the optimal number of clusters for K-means clustering.

• X-axis (Number of clusters k): Displays the number of clusters chosen for the K-means algorithm, ranging from 1 to 9.
• Y-axis (Within-cluster sum of squares): Indicates the within-cluster sum of squares (WCSS), which quantifies the compactness of the clusters. It is the sum of the squared distances between each data point and the centroid of the cluster it belongs to.
• Elbow Point: The graph is used to locate an “elbow” point where the rate of decrease in WCSS slows down significantly. In this plot, the elbow appears to be less pronounced, but there’s a subtle bend around k=4. This suggests that setting k=4 might be a reasonable choice for the number of clusters because beyond this point, the reduction in WCSS for each additional cluster is less significant.

6. Conclusion

K-means clustering is an important tool in machine learning for uncovering hidden patterns in data. It is particularly useful for exploratory data analysis, enabling the discovery of natural groupings within datasets. Through the iterative process of assigning data points to clusters and updating cluster centroids, K-means refines these groupings to optimize similarity within clusters and differences between them. The choice of the number of clusters, k, is pivotal, with methods like the Elbow and Silhouette providing guidance on the optimal cluster count.

Despite its simplicity, K-means faces challenges such as sensitivity to initial conditions and the potential for convergence to local optima. Nevertheless, when applied with considerations like data standardisation and careful initialisation, K-means clustering can yield insightful and actionable information across a multitude of applications.

• For understanding cluster analysis techniques: Understanding Cluster Analysis
• For building K-means clustering algorithms from scratch: Building K-means Cluster Algorithms from Scratch

VGG16 CNN Architecture

Table of Contents

1. Introduction to Convolutional Neural Networks (CNNs)
2. Understanding CNN Architectures
3. Convolutional Layers
4. Pooling Layers
5. Understanding Convolutional Blocks
6. Fully Connected Classifier
7. Regularisation and Advanced CNN Concepts
8. Conclusion

1. Introduction

How does your phone know a photo contains a dog and not a cat? How does a bank read the handwritten digits on a cheque? Behind most of these everyday tricks is a Convolutional Neural Network (CNN). CNNs are the foundation of modern computer vision, and whenever a model has to answer “What is in this image?”, there is a good chance a CNN is doing the work. I recently studied how they work, and instead of only reading the theory, I built two small projects to see it in action: I classified my own image with a pre-trained VGG-16 model, and I trained a CNN of my own to recognise handwritten digits. In this article, I explain CNNs in simple language, step by step. We start with why regular neural networks struggle with images, then look at how convolution, stride and padding work, how filters and feature maps learn patterns, why pooling matters, and how the final classifier turns features into predictions. Each concept comes with code you can copy and run yourself. Whether you are a student, a beginner in deep learning, or just curious about how machines “see”, this guide is for you. Let’s start with a simple question: why can’t a regular neural network do this job?

• A CNN (Convolutional Neural Network) is a deep learning architecture specifically developed to analyse and process image data.
• It consists of multiple layers, including convolutional, activation, pooling, and fully connected layers, which automatically extract relevant features and generate predictions.
• CNNs are commonly applied in image classification, object detection, facial recognition, image recognition, and various other computer vision applications.
• Well-known CNN architectures include LeNet, AlexNet, VGG, ResNet, and EfficientNet.
• Selecting an appropriate CNN architecture depends on the task, dataset characteristics, and required performance.

Why Regular Neural Networks Struggle with Images

Before CNNs, the obvious way to classify images was a fully connected network, also called a Multilayer Perceptron (MLP). I used one myself to classify handwritten digits, and it works fine on small images. The trouble starts when images get bigger, because an MLP runs into three problems.

The first is the sheer number of parameters. An MLP needs one input neuron for every pixel, and every one of those neurons connects to every neuron in the next layer. A colour image of 224 x 224 pixels with 3 channels has 224 x 224 x 3 = 150,528 values. Connect that to a single hidden layer of just 128 neurons and you already have about 19 million weights. Add two more hidden layers of 128 and the total barely changes, but swap the first hidden layer for 1,000 neurons and you jump to roughly 150 million weights. The bigger the image, the faster this grows, and training becomes slow and expensive.

The second problem is that an MLP is not translation invariant. It does not understand that a digit or an object is the same thing wherever it appears. If the content of the image shifts a few pixels to the left or right, the input values change completely, so the network may give a different answer for the same object. In my digit project, a “7” that sits slightly off-centre can confuse an MLP that has only seen centred digits.

The third problem is overfitting. With millions of weights and a limited amount of training data, the model can simply memorise the training images instead of learning the general patterns that make a dog look like a dog. It then scores well on training data and poorly on new images.

This is where CNNs come in. Instead of connecting every pixel to every neuron, a CNN slides a small filter across the image and reuses the same weights at every position. This idea, called parameter sharing, cuts the number of weights dramatically, and because the same filter checks the whole image, it can spot a pattern wherever it appears. Fewer parameters also means less overfitting.

2. What Is a CNN Architecture?

A Convolutional Neural Network (CNN) is a deep learning architecture specifically designed to process and analyse visual data, including images and videos. It is widely used in computer vision because of its ability to automatically identify and extract meaningful features from visual information. Unlike traditional fully connected neural networks, such as Multilayer Perceptrons (MLPs), which process input data through densely connected layers, CNNs examine smaller regions of an image to identify important patterns, including edges, textures, shapes, and other visual characteristics. This approach makes CNNs particularly effective for image-related tasks while reducing the need to connect every input pixel to every neuron in the network.

One of the main limitations of using fully connected MLP networks for image processing is the large amount of data contained in images. Flattening image data into a single-dimensional input can result in a substantial number of trainable parameters, increasing computational complexity, memory requirements, and training time. Consequently, training fully connected networks on large image datasets can become computationally expensive and challenging. CNN architecture addresses these limitations through a structured arrangement of layers, including convolutional, activation, pooling, and fully connected layers. These layers work together to automatically extract relevant features, reduce the dimensionality of the data, and generate predictions. As an image passes through the network, the initial convolutional layers learn basic visual characteristics, such as edges and textures, while deeper layers identify more complex patterns, shapes, and object representations. This hierarchical feature-learning process enables CNNs to recognise and classify visual objects effectively.

CNNs are widely applied in various computer vision tasks, including image classification, object detection, image recognition, facial recognition, and image analysis. Several established CNN architectures have been developed, including LeNet, AlexNet, VGG, ResNet, and EfficientNet, each offering different approaches to network design, feature extraction, and computational efficiency. The selection of an appropriate CNN architecture depends on several factors, including the nature of the problem, dataset size and complexity, computational resources, and required model performance.

MLPs Lack Translation Invariance

One of the limitations of using Multilayer Perceptrons (MLPs) for image processing is their inability to naturally handle changes in the spatial position of objects within an image. This means that when the main object or visual feature is shifted to a different location, the network may interpret it as a completely different input. Consequently, even if two images contain the same object, a change in its position can affect the network’s classification results. This makes image recognition more challenging and may reduce prediction consistency when objects appear in different locations.

MLPs Are Prone to Overfitting

Another significant limitation of using Multilayer Perceptrons (MLPs) for image processing is the large number of trainable parameters required to handle image data. In a fully connected MLP, each input pixel is connected to neurons in the subsequent layer, causing the number of weights to increase rapidly as the image dimensions and colour channels grow.

MLP

For instance, consider a colour image with dimensions of 224 × 224 × 3. Flattening this image into a one-dimensional input would produce 150,528 input features. If the network contains three hidden layers, each consisting of 128 neurons, the total number of trainable parameters would exceed 19 million, excluding biases. This creates considerable computational and memory demands during training. Furthermore, networks with excessively large numbers of trainable parameters are more likely to overfit the training data, meaning they may perform well on training examples but struggle to generalise to previously unseen images. Consequently, training such models can become computationally expensive, time-consuming, and less effective when working with large and complex image datasets.

Convolutional Neural Networks (CNNs) were introduced to provide a more efficient and effective approach to processing image data. Their effectiveness comes primarily from convolution operations, which enable the network to identify and extract relevant features from different regions of an image. An important characteristic of convolutional layers is parameter sharing, whereby the same set of weights (filters or kernels) is applied across multiple locations within the input image. As these filters move across the image, they can recognise similar patterns regardless of their spatial position, providing a degree of translation invariance. This approach reduces the number of trainable parameters considerably compared with fully connected neural networks, lowering computational requirements and improving training efficiency.

VGG-16 CNN Architecture

At a general level, a Convolutional Neural Network (CNN) consists of two main components: a feature extractor and a classifier. The feature extraction component is also commonly known as the backbone or body of the network, while the classifier is referred to as the head. CNN architectures are built using different types of layers that work together to perform feature extraction and classification. To understand how these components are organised and function, the VGG-16 architecture can be used as an example.

VGG-16 consists of five main convolutional blocks, which form the feature extraction component of the network. Each convolutional block contains a sequence of layers designed to identify and learn relevant visual features from the input image. Once feature extraction is completed, the resulting representations are passed to the classifier, which uses the learned information to determine the appropriate class. VGG-16 was trained using the ImageNet dataset, which contains 1,000 different object categories. Therefore, its final output layer consists of 1,000 neurons, with each neuron representing the predicted probability of the input image belonging to a particular category. The class associated with the highest probability is selected as the model’s final prediction.

VGG-16

Each layer in the VGG-16 architecture has two important dimensions: spatial dimensions and depth, which describe the shape and structure of the data as it moves through the network. VGG-16 is designed to process colour images with an input size of 224 × 224 × 3, where the first two dimensions represent the image’s height and width, while the third represents the three RGB colour channels (red, green, and blue). As the image moves through the different layers of the network, its dimensions gradually change. The spatial dimensions (height and width) are progressively reduced through operations such as pooling, while the depth increases as convolutional layers generate additional feature maps. This depth is commonly referred to as the number of channels, representing the different features extracted by the network at each stage.

Convolutional Blocks and Pooling Layers

The VGG-16 architecture consists of five main convolutional blocks, labelled Conv-1 through Conv-5, which form the core of its feature extraction process. These blocks are responsible for identifying and learning different visual patterns from the input images. Although the number and arrangement of layers within each convolutional block may differ across CNN architectures, they generally consist of one or more 2D convolutional layers followed by a pooling layer. Additional layers may also be included depending on the network design.

In the final stage of the network, a Softmax layer is applied to the outputs produced by the last fully connected layer (FC-8). The Softmax function converts the raw output values (logits) into normalised scores ranging between 0 and 1, with the scores summing to 1. These values represent the predicted probabilities for each class, allowing the network to determine which category the input image most likely belongs to.

CNN Block and Pooling layers

3. Convolutional Layers

A convolutional layer is one of the fundamental components of a Convolutional Neural Network (CNN) and can be considered the feature-detection mechanism of the network. Its primary purpose is to identify specific patterns and characteristics within input data, such as edges, textures, shapes, and other visual features. The input to a convolutional layer is generally a multidimensional array, which may be an original image or the output generated by a preceding layer. Depending on the image type, the input may contain a single channel for grayscale images or three channels for RGB colour images.

In the VGG-16 architecture, the network accepts colour images with an input dimension of 224 × 224 × 3, representing the image’s height, width, and three colour channels. Convolutional layers process this input using filters, also known as kernels, which move across different regions of the image. At each position, the filter performs a mathematical convolution operation to calculate a numerical value representing the detected feature. These values are passed through an activation function, typically ReLU, and arranged into an output representation known as an activation map or feature map. This map captures the relevant features detected by the filter and provides a structured representation of the original input for subsequent layers of the network.

Convolutional Layers

The Convolution Operation (Input × Kernel)

To understand how convolutional layers work, it is important to first examine the convolution operation, which is the fundamental process used to extract features from input images. A convolution operation involves applying a small filter, commonly known as a kernel, to different regions of the input data. For example, consider a 6 × 6 input matrix being processed using a 3 × 3 filter. In traditional image processing, predefined kernels such as the Sobel kernel are commonly used to identify specific features, including vertical edges. However, in CNNs, kernel values are generally learnable weights that are automatically adjusted during the training process to identify useful patterns in the input data.

The convolution process begins by positioning the kernel over a small section of the input image. Each element of the kernel is multiplied by its corresponding element in the selected input region. The resulting products are then added together to produce a single numerical value. This calculation is equivalent to computing the dot product between the kernel and the selected input region, which is known as the receptive field.

After calculating the first output value, the kernel moves across the input image to process the next region. This movement is controlled by a parameter called stride. For example, a stride of one means that the kernel shifts by one pixel at a time. The process continues until the kernel has moved across all valid regions of the input, generating an output matrix known as a feature map or activation map.

Each convolution operation produces one numerical value at a corresponding position in the output feature map. By repeatedly applying this process across the input, the CNN can identify and represent important visual patterns that are subsequently used by deeper layers to learn more complex features.

Convolutional Operations

Stride

Stride refers to the number of pixels by which a convolutional filter moves across the input image during each operation. For instance, a stride value of 1 means that the kernel shifts one pixel at a time horizontally and vertically, calculating the dot product at each position. A smaller stride allows the filter to examine more overlapping regions of the input, producing larger feature maps and capturing more detailed information. In contrast, increasing the stride causes the kernel to move across the image in larger steps, reducing the number of convolution operations and resulting in smaller output feature maps.

Padding

Padding is a technique used in CNNs to control the spatial dimensions of the output feature maps by adding additional pixels, usually containing zeros, around the boundaries of the input image. This helps preserve information near the image edges, which might otherwise be lost during convolution operations. Padding also provides greater control over the output dimensions, allowing the network to maintain the original spatial size when required.

Padding

Several padding methods are available in CNNs, including zero padding, same padding, and valid padding, each controlling how the input dimensions are handled during convolution operations. Among these approaches, zero padding is commonly used because of its simplicity and computational efficiency. It works by adding rows and columns of zeros around the boundaries of the input matrix, allowing convolutional filters to process regions near the image edges more effectively. This technique is also used in established CNN architectures, such as AlexNet, to help manage spatial dimensions and retain important boundary information.

Padding

Convolution Output Spatial Dimensions

The output dimensions of a 2D convolutional layer are determined by the input size, kernel dimensions, padding, and stride. The spatial size of the resulting feature map can be calculated using the following equation:

• O: Output dimension
• n: Input dimension (height or width)
• f: Kernel size
• p: Padding
• s: Stride

Suppose we have:

Same Padding

• Input image: 32 × 32
• Kernel: 3 × 3
• Padding: 1
• Stride: 1

Same Padding

Output: 32 × 32

This means the output feature map retains the same spatial dimensions as the original image.

Valid Padding

Now consider another example:

• Input image: 64 × 64
• Kernel: 3 × 3
• Padding: 0
• Stride: 1

Valid Padding

Sobel Kernel

The Sobel kernel is a commonly used image-processing filter designed to identify edges within an image. It works by applying a convolution operation to detect changes in pixel intensity, helping distinguish boundaries between different regions of an image.

For example, a Sobel kernel designed to detect vertical edges contains positive values in its left column, zeros in the middle column, and negative values in its right column. As the kernel moves across the image, it calculates the weighted sum of the corresponding pixel values. This operation measures changes in intensity along the horizontal direction, allowing the detection of vertical edges in the image.

Different kernel configurations can be designed to identify different visual patterns, including horizontal edges, vertical edges, and diagonal structures. These filters help extract important visual information that can be used in subsequent image-processing operations.

In Convolutional Neural Networks (CNNs), this principle is extended through learnable kernel weights. Rather than relying on predefined filters such as Sobel kernels, CNNs automatically learn the most useful filter values during the training process. This allows the network to identify a wide range of visual features, from simple edges and textures to more complex shapes and object patterns, supporting accurate image classification and other computer vision tasks.

Source: https://setosa.io/ev/image-kernels/

sobel kernel operation

Convolutional Layer Properties

In a Convolutional Neural Network (CNN), the output generated by a convolution operation is passed through an activation function, producing what are known as activation maps or feature maps. A convolutional layer generally contains multiple filters, with each filter generating its own activation map. Consequently, the output consists of multiple feature maps that represent different characteristics extracted from the input image. As data moves through successive convolutional layers, its spatial dimensions and depth are progressively transformed to capture increasingly complex visual information.

The following properties are important for understanding how convolutional layers operate:

• Filter Depth: The depth of a filter must correspond to the number of channels in the input data. For example, an RGB image containing three channels requires filters with a depth of three.
• Filter Spatial Dimensions: The height and width of a filter are determined by the network architecture. Commonly used filter sizes include 3 × 3 and 5 × 5, depending on the feature extraction requirements.
• Number of Filters: The number of filters within a convolutional layer determines how many activation maps are generated. Each filter learns to identify different features from the input data.
• Activation Maps and Channels: The outputs produced by multiple filters are combined to form a feature map with multiple channels. Each individual channel represents an activation map generated by a corresponding filter.
• Kernels and Filters: A filter consists of a collection of weights arranged across the input channels. Each individual channel within the filter contains a kernel, which performs convolution operations on the corresponding input channel. For single-channel input data, a filter contains only one kernel, making the terms filter and kernel effectively interchangeable.
• Learnable Weights: The weights within convolutional filters are typically initialised with small random values and subsequently adjusted during training through backpropagation and optimisation. This enables the network to learn useful visual features automatically.
• Trainable Parameters: The total number of trainable parameters in a convolutional layer depends on the filter dimensions, the number of input channels, and the number of filters. These parameters determine the learning capacity and computational requirements of the layer.

Convolutional Layer with a Single Filter

To understand how a convolutional layer operates, consider an input colour image with dimensions of 224 × 224 × 3, representing its height, width, and three RGB colour channels. When applying a convolution operation, the filter dimensions must be compatible with the input depth. Therefore, since the input image contains three channels, the filter must also have a depth of three.

In this example, a single convolutional filter with dimensions of 3 × 3 × 3 is used. This filter contains three individual 3 × 3 kernels, with each kernel processing its corresponding RGB channel. The convolution operations combine the information extracted from all three channels to generate a single numerical value at each filter position. By moving the filter across the entire input image, the network produces one activation map that represents the visual features detected by that filter.

• Colour input image (224 × 224 × 3): three stacked layers in red, green and blue, with an orange 3 × 3 patch marked on the front one.
• One 3 × 3 × 3 filter (F₁): three kernels (k₁, k₂, k₃), one per colour channel, with the weights w₁ to w₉ labelled on the front one. Dashed lines link the patch to the filter.
• Activation function: a card showing f(w·x + b) with a small sigmoid curve.
• Activation map (224 × 224 × 1): a single teal pixel marks the one number produced at each filter position.
• Two brackets: “Convolution at one filter location” and “Apply activation”.
• Parameter count: 3 × 3 × 3 = 27 weights + 1 bias = 28 trainable parameters.

Convolutional Layer with single filter

Trainable Parameters and Output Channels in a Convolutional Layer

In this example, the convolutional layer contains a single filter with dimensions of 3 × 3 × 3, which consists of three individual kernels corresponding to the three RGB input channels. Each kernel contains nine weights, resulting in a total of 27 trainable weights across the filter. Additionally, the filter includes one bias parameter, bringing the total number of trainable parameters to 28.

During the convolution operation, each of the three kernels processes its corresponding input channel independently. The resulting values are then combined by calculating their weighted sum and adding the bias term. This combined value is passed through an activation function, producing a single numerical output at each filter position. Repeating this operation across the input image generates one activation map. Since the convolutional layer contains only one filter, its output has a depth of one channel.

An important principle of convolutional layers is that the depth of each filter must match the number of channels in the input data, while the number of filters determines the number of output activation maps (channels).

For example, consider a convolutional layer containing 32 filters. Each filter processes all the input channels and generates one activation map. Consequently, the output feature map contains 32 channels. If another convolutional layer is added after this layer, its filters must have a depth of 32 to match the number of channels produced by the preceding layer. The number of filters in this subsequent layer can be selected independently, depending on the architecture’s feature extraction requirements.

Convolutional Layer with Two Filters

This example demonstrates how a convolutional layer operates when multiple filters are applied to an input tensor. It is important to understand that the input depth of three does not necessarily represent the RGB colour channels of an image. Instead, it indicates that the input data contains three channels, which could also be feature maps generated by a preceding convolutional layer within the network.

In this example, the convolutional layer uses two filters, each with dimensions of 3 × 3 × 3. Since the input tensor contains three channels, both filters must have a depth of three to match the input dimensions. Each filter therefore contains 27 trainable weights (3 × 3 × 3). With two filters, the total number of trainable weights is 54. Additionally, each filter has its own bias parameter, resulting in two bias terms.

Therefore, the convolutional layer contains a total of 56 trainable parameters, calculated as follows:

(3 X 3 X 3 X 2) + 2 = 56

• Input tensor (w × h × 3): the same R, G, B stack with the orange 3 × 3 patch.
• Two 3 × 3 × 3 filters: F₁ and F₂ side by side, each with three colour-coded kernels and weights w₁ to w₉. Dashed lines link the patch to F₁ and F₂.
• Activation function: the same f(w·x + b) card with the sigmoid curve.
• Activation map (w × h × 2): two overlapping output layers, with a teal pixel for F₁ and a blue pixel for F₂.
• Parameter count: 2 × 3 × 3 × 3 = 54 weights + 2 biases = 56 trainable parameters.

Each filter independently processes the entire input tensor and generates one activation map. As a result, using two filters produces an output feature map with two channels, with each channel representing the features detected by its corresponding filter.

Two Filter

Filters Learn to Detect Different Structures

Each filter within a convolutional layer can be understood as a feature detector that learns to recognise specific patterns and structures in the input data. During the training process, the weights within each filter are automatically adjusted to identify useful visual characteristics. The individual channels within the resulting activation maps represent different features extracted from the input image.

For example, consider a convolutional layer containing three different filters applied to an image containing various linear structures and edges. Each filter can learn to respond to a different type of edge pattern, such as horizontal, vertical, or diagonal edges. When these filters process the input image, they generate separate activation maps, with each map highlighting the regions where the corresponding features are detected.

In these activation maps, neurons with stronger activation values indicate regions where the learned filter has identified its associated features more prominently. This demonstrates how CNNs automatically learn different visual patterns through multiple filters, allowing deeper layers to combine these simpler features into more complex structures for image recognition and classification.

Filter learn to detect structure

CNNs Learn Hierarchical Features

A landmark 2013 paper, https://arxiv.org/pdf/1311.2901 provided valuable insights into why Convolutional Neural Networks (CNNs) are so effective. The researchers introduced a novel visualisation technique that helps explain how intermediate layers within a CNN process and represent information.

The two examples illustrated in the diagram demonstrate how CNNs learn features hierarchically. Filters in the initial layers identify simple visual patterns, such as edges and colour regions, while deeper layers progressively combine these basic features to recognise more complex structures and patterns.

Visualizing and Understanding Convolutional Networks

Top: A deconvnet layer (left) attached to a convnet layer (right). The deconvnet will reconstruct an approximate version of the convnet features from the layer beneath. Bottom: An illustration of the unpooling operation in the deconvnet, using switches which record the location of the local max in each pooling region (colored zones) during pooling in the convnet.

Convnet Visualization

Visualisation of features in a fully trained model. For layers 2–5 we show the top 9 activations in a random subset of feature maps across the validation data, projected down to pixel space using our deconvolutional network approach. Our reconstructions are not samples from the model: they are reconstructed patterns from the validation set that cause high activations in a given feature map. For each feature map we also show the corresponding image patches. Note: (i) the the strong grouping within each feature map, (ii) greater invariance at higher layers and (iii) exaggeration of discriminative parts of the image, e.g. eyes and noses of dogs (layer 4, row 1, cols 1). Best viewed in electronic form

4. Max Pooling Layers

In CNN architectures, the spatial dimensions of feature maps are commonly reduced at regular intervals using pooling layers. These layers are generally applied after a sequence of convolutional layers to decrease the spatial size of the resulting activation maps. Pooling can be understood as a process of summarising the most relevant information within a feature map.

By reducing the dimensions of the input passed to subsequent layers, pooling helps decrease the overall number of parameters and computational operations required by the network. This makes the training process more efficient and can also help reduce the risk of overfitting by limiting the complexity of the model.

• Single depth slice (4 × 4): the same numbers as your reference, split into four 2 × 2 windows in pink, green, yellow and blue. The largest value in each window is circled in that window’s colour.
• Arrow: labelled “Max pooling, 2 × 2 window, stride 2”.
• Output (2 × 2): 7, 7 / 6, 8, with each cell in the colour of its source window so readers can trace each result back.

A note at the bottom says the size drops from 4 × 4 to 2 × 2 with no learnable parameters. That second point is worth stating, because pooling has no weights to train, unlike the convolution layers.

The size follows the same formula as your convolution images: ⌊(4 − 2) / 2⌋ + 1 = 2. That link is a good place to remind readers that the output formula works for pooling too, with the window size in place of the kernel size and no padding.

This image belongs after the filters image, as the step after convolution and activation. Pooling shrinks each feature map while keeping the strongest responses, which also makes the network less sensitive to small shifts in the image. That connects to the previous image, where the green cells are the strong responses that max pooling would preserve.

Max pooling

Pooling is a downsampling operation that uses a two-dimensional sliding window to reduce the spatial dimensions of an input feature map. This window moves across the input according to a configurable parameter known as the stride, which determines how many pixels the filter shifts between consecutive positions. There are two main types of pooling operations: average pooling and max pooling, with max pooling being the more commonly used approach. In max pooling, the values covered by the filter at each position are passed through a max() operation. The largest value within that window is selected and added to the output feature map. For example, as illustrated above, a 4×4 input feature map processed using a 2×2 filter and a stride of 2 produces a downsampled output of size 2×2. Unlike convolutional filters, pooling filters do not contain trainable parameters. Instead, the filter simply defines the size of the window over which the maximum value is selected.

5. Convolutional Block

Let’s examine a simple convolutional block located at the beginning of a CNN. To make the illustration easier to understand, consider a block containing just one convolutional layer with a single filter. The input to this layer is an image. In an architecture such as VGG-16, the input is a colour image represented by three channels: Red, Green, and Blue (RGB). Once the convolution operation is complete, a max pooling layer is applied to reduce the spatial dimensions of the resulting activation map.

The main distinction between this simplified example and the convolutional blocks used in architectures such as VGG-16 is the number of layers and filters. In practice, these blocks typically contain two or three consecutive convolutional layers before the max pooling layer. Furthermore, each convolutional layer generally uses at least 32 filters to extract a wider range of visual features.

Convolutional Block

• Left to right: the colour input (224 × 224 × 3), one 3 × 3 × 3 filter, the activation function card, the activation map (224 × 224 × 1), then max pooling down to the pooled map (112 × 112 × 1, in pink). A final arrow marks it as the input to the next block.
• Parameter count: 27 weights + 1 bias = 28 trainable parameters, shown under the filter.
• Three levels of brackets: Convolution, Activation and Max pooling. Convolutional layer, which covers convolution plus activation. Convolutional block, which covers the whole chain through to max pooling.

The sizes agree with the max pooling image: a 2 × 2 window with stride 2 gives

⌊(224 − 2) / 2⌋ + 1 = 112.

One change from your reference is that the pooled map is labelled “Pooled map” and has a note that it is the downsized activation map, so the two maps can’t be confused. The layer and block brackets are the main teaching point here, because readers often mix up “layer” and “block”. It’s worth defining both in one sentence each in the text.

This is the natural closing image for Section 4. It ties together the earlier images: the formula and padding examples, the Sobel kernel, the single and double filters, the filter patterns and max pooling. If you want, I can also write a short caption for each of the nine images.

Neuron Connectivity in CNNs

Because of the complexity of Convolutional Neural Networks (CNNs), architectural diagrams rarely illustrate individual neurons and their weighted connections. Representing these connections visually can be challenging because convolutional filters use shared weights across multiple neurons in the input layer.

However, each neuron in an output activation map is connected only to a small group of neurons in the input volume through the filter’s weights. For example, when using a 3x3 filter, each output neuron processes information from just nine input neurons. This means that the neuron does not examine the entire input image but focuses on a specific local region determined by the filter’s spatial dimensions.

This local region of the input is known as the receptive field, illustrated in green in the diagram. It represents the area of the input image that contributes to the computation of a particular neuron in the output activation map.

6. Fully Connected Classifier

The fully connected (dense) layers of a CNN transform the extracted features into class probabilities. In VGG-16, the output of the final convolutional block (Conv-5) is a set of activation maps with shape 7 × 7 × 512. The number of channels at key points in the architecture is shown in the figure above.

Before this data can flow through the classifier, it must be flattened into a one-dimensional vector of length 25,088 (7 × 7 × 512). This vector is then fully connected to FC-6, as shown below.

• Feature extractor: five convolutional blocks (Conv-1 to Conv-5), each ending in a yellow max pooling layer. Blocks 1 and 2 have two conv layers each, and blocks 3, 4 and 5 have three.
• Classifier: Flatten, then three dense layers (FC-6, FC-7, FC-8) in green, then an orange SoftMax.
• Channels row: dots under each block end with the channel counts 64, 128, 256, 512 and 512, then 4096, 4096 and 1000 for the dense layers.
• Input and Output: pink arrows at each end.

Fully Connected Classifier

• Conv-5 activation maps: 7 × 7 with 512 channels.
• Flatten: produces a vector of shape [25088, 1], since 7 × 7 × 512 = 25,088.
• FC-6, FC-7 and FC-8: shapes [4096, 1], [4096, 1] and [1000, 1], with “Fully connected” labels between the columns.
• Softmax: turns the 1000 outputs into class probabilities, y′.
• Argmax(y′): picks index 3, so the prediction is “Cat”. The 0.94 cell is boxed in red, with a dashed line leading to the argmax box.
• Brackets: “Reshaped data” and “VGG-16 dense classifier”.

Feature Maps

Why fully connected layers are important and well-suited to image classification tasks. Recall that the number of neurons in the network’s output layer corresponds to the total number of classes the model is designed to recognise. Each output neuron represents a specific class. After the Softmax function is applied, the value produced by each output neuron represents the predicted probability that the input image belongs to its corresponding class. In this way, the network can estimate the likelihood of the image belonging to each possible category.

7. Regularisation and Advanced CNN Concepts

The activation maps produced by the final convolutional layer of a trained CNN contain meaningful information about the contents of the input image. Each spatial position within a feature map maintains a relationship with a corresponding region of the original image. Therefore, using fully connected layers in the classifier enables the network to combine and analyse information extracted from across the entire image. By connecting the flattened output of the final convolutional layer to the classifier, the network can consider features from all spatial locations when making its classification decision.

Note: Flattening the output of Conv-5 is necessary to pass the activation maps into the fully connected classifier. However, this operation does not change the underlying spatial meaning of the extracted features. It simply reorganises the data into a one-dimensional format so that it can be processed by the subsequent layers.

To understand this more clearly, consider a trained CNN in which all network weights have already been learned and remain fixed. Now, suppose we provide two different input images: one containing a car and another containing an elephant. As each image passes through the network, the final convolutional layer produces different activation maps because the visual features present in the two images are different. When these activation maps are passed through the fully connected classifier, the combination of extracted features and learned weights produces different activation patterns in the output neurons. Consequently, the output neuron associated with the car class will produce a higher activation for the car image, while the neuron associated with the elephant class will produce a higher activation for the elephant image.

The dotted lines illustrated between the fully connected layers represent conceptual pathways that contribute to stronger activations in particular output neurons. These pathways can be understood as combinations of learned weights and feature activations that influence the final classification. In a real VGG-16 architecture, there are more than 100 million connections between the flattened activation maps and the first fully connected layer of the classifier. It is also important to note that the weights remain identical in both examples because they belong to the same trained network. The values shown in the diagram are illustrative rather than actual model outputs.

For problems beyond simple illustrative examples, it is impractical to understand every mathematical operation involved in producing the final classification. However, at a conceptual level, we know that the network’s weights are systematically learned during training by minimising a loss function. Through this optimisation process, the network learns to extract meaningful features from input images and activate the appropriate output neurons for the corresponding classes.

This is the fundamental principle behind how a CNN transforms extracted visual features into class probabilities and makes its final classification prediction.

Dropout

Dropout is a regularisation technique used in CNNs to reduce overfitting. It can be applied to fully connected or convolutional layers by randomly setting a proportion of neuron activations to zero during training. This prevents the network from becoming overly dependent on particular neurons or connections and encourages it to learn more robust and generalisable features.

Batch Normalisation

Batch Normalisation is a technique used to stabilise and accelerate the training process in CNNs. It works by normalising the output of a previous layer using the mean and standard deviation of the current mini-batch. The normalised values are then adjusted using learnable scaling and shifting parameters. This helps maintain more consistent activation distributions during training and can improve the network’s ability to generalise to unseen data.

3D Convolution

Unlike standard 2D convolution, which operates across the height and width dimensions, 3D convolution uses filters that move across three dimensions: height, width, and depth. This allows the network to capture spatial and temporal or volumetric patterns in data such as medical scans and video sequences.

8. Conclusion

Convolutional Neural Networks (CNNs) have become one of the most important architectures in deep learning, particularly for tasks involving image and visual data. Their ability to automatically learn meaningful features from raw inputs makes them highly effective for image classification, object detection, facial recognition, and many other computer vision applications.

Throughout this guide, we explored the fundamental components of CNN architectures, including convolutional layers, filters, kernels, stride, padding, and pooling operations. We also examined how CNNs progressively learn hierarchical features, beginning with simple patterns such as edges and colour regions before developing the ability to recognise more complex structures. We then explored how convolutional blocks extract visual information, how receptive fields determine neuron connectivity, and how fully connected layers transform the extracted features into class predictions. Using the VGG-16 architecture as an example, we demonstrated how feature extraction, flattening, and classification work together within a CNN.

Finally, we introduced additional techniques such as dropout, batch normalisation, and 3D convolution, which extend the capabilities of CNNs and help improve training efficiency and generalisation.

References :

https://cs231n.github.io/convolutional-networks/#conv

https://arxiv.org/pdf/1311.2901

https://www.upgrad.com/blog/basic-cnn-architecture/

https://poloclub.github.io/cnn-explainer/#article-convolution

https://www.appliedaicourse.com/blog/convolutional-neural-network-cnn-in-machine-learning/

https://encord.com/blog/convolutional-neural-networks-explained/