Image Processing with Deep Neural Networks
The pre-trained convolutional neural network (CNN) architecture, like VGG16, is a model that has been trained on the massive ImageNet database (over 14 million images). The model’s 138 million parameters are already tuned, meaning it knows how to recognise edges, shapes, and textures and features of images. In this case, you don’t need to train a model from scratch, which is computationally expensive.
The project will use a pre-trained CNN model, VGG16, to extract features of animal images from two datasets. After extracting image features, I will classify images using a CNN-based linear classifier and KNN, and critically compare their performance based on confusion matrices, precision, recall and F1-score
Two data sets, OxfordPets (OP)and Kaggle cats and dogs (KCD), are used for Image Processing in this article. OP data set has 7390 images with 35 categories. KCD has 24959 images with 2 types (dogs and cats).
Feature Extraction
Before feature extraction, I will clean images to maintain quality. I designed some algorithms to detect too dark or too light images, blurry or noisy images. I used the low Laplacian to identify blur, and dark and light ratios that are greater than 80% to identify too dark or too light images. Additionally, a pre-trained model requires the size of all the input images to be 224×224, so images less than 200×200 are recognised as bad images because resizing very small images to 224×224 can introduce significant interpolation artefacts and blur, which degrades the quality of the extracted features in VGG16[1].
#detect low Laplacian of an image
def is_blurry(img_np,gray,thresh=20):
fm = cv2.Laplacian(gray, cv2.CV_64F).var()
return fm < thresh, fm
#identify too dark or too light images
def blacklight_detection(gray):
# less than 30 or greater than 230
dark_ratio=np.mean(gray<30)
light_ratio=np.mean(gray>230)
return dark_ratio,light_ratio
gray = cv2.cvtColor(img_np, cv2.COLOR_RGB2GRAY)
dark_ratio,light_ratio=blacklight_detection(gray)
if dark_ratio>0.8:
BAD_IMAGES.append((path, f"Too dark image:(ratio={dark_ratio:.2f})"))
if light_ratio>0.8:
BAD_IMAGES.append((path, f"Too light image:(ratio={light_ratio:.2f})"))#detect small size of images
h, w = img_np.shape[:2]
if h < 200 or w < 200:
BAD_IMAGES.append((path, f"Too small: ({h}x{w})"))The total number of bad images is 2497. Among them, one image could not be loaded (collapsed image), 205 images are blurry (Laplacian < 20), 2 images are too dark, 10 images are too light, and 26 images are centre blurry. Most of the bad images are too small (2253 images).
The first figure is one of the bad images detected with a low Laplacian (9.9). Right image is too light; The image has 0.81 light ratio and is low quality, which is not good for feature extraction.
The OxfordPetDataset class defines how individual samples are accessed by mapping an index to an image–label pair. The DataLoader repeatedly queries the dataset, groups samples into mini-batches, and handles batching and filtering. During iteration, each batch of images is passed through the pre-trained VGG16 to extract features, and the corresponding labels are collected. Finally, features and labels from all batches are concatenated to form the full feature matrix and label vector. The complete code can be found on my portfolio website ‘code’ part.
Get Xiaomei Ge’s stories in your inbox
Join Medium for free to get updates from this writer.
The final fully connected classification layer of VGG16 is removed, and the remaining network is used to extract deep image features. All input images are resized to 224×224 pixels and normalised using ImageNet statistics before being fed into the model. For each image, a fixed-length (4096) feature vector is obtained from the network output. These feature vectors are subsequently used for dimensionality reduction, clustering, and classification.
#Create OxfordPet Dataset class
class OxfordPetDataset(torch.utils.data.Dataset):
def __init__(self, img_dir, transform):
self.img_dir = img_dir
self.transform = transform
self.files = [f for f in os.listdir(img_dir) if f.endswith(".jpg")]
self.labels = [f.split("_")[0] for f in self.files]
self.class_to_idx = {cls: i for i, cls in enumerate(sorted(set(self.labels)))}
def __len__(self):
return len(self.files)
def __getitem__(self, idx):
filename = self.files[idx]
path = os.path.join(self.img_dir, filename)
img = load_and_check_image(path)
if img is None:
return None
if self.transform is not None:
img = self.transform(img)
label = self.class_to_idx[self.labels[idx]]
return img, label
Image Clustering
Using t-SNE and PCA to reduce the dimensionality of image feature vectors and visualise them in 2D.
After feature extraction, I will apply K-means to these two datasets. The number of clusters is determined by both the ground-truth number of classes and the K value that yields the lowest Davies–Bouldin Index (DBI) during K-means clustering[2]. These K values are compared, and when the DBI-optimal K differs from the ground-truth class number, the discrepancy is further analysed.
A. Kaggle Cats and Dogs Dataset
Training a K-means model by using ground-truth class number (2), the Purity and BDI are 0.97 and 2.68, respectively. Train K-means with a K value ranging from 2 to 8. As shown in Figure 1 DBI-optimal K value is equal to the true classes of the data set. This indicates that in the latent space, the extracted features exhibit a clustering structure that is consistent with the ground-truth class labels. This means features are highly discriminative for separating the two classes, and K-means clustering can reliably recover the underlying class structure of the dataset as well.
B. OxfordPets Dataset
While the DBI-optimal K value is 22 in the OxfordPets dataset (Figure 2), this is divergent from the number of true labels (35). This suggests that the extracted image features form 22 more separable clusters in the feature space, indicating that some fine-grained classes are not well separated by the features. This may be due to visual similarity among certain pet breeds. To investigate this issue further, the code below is used to calculate the top two ground-truth class proportions within each of the 22 clusters.
from collections import Counter
for c in np.unique(cluster_labels_22):
idx = np.where(cluster_labels_22 == c)[0]
labels_in_cluster = true_labels[idx]
counter = Counter(labels_in_cluster)
total = len(labels_in_cluster)
most_common = counter.most_common(2)
label2name = {v: k for k, v in class_to_idx.items()}
print(f"Cluster {c} (size={total}):")
for label, count in most_common:
print(f" Class {label2name[label]}: {count/total:.2f}")The table below shows that there are 8 clusters that include the combined proportion of the top two ground-truth classes exceeding 70%, and their individual proportions are relatively close, meaning K-means found these top two breeds are same cluster in feature space. For example, Miniature Dog and Chihuahua Dog are two breeds that share very similar visual characteristics, such as small body size, short muzzles, and large eyes. When images lack clear scale or distinctive features, the model tends to focus on general “small dog” traits.
The performance of K-means on OxforedPet dataset highlights a bottleneck of unsupervised learning for classification tasks when the class features are highly similar, and the number of classes is large. However, supervised learning classifiers perform better in the same scenario.
Image Classification
The extracted VGG16 features are classified using a CNN-based linear classifier implemented as a single fully connected layer. A KNN is used as a baseline classifier for comparison
A. Kaggle Cats and Dogs Dataset
Since this dataset has only two classes and is naturally separable, it follows that CNN-based linear classifier and KNN models achieve strong classification performance, with accuracy exceeding 98%.
B. OxfordPets Dataset
Figure 3&4 show each class precision, recall, and F1-score of the CNN and the KNN classifier on the Oxford Pets dataset. These two models perform well across most classes, with precision, recall, and F1-scores generally above 0.85, which suggests that the extracted features are effective for subtle difference pet breed classification. However, the DNN (CNN-based linear classifier) demonstrates more stable performance across classes, with smoother metric curves and fewer extreme drops in recall or F1-score. While KNN exhibits larger fluctuations for certain classes (class1,8,9), suggesting that distance-based classification is more sensitive to class overlap.
Confusion matrices (Figure 5) further support this observation. Both models show correct classification for most samples. Compared to KNN, the CNN produces fewer off-diagonal errors in highlight regions, implying better separation of confusing classes.
Conclusion
Overall, high-dimensional image features extracted by the pre-trained VGG16 model are effective for image classification.
For the Cats & Dogs dataset, which contains visually distinct categories, both supervised and unsupervised methods perform well. In particular, the latent feature space forms two natural and well-separated clusters that closely align with the ground-truth labels, indicating strong semantic separability of the extracted features.
However, for the OxfordPets dataset, which involves a large number of visually similar breeds, the clustering structure in the feature space differs from the ground-truth class labels. Unsupervised clustering methods tend to identify a smaller number of more separable clusters, as similar breeds exhibit overlapping feature representations and cannot form clearly separated clusters in the latent space. In this scenario, supervised learning methods achieve better classification performance.
The CNN-based linear classifier learns a global decision boundary optimised for class discrimination, whereas KNN relies on local distance comparisons and is more sensitive to feature overlap among similar classes. As a result, the CNN-based classifier provides more stable and robust performance on fine-grained datasets.
Reference
[1] Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 60.
[2] Davies, D. L., & Bouldin, D. W. (1979). A cluster separation measure.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 1(2), 224–227.
