<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tony Chisenga on Medium]]></title>
        <description><![CDATA[Stories by Tony Chisenga on Medium]]></description>
        <link>https://medium.com/@tony_chisenga?source=rss-4f29fe44b012------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ZhE4LG2rpQHaOQUM</url>
            <title>Stories by Tony Chisenga on Medium</title>
            <link>https://medium.com/@tony_chisenga?source=rss-4f29fe44b012------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:57:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tony_chisenga/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Training a Conditional Generative Adversarial Network(cGAN) for Image Deblurring]]></title>
            <link>https://medium.com/@tony_chisenga/training-a-conditional-generative-adversarial-network-cgan-for-image-deblurring-157827597bc1?source=rss-4f29fe44b012------2</link>
            <guid isPermaLink="false">https://medium.com/p/157827597bc1</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Tony Chisenga]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 20:11:59 GMT</pubDate>
            <atom:updated>2025-08-26T20:11:59.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I59nwMtXbEja7-sGLQm7JA.png" /></figure><p>Image deblurring is a classic computer vision task focused on restoring a sharp image from a blurred input. This project implements a Conditional Generative Adversarial Network (cGAN) that learns the direct mapping from blurred to sharp imagery directly from data. The model is trained on the publicly available Wider-Face dataset from <a href="https://www.kaggle.com/datasets/jishnuparayilshibu/a-curated-list-of-image-deblurring-datasets">Kaggle</a>, which contains 4,530 images for training, with 567 each reserved for validation and testing. By learning from paired examples of “before” (blurry) and “after” (sharp) images, the model is designed to reverse the blurring process and generate visually plausible restorations.</p><p>Let us import the necessary packages and load the data.</p><pre>import os<br>import random<br>import time<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import tensorflow as tf<br>from IPython.display import clear_output<br>from PIL import Image<br><br>DATA_DIR = &#39;/kaggle/input/a-curated-list-of-image-deblurring-datasets/DBlur/Wider-Face&#39;</pre><p>We will now define a utility function to print the directory structure of the data. Additionally, we will write code to verify the number of samples in the train, validation and test splits.</p><pre>def print_directory_structure(data_dir):<br>    &quot;&quot;&quot;<br>    Prints the directory structure<br>    &quot;&quot;&quot;<br>    for root, dirs, files in os.walk(data_dir):<br>        level = root.replace(data_dir, &#39;&#39;).count(os.sep)<br>        indent = &#39;    &#39; * level<br>        print(f&quot;{indent}{os.path.basename(root)}/&quot;)<br><br>print_directory_structure(DATA_DIR)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*mWmfrOpvjKQ2553zfDyxDw.png" /></figure><pre>splits = [&#39;train&#39;, &#39;validation&#39;,&#39;test&#39;]<br><br>for split in splits:<br>    sharp_imgs_path = os.path.join(DATA_DIR, split, &#39;sharp&#39;)<br>    blur_imgs_path = os.path.join(DATA_DIR, split, &#39;blur&#39;)<br><br>    sharp_imgs_count = len(os.listdir(sharp_imgs_path))<br>    blur_imgs_count = len(os.listdir(blur_imgs_path))<br><br>    print(f&quot;{split.capitalize()} dataset:&quot;)<br>    print(f&quot;  Sharp Images: {sharp_imgs_count}&quot;)<br>    print(f&quot;  Blur Images: {blur_imgs_count}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*XlcSqHgdmpHjCytT7rOArA.png" /></figure><h4>Data Visualization</h4><p>Let us define a function that randomly selects a specified number of image pairs from a chosen dataset split and displays them. Each row of the grid shows a corresponding sharp image and its blurred counterpart side-by-side.</p><pre>def visualize_sharp_blur_pairs(data_dir, split, num_samples):<br>    sharp_dir = os.path.join(data_dir, split, &#39;sharp&#39;)<br>    blur_dir = os.path.join(data_dir, split, &#39;blur&#39;)<br>    <br>    # Get list of all images in sharp dir<br>    all_images = [f for f in os.listdir(sharp_dir) if f.endswith(&#39;.jpg&#39;)]<br>    <br>    # Randomly select samples<br>    selected_images = random.sample(all_images, min(num_samples, len(all_images)))<br>    <br>    # Plot<br>    fig, axes = plt.subplots(len(selected_images), 2, figsize=(10, 5 * len(selected_images)))<br>    <br>    if len(selected_images) == 1:<br>        axes = axes.reshape(1, -1)<br>    <br>    for i, img_name in enumerate(selected_images):<br>        # Load images<br>        sharp_img = Image.open(os.path.join(sharp_dir, img_name))<br>        blur_img = Image.open(os.path.join(blur_dir, img_name))<br>        <br>        # Plot sharp image<br>        axes[i, 0].imshow(sharp_img)<br>        axes[i, 0].set_title(f&#39;Sharp: {img_name}&#39;, fontsize=12)<br>        axes[i, 0].axis(&#39;off&#39;)<br>        <br>        # Plot blur image<br>        axes[i, 1].imshow(blur_img)<br>        axes[i, 1].set_title(f&#39;Blur: {img_name}&#39;, fontsize=12)<br>        axes[i, 1].axis(&#39;off&#39;)<br>    <br>    plt.tight_layout()<br>    plt.show()<br><br>visualize_sharp_blur_pairs(DATA_DIR, &#39;train&#39;, 3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*P-8KWeblda8dqQiqeCl6Cg.png" /></figure><pre>visualize_sharp_blur_pairs(DATA_DIR, &#39;test&#39;, 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*10PZ8KSrsLv5UNuFWWXlRA.png" /></figure><pre>visualize_sharp_blur_pairs(DATA_DIR, &#39;validation&#39;, 2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*qXCSo2XfpXaLLBq7pHIOkw.png" /></figure><p><strong>Data Preparation for Model Training</strong></p><p>Let us now prepare the data for training the GAN. We will create a pipeline that loads pairs of blurry and sharp images, preprocesses them to a consistent size and pixel range, and serves them in randomized batches to the model during training. This pipeline consists of three core functions namely load_image(), load_image_pair(), and create_dataset().</p><p><strong>load_image(): </strong>This function loads a single image file from the provided path and applies a standard preprocessing pipeline. It reads the file, decodes it from JPEG into a 3-channel tensor, converts its data type to float32, resizes it to the defined target dimensions, and finally normalizes its pixel values from the range [0, 1] to [-1, 1], which is a common requirement for training GANs.</p><p><strong>load_image_pair(): </strong>This function utilizes the load_imageto load and preprocess each image, ensuring they are a matched pair, and then returns them as a tuple (blur_image, sharp_image).</p><p><strong>create_dataset():</strong> This function constructs a TensorFlow Dataset object for efficient data loading and processing. Firstly, it gathers the file paths for all image pairs in the specified dataset split, creates a dataset from these paths and then maps the load_image_pair function to each pair to load the actual images. After that, it shuffles the data to minimize order bias, groups the image pairs into batches for model training, and finally sets up prefetching to optimize performance by preparing subsequent batches while the current one is being processed.</p><pre>BATCH_SIZE = 32<br>IMG_HEIGHT = 256  <br>IMG_WIDTH = 256  <br>BUFFER_SIZE = 1000<br><br>def load_image(image_path):<br>    image = tf.io.read_file(image_path)<br>    image = tf.image.decode_jpeg(image, channels=3)<br>    image = tf.image.convert_image_dtype(image, tf.float32)<br>    image = tf.image.resize(image, [IMG_HEIGHT, IMG_WIDTH]) <br>    image = (image - 0.5) * 2.0 # Normalize to [-1, 1]<br>    return image<br><br>def load_image_pair(blur_path, sharp_path):<br>    blur_image = load_image(blur_path)<br>    sharp_image = load_image(sharp_path)<br>    return blur_image, sharp_image<br><br>def create_dataset(data_dir, split=&#39;train&#39;, batch_size=BATCH_SIZE):<br>    blur_dir = os.path.join(data_dir, split, &#39;blur&#39;)<br>    sharp_dir = os.path.join(data_dir, split, &#39;sharp&#39;)<br>    <br>    # Get all image files<br>    image_files = [f for f in os.listdir(sharp_dir) if f.endswith(&#39;.jpg&#39;)]<br>    <br>    # Create full paths<br>    blur_paths = [os.path.join(blur_dir, fname) for fname in image_files]<br>    sharp_paths = [os.path.join(sharp_dir, fname) for fname in image_files]<br>    <br>    # Create TensorFlow dataset<br>    dataset = tf.data.Dataset.from_tensor_slices((blur_paths, sharp_paths))<br>    dataset = dataset.map(load_image_pair, num_parallel_calls=tf.data.AUTOTUNE)<br>    dataset = dataset.shuffle(BUFFER_SIZE)<br>    dataset = dataset.batch(batch_size)<br>    dataset = dataset.prefetch(tf.data.AUTOTUNE)<br>    <br>    return dataset<br><br><br># Create datasets<br>train_dataset = create_dataset(DATA_DIR, &#39;train&#39;)<br>val_dataset = create_dataset(DATA_DIR, &#39;validation&#39;)<br>test_dataset = create_dataset(DATA_DIR, &#39;test&#39;)</pre><h4>Building the cGAN</h4><p>Our conditional GAN will learn to deblur images by training two competing neural networks: a generator and a discriminator. The generator will attempt to create a sharp version from a blurry input image, while the discriminator will evaluate whether the generated image is real or artificially produced. Through this adversarial process, the generator is forced to continually improve until it can produce highly realistic, deblurred results.</p><p><strong>The generator(U-Net):</strong> First, let us define the building blocks for the U-Net generator. The architecture consists of a downsampling path (encoder), an upsampling path (decoder), and skip connections that bridge them. The downsample blocks progressively reduce the spatial dimensions while increasing the number of feature channels to capture contextual information. On the other hand, the upsample blocks symmetrically increase the resolution and reduce channels back down, using skip connections to preserve details from the encoder. The final layer uses a tanh activation to project the 512-channel features back into a 3-channel RGB image, generating pixel values in the range [-1, 1].</p><pre>OUTPUT_CHANNELS = 3<br><br>def downsample(filters, size, apply_batchnorm=True):<br>  &quot;&quot;&quot;Encoder layer&quot;&quot;&quot;<br>  initializer = tf.random_normal_initializer(0., 0.02)<br>  result = tf.keras.Sequential()<br>  result.add(<br>      tf.keras.layers.Conv2D(filters, size, strides=2, padding=&#39;same&#39;,<br>                             kernel_initializer=initializer, use_bias=False))<br>  if apply_batchnorm:<br>    result.add(tf.keras.layers.BatchNormalization())<br>  result.add(tf.keras.layers.LeakyReLU())<br>  return result<br><br>def upsample(filters, size, apply_dropout=False):<br>  &quot;&quot;&quot;Decoder layer&quot;&quot;&quot;<br>  initializer = tf.random_normal_initializer(0., 0.02)<br>  result = tf.keras.Sequential()<br>  result.add(<br>    tf.keras.layers.Conv2DTranspose(filters, size, strides=2,<br>                                    padding=&#39;same&#39;,<br>                                    kernel_initializer=initializer,<br>                                    use_bias=False))<br>  result.add(tf.keras.layers.BatchNormalization())<br>  if apply_dropout:<br>      result.add(tf.keras.layers.Dropout(0.5))<br>  result.add(tf.keras.layers.ReLU())<br>  return result<br><br>def Generator():<br>  &quot;&quot;&quot;U-Net Generator model&quot;&quot;&quot;<br>  inputs = tf.keras.layers.Input(shape=[256, 256, 3])<br><br>  down_stack = [<br>    downsample(64, 4, apply_batchnorm=False),<br>    downsample(128, 4),<br>    downsample(256, 4),<br>    downsample(512, 4),<br>    downsample(512, 4),<br>    downsample(512, 4),<br>    downsample(512, 4),<br>    downsample(512, 4),<br>  ]<br><br>  up_stack = [<br>    upsample(512, 4, apply_dropout=True),<br>    upsample(512, 4, apply_dropout=True),<br>    upsample(512, 4, apply_dropout=True),<br>    upsample(512, 4),<br>    upsample(256, 4),<br>    upsample(128, 4),<br>    upsample(64, 4),<br>  ]<br><br>  initializer = tf.random_normal_initializer(0., 0.02)<br>  last = tf.keras.layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,<br>                                         strides=2,<br>                                         padding=&#39;same&#39;,<br>                                         kernel_initializer=initializer,<br>                                         activation=&#39;tanh&#39;)<br><br>  x = inputs<br>  skips = []<br>  for down in down_stack:<br>    x = down(x)<br>    skips.append(x)<br>  skips = reversed(skips[:-1])<br><br>  for up, skip in zip(up_stack, skips):<br>    x = up(x)<br>    x = tf.keras.layers.Concatenate()([x, skip])<br><br>  x = last(x)<br>  return tf.keras.Model(inputs=inputs, outputs=x)<br><br>#instantiate generator<br>generator = Generator()</pre><p><strong>The discriminator(PatchGAN): </strong>Unlike a standard discriminator that classifies an entire image as simply “real” or “fake”, the PatchGAN operates on a more nuanced principle. It makes independent predictions for overlapping N x N patches of the image. This approach is powerful because it forces the generator to not just create a globally coherent image, but to produce convincing, high-frequency details in every single region.</p><p>The model’s architecture is designed for this specific task. It takes two separate inputs: a condition image (the blurry input) and a target image (the corresponding sharp image). These are concatenated along the channel dimension — a critical step that defines the discriminator’s role. It does not evaluate the target image in isolation; instead, it assesses whether the target is a plausibly sharp and realistic version of the given blurry condition. This allows the network to learn the complex, pixel-level relationships between an input and its desired output, determining if the pair is structurally consistent.</p><p>This concatenated 6-channel tensor is then fed through a series of convolutional layers (downsample blocks). These layers act as a feature extraction network, progressively reducing the spatial resolution while increasing the depth, thereby building a hierarchical understanding of the image pair.</p><p>The final output is not a single number but a 30x30 grid of predictions. Each value in this grid represents the “realness” score for a specific, corresponding patch (approximately 70x70 pixels) in the original input image.</p><pre>def Discriminator():<br>  &quot;&quot;&quot;PatchGAN Discriminator model.&quot;&quot;&quot;<br>  initializer = tf.random_normal_initializer(0., 0.02)<br><br>  inp = tf.keras.layers.Input(shape=[256, 256, 3], name=&#39;input_image&#39;)<br>  tar = tf.keras.layers.Input(shape=[256, 256, 3], name=&#39;target_image&#39;)<br><br>  # (batch_size, 256, 256, channels*2)<br>  x = tf.keras.layers.concatenate([inp, tar])  <br><br>  down1 = downsample(64, 4, False)(x)  # (batch_size, 128, 128, 64)<br>  down2 = downsample(128, 4)(down1)  # (batch_size, 64, 64, 128)<br>  down3 = downsample(256, 4)(down2)  # (batch_size, 32, 32, 256)<br><br>  zero_pad1 = tf.keras.layers.ZeroPadding2D()(down3)  # (batch_size, 34, 34, 256)<br>  conv = tf.keras.layers.Conv2D(512, 4, strides=1,<br>                                kernel_initializer=initializer,<br>                                use_bias=False)(zero_pad1)  # (batch_size, 31, 31, 512)<br><br>  batchnorm1 = tf.keras.layers.BatchNormalization()(conv)<br>  leaky_relu = tf.keras.layers.LeakyReLU()(batchnorm1)<br>  zero_pad2 = tf.keras.layers.ZeroPadding2D()(leaky_relu)  # (batch_size, 33, 33, 512)<br><br>  last = tf.keras.layers.Conv2D(1, 4, strides=1,<br>                                kernel_initializer=initializer)(zero_pad2)  # (batch_size, 30, 30, 1)<br><br>  return tf.keras.Model(inputs=[inp, tar], outputs=last)<br><br>#instantiate discriminator<br>discriminator = Discriminator()</pre><h4>Losses and Optimizers</h4><p>This part establishes the core adversarial mechanics of the GAN. The discriminator loss function quantifies the model’s ability to distinguish real from generated images by summing two error terms: one penalizing it for misclassifying a real image as fake, and another for misclassifying a generated image as real.</p><p>The generator loss is a composite objective designed to achieve two goals. The first is an adversarial loss that rewards the generator for successfully fooling the discriminator. The second is a reconstruction loss (L1) that ensures pixel-wise accuracy between the generated image and the ground-truth sharp target. Both networks are trained to minimize their respective losses using the Adam optimizer.</p><pre>loss_object = tf.keras.losses.BinaryCrossentropy(from_logits=True)<br>LAMBDA = 100 # Weight for the L1 <br><br>def discriminator_loss(disc_real_output, disc_generated_output):<br>  real_loss = loss_object(tf.ones_like(disc_real_output), disc_real_output)<br>  generated_loss = loss_object(tf.zeros_like(disc_generated_output), disc_generated_output)<br>  total_disc_loss = real_loss + generated_loss<br>  return total_disc_loss<br><br>def generator_loss(disc_generated_output, gen_output, target):<br>  gan_loss = loss_object(tf.ones_like(disc_generated_output), disc_generated_output)<br>  l1_loss = tf.reduce_mean(tf.abs(target - gen_output))<br>  total_gen_loss = gan_loss + (LAMBDA * l1_loss)<br>  return total_gen_loss, gan_loss, l1_loss<br><br># Optimizers<br>generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)<br>discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)</pre><p>Let us now configure a checkpoint manager to save and restore the state of our models and optimizers.</p><pre>OUTPUT = &#39;/kaggle/working/&#39;<br>CHECKPOINT_DIR = f&quot;{OUTPUT}training_checkpoints&quot;<br><br>checkpoint_prefix = os.path.join(CHECKPOINT_DIR, &quot;ckpt&quot;)<br>checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,<br>                                 discriminator_optimizer=discriminator_optimizer,<br>                                 generator=generator,<br>                                 discriminator=discriminator)</pre><h4>The Training Step</h4><p>We now define a function which will be the core training step of our adversarial network. It will begin by using the generator to create a deblurred image from a blurry input. Then the discriminator will evaluate both the real and generated image pairs, allowing us to calculate its loss. We also compute the generator’s composite loss based on its ability to fool the discriminator and its reconstruction accuracy. Finally, these losses will be used to compute and apply gradients, updating the weights of both models to improve their performance.</p><pre>log_dir=&quot;logs/&quot;<br>summary_writer = tf.summary.create_file_writer(<br>  log_dir + &quot;fit/&quot; + time.strftime(&quot;%Y%m%d-%H%M%S&quot;))<br><br>@tf.function<br>def train_step(input_image, target, step):<br>  with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:<br>    gen_output = generator(input_image, training=True)<br><br>    disc_real_output = discriminator([input_image, target], training=True)<br>    disc_generated_output = discriminator([input_image, gen_output], training=True)<br><br>    gen_total_loss, gen_gan_loss, gen_l1_loss = generator_loss(disc_generated_output, gen_output, target)<br>    disc_loss = discriminator_loss(disc_real_output, disc_generated_output)<br><br>  generator_gradients = gen_tape.gradient(gen_total_loss,<br>                                          generator.trainable_variables)<br>  discriminator_gradients = disc_tape.gradient(disc_loss,<br>                                               discriminator.trainable_variables)<br><br>  generator_optimizer.apply_gradients(zip(generator_gradients,<br>                                          generator.trainable_variables))<br>  discriminator_optimizer.apply_gradients(zip(discriminator_gradients,<br>                                              discriminator.trainable_variables))<br><br>  with summary_writer.as_default():<br>    tf.summary.scalar(&#39;gen_total_loss&#39;, gen_total_loss, step=step//1000)<br>    tf.summary.scalar(&#39;gen_gan_loss&#39;, gen_gan_loss, step=step//1000)<br>    tf.summary.scalar(&#39;gen_l1_loss&#39;, gen_l1_loss, step=step//1000)<br>    tf.summary.scalar(&#39;disc_loss&#39;, disc_loss, step=step//1000)</pre><p>Let us define the fit() function that will manage the entire training process. This function iterates through the dataset for a specified number of epochs, calls the train_step() function for each batch, and saves checkpoints periodically.</p><pre>def fit(train_ds, test_ds, steps):<br>  example_input, example_target = next(iter(test_ds.take(1)))<br>  start = time.time()<br><br>  for step, (input_image, target) in train_ds.repeat().take(steps).enumerate():<br>    if (step) % 1000 == 0:<br>      if step != 0:<br>        print(f&#39;Time taken for 1000 steps: {time.time() - start:.2f} sec\n&#39;)<br>      start = time.time()<br>      print(f&quot;Step: {step//1000}k&quot;)<br><br>    train_step(input_image, target, step)<br><br>    # Training visualizations<br>    if (step + 1) % 5000 == 0:<br>      checkpoint.save(file_prefix=checkpoint_prefix)<br><br>  checkpoint.save(file_prefix=checkpoint_prefix)</pre><p>Now, we can finally start training the model with the number of steps set to 40000.</p><pre>fit(train_dataset, test_dataset, steps=40000)</pre><p>After training is complete, we restore the latest checkpoint and run the generator on a few images from the test set to visualize the deblurring results.</p><pre>#restore the latest checkpoint<br>checkpoint.restore(tf.train.latest_checkpoint(CHECKPOINT_DIR))<br><br>def generate_images(model, test_input, tar):<br>  prediction = model(test_input, training=True)<br>  plt.figure(figsize=(15, 5))<br><br>  display_list = [test_input[0], tar[0], prediction[0]]<br>  title = [&#39;Input Image (Blurred)&#39;, &#39;Ground Truth (Sharp)&#39;, &#39;Predicted Image&#39;]<br><br>  for i in range(3):<br>    plt.subplot(1, 3, i+1)<br>    plt.title(title[i])<br><br>    # Getting the pixel values in the [0, 1] range to plot.<br>    plt.imshow(display_list[i] * 0.5 + 0.5)<br>    plt.axis(&#39;off&#39;)<br>  plt.show()<br><br># Run the model on one example from the test set<br>for example_input, example_target in test_dataset.take(3):<br>  generate_images(generator, example_input, example_target)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*wZvT7jPNwqnNwRoJbxQRyQ.png" /></figure><h4>Conclusion</h4><p>We have trained a cGAN model which is capable of learning the transformation from a blurred image to a sharp one. After the training process is complete, the Generator model can be used as a standalone tool. When given a new, unseen blurry image, it will apply its learned knowledge to produce a high-quality, deblurred version, restoring details lost to motion or focus issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=157827597bc1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Class Semantic Segmentation of Aerial Imagery using U-Net with EfficientNetB4]]></title>
            <link>https://medium.com/@tony_chisenga/multi-class-semantic-segmentation-of-aerial-imagery-using-u-net-with-efficientnetb4-85ac7d7c7d3f?source=rss-4f29fe44b012------2</link>
            <guid isPermaLink="false">https://medium.com/p/85ac7d7c7d3f</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-segmentation]]></category>
            <dc:creator><![CDATA[Tony Chisenga]]></dc:creator>
            <pubDate>Tue, 19 Aug 2025 20:05:13 GMT</pubDate>
            <atom:updated>2025-08-19T20:05:13.265Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Cwb-M4Gp7-0O6nB_3W-ig.png" /></figure><p>Semantic segmentation plays a vital role in modern computer vision. It enables machines to interpret visual scenes with pixel-level precision for applications like autonomous driving, urban planning, and disaster monitoring. Here, we show how we can use segmentation to analyze flood-impacted areas by distinguishing critical classes such as flooded buildings, roads, and water bodies. We will train a U-Net model with an EfficientNetB4 backbone on the publicly available Aerial Imagery Dataset from the FloodNet Challenge on <a href="https://www.kaggle.com/datasets/aletbm/aerial-imagery-dataset-floodnet-challenge">Kaggle</a>.</p><p>Let us import essential libraries and load the dataset. Additionally, we enable mixed precision training with a global float32 policy to optimize GPU performance and prevent data type conflicts.</p><pre>import os<br>import random<br>import cv2<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import tensorflow as tf<br>from matplotlib.colors import ListedColormap<br>from PIL import Image<br>from sklearn.model_selection import train_test_split<br>from tensorflow.keras import layers, models<br>from tensorflow.keras.applications import EfficientNetB4<br>from tensorflow.keras.callbacks import (EarlyStopping, ModelCheckpoint,<br>                                      ReduceLROnPlateau)<br>from tensorflow.keras.layers import Concatenate, UpSampling2D<br>from tensorflow.keras.optimizers import Adam<br>from tensorflow.keras.preprocessing.image import img_to_array, load_img<br>from tensorflow.keras.utils import Sequence<br><br># set mixed precision policy<br>tf.keras.mixed_precision.set_global_policy(&#39;float32&#39;)<br><br>DATA_DIR = &#39;/kaggle/input/aerial-imagery-dataset-floodnet-challenge/FloodNet Challenge - Track 1&#39;<br><br>print(&quot;GPU Available:&quot;, tf.config.list_physical_devices(&#39;GPU&#39;))</pre><h4>Class Definitions</h4><p>The original segmentation masks in the dataset use grayscale values. So we define: GRAYSCALE_CLASS_MAP to associate each pixel value with its semantic class, and CLASS_COLORS to map these classes to distinct RGB colors for visualization. The colors are assigned as follows: Black (0,0,0) for Background, Red (255,0,0) and Dark Red (128,0,0) for Flooded/Non-Flooded Buildings, Yellow (255,255,0) and Dark Yellow (128,128,0) for Flooded/Non-Flooded Roads, Blue (0,0,255) for Water, Dark Green (0,128,0) for Trees, Purple (128,0,128) for Vehicles, Cyan (0,255,255) for Pools, and Green (0,255,0) for Grass.</p><pre>GRAYSCALE_CLASS_MAP = {<br>    0: &quot;Background&quot;,<br>    1: &quot;Building-Flooded&quot;, <br>    2: &quot;Building-Non-Flooded&quot;,<br>    3: &quot;Road-Flooded&quot;,<br>    4: &quot;Road-Non-Flooded&quot;, <br>    5: &quot;Water&quot;,<br>    6: &quot;Tree&quot;,<br>    7: &quot;Vehicle&quot;,<br>    8: &quot;Pool&quot;,<br>    9: &quot;Grass&quot;<br>}<br><br>CLASS_COLORS = {<br>    0: [0, 0, 0],     <br>    1: [255, 0, 0],    <br>    2: [128, 0, 0],     <br>    3: [255, 255, 0],   <br>    4: [128, 128, 0],   <br>    5: [0, 0, 255],    <br>    6: [0, 128, 0],     <br>    7: [128, 0, 128],  <br>    8: [0, 255, 255],   <br>    9: [0, 255, 0] <br>}<br><br>NUM_CLASSES = len(GRAYSCALE_CLASS_MAP)<br>print(f&quot;Number of classes: {NUM_CLASSES}&quot;)</pre><h4>Data Loading and Exploration</h4><p>Let us scan through both “Flooded” and “Non-Flooded” subdirectories to collect valid image-mask pairs and verify that each JPG image has a corresponding mask file with the “_lab.png” suffix. In total, we have 398 training samples.</p><pre>base_path = os.path.join(DATA_DIR, &quot;Train&quot;, &quot;Labeled&quot;)<br><br>all_images = []<br>all_masks = []<br><br># Process both flooded and non-flooded data<br>for category in [&quot;Flooded&quot;, &quot;Non-Flooded&quot;]:<br>    img_dir = os.path.join(base_path, category, &quot;image&quot;)<br>    mask_dir = os.path.join(base_path, category, &quot;mask&quot;)<br>    <br>    print(f&quot;Checking {category} category:&quot;)<br>    print(f&quot;  Image dir exists: {os.path.exists(img_dir)}&quot;)<br>    print(f&quot;  Mask dir exists: {os.path.exists(mask_dir)}&quot;)<br>    <br>    if os.path.exists(img_dir) and os.path.exists(mask_dir):<br>        img_files = [f for f in os.listdir(img_dir) if f.endswith(&#39;.jpg&#39;)]<br>        print(f&quot;  Found {len(img_files)} image files&quot;)<br>        <br>        for img_file in img_files:<br>            base_name = os.path.splitext(img_file)[0]<br>            mask_file = f&quot;{base_name}_lab.png&quot;<br>            <br>            img_path = os.path.join(img_dir, img_file)<br>            mask_path = os.path.join(mask_dir, mask_file)<br>            <br>            if os.path.exists(mask_path):<br>                all_images.append(img_path)<br>                all_masks.append(mask_path)<br><br>print(f&quot;\nTotal training samples found: {len(all_images)}&quot;)<br><br>if len(all_images) == 0:<br>    raise ValueError(&quot;No training data found! Check your data directory structure.&quot;)</pre><h4>Sample Images from the Dataset</h4><p>Let us visualize some images from the dataset. We define the following functions to help us achieve this:</p><ol><li><strong>_get_valid_pairs()</strong>: This helper function scans image and mask directories to find valid matching pairs where each JPG image has a corresponding ‘_lab.png’ mask file, returning a list of (image_filename, mask_filename) tuples.</li><li><strong>_plot_pair(): </strong>This helper displays a single image-mask pair on provided matplotlib axes, showing the original image on one axis and its grayscale mask on another.</li><li><strong>visualize_samples(): </strong>This is the main visualization function that randomly selects and displays image-mask pairs from both Flooded and Non-Flooded categories.</li></ol><pre>def _get_valid_pairs(img_dir, mask_dir):<br>    valid_pairs = []<br>    if not os.path.exists(img_dir) or not os.path.exists(mask_dir):<br>        return []<br>    <br>    for img_file in os.listdir(img_dir):<br>        if img_file.lower().endswith(&#39;.jpg&#39;):<br>            base_name = os.path.splitext(img_file)[0]<br>            mask_file = f&quot;{base_name}_lab.png&quot;<br>            if os.path.exists(os.path.join(mask_dir, mask_file)):<br>                valid_pairs.append((img_file, mask_file))<br>    return valid_pairs<br><br>def _plot_pair(ax_img, ax_mask, img_dir, mask_dir, pair, title_prefix):<br>    img_file, mask_file = pair<br>    try:<br>        img = Image.open(os.path.join(img_dir, img_file))<br>        mask = Image.open(os.path.join(mask_dir, mask_file))<br>        <br>        ax_img.imshow(img)<br>        ax_img.set_title(f&quot;{title_prefix} Image&quot;)<br>        ax_img.axis(&#39;off&#39;)<br>        <br>        ax_mask.imshow(mask, cmap=&#39;gray&#39;)<br>        ax_mask.set_title(f&quot;{title_prefix} Mask&quot;)<br>        ax_mask.axis(&#39;off&#39;)<br>    except FileNotFoundError:<br>        ax_img.text(0.5, 0.5, &#39;File Missing&#39;, ha=&#39;center&#39;, va=&#39;center&#39;)<br>        ax_mask.text(0.5, 0.5, &#39;File Missing&#39;, ha=&#39;center&#39;, va=&#39;center&#39;)<br>        ax_img.axis(&#39;off&#39;)<br>        ax_mask.axis(&#39;off&#39;)<br><br>def visualize_samples(data_dir, num_samples=4, seed=None):<br>    if seed is not None:<br>        random.seed(seed)<br><br>    base_labeled_path = os.path.join(data_dir, &quot;Train&quot;, &quot;Labeled&quot;)<br>    <br>    categories_data = {<br>        &quot;Flooded&quot;: {<br>            &quot;pairs&quot;: _get_valid_pairs(os.path.join(base_labeled_path, &quot;Flooded/image&quot;), os.path.join(base_labeled_path, &quot;Flooded/mask&quot;)),<br>            &quot;img_dir&quot;: os.path.join(base_labeled_path, &quot;Flooded/image&quot;),<br>            &quot;mask_dir&quot;: os.path.join(base_labeled_path, &quot;Flooded/mask&quot;)<br>        },<br>        &quot;Non-Flooded&quot;: {<br>            &quot;pairs&quot;: _get_valid_pairs(os.path.join(base_labeled_path, &quot;Non-Flooded/image&quot;), os.path.join(base_labeled_path, &quot;Non-Flooded/mask&quot;)),<br>            &quot;img_dir&quot;: os.path.join(base_labeled_path, &quot;Non-Flooded/image&quot;),<br>            &quot;mask_dir&quot;: os.path.join(base_labeled_path, &quot;Non-Flooded/mask&quot;)<br>        }<br>    }<br>    <br>    min_available = min(len(data[&quot;pairs&quot;]) for data in categories_data.values())<br>    if min_available &lt; num_samples:<br>        print(f&quot;Warning: Requested {num_samples}, but only {min_available} pairs available. Showing {min_available}.&quot;)<br>        num_samples = min_available<br>    if num_samples == 0:<br>        raise ValueError(&quot;No valid image-mask pairs found.&quot;)<br>        <br>    fig, axes = plt.subplots(4, num_samples, figsize=(4 * num_samples, 12), squeeze=False)<br>    fig.suptitle(&quot;Image-Mask Pairs&quot;, fontsize=16)<br><br>    flooded_samples = random.sample(categories_data[&quot;Flooded&quot;][&quot;pairs&quot;], num_samples)<br>    non_flooded_samples = random.sample(categories_data[&quot;Non-Flooded&quot;][&quot;pairs&quot;], num_samples)<br><br>    for i in range(num_samples):<br>        # Plot Flooded<br>        _plot_pair(axes[0, i], axes[1, i], categories_data[&quot;Flooded&quot;][&quot;img_dir&quot;], categories_data[&quot;Flooded&quot;][&quot;mask_dir&quot;], flooded_samples[i], &quot;Flooded&quot;)<br>        # Plot Non-Flooded<br>        _plot_pair(axes[2, i], axes[3, i], categories_data[&quot;Non-Flooded&quot;][&quot;img_dir&quot;], categories_data[&quot;Non-Flooded&quot;][&quot;mask_dir&quot;], non_flooded_samples[i], &quot;Non-Flooded&quot;)<br><br>    plt.tight_layout(rect=[0, 0, 1, 0.96])<br>    plt.show()<br><br><br>visualize_samples(DATA_DIR, num_samples=5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*u_fnnp2mhaFap7hvS6fAwA.png" /></figure><h4>Mask Value Inspection for Data Validation</h4><p>Let us validate the mask data by randomly sampling five segmentation masks and analyzing their pixel value distributions. For each mask, we will check the unique class values, minimum/maximum range, and spatial dimensions to ensure consistency with expected formats (0–9 class indices and 256x256 resolution). This step will help us to identify potential preprocessing issues, such as incorrect value scaling, before we begin model training.</p><pre>sample_masks = np.random.choice(all_masks, size=min(5, len(all_masks)), replace=False)<br><br>for i, mask_path in enumerate(sample_masks):<br>    mask = load_img(mask_path, color_mode=&#39;grayscale&#39;, target_size=(256, 256))<br>    mask_array = img_to_array(mask, dtype=np.float32).squeeze()<br>    unique_values = np.unique(mask_array)<br>    <br>    print(f&quot;Mask {i+1}: {os.path.basename(mask_path)}&quot;)<br>    print(f&quot;  Unique values: {unique_values}&quot;)<br>    print(f&quot;  Min: {np.min(mask_array)}, Max: {np.max(mask_array)}&quot;)<br>    print(f&quot;  Shape: {mask_array.shape}&quot;)<br>    print()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*JZ4WEumFvYyM9YTfr1fz0w.png" /></figure><h4>Class Weight Calculation for Handling Imbalanced Data</h4><p>Let us define a function to calculate balanced class weights for our dataset. Firstly, the function loads and preprocesses each sampled mask, normalizing values to match the 0–9 class range, then counts pixel occurrences per class. Using inverse frequency weighting with smoothing (to prevent division by zero), it computes normalized weights that give underrepresented classes higher importance during training. Finally, it converts these weights to a TensorFlow constant for model integration.</p><pre>def calculate_class_weights(mask_paths, sample_size=50):<br>    &quot;&quot;&quot;Calculate class weights by sampling actual mask files&quot;&quot;&quot;<br>    class_counts = np.zeros(NUM_CLASSES, dtype=np.float32)<br>    <br>    # Sample random masks to calculate weights<br>    sample_masks = np.random.choice(mask_paths, size=min(sample_size, len(mask_paths)), replace=False)<br>    <br>    for mask_path in sample_masks:<br>        try:<br>            mask = load_img(mask_path, color_mode=&#39;grayscale&#39;, target_size=(256, 256))<br>            mask = img_to_array(mask, dtype=np.float32).squeeze()<br>            <br>            # Handle mask value mapping <br>            if np.max(mask) &gt; 9:<br>                mask = np.round(mask * 9 / 255).astype(np.int32)<br>            else:<br>                mask = mask.astype(np.int32)<br>            <br>            mask = np.clip(mask, 0, NUM_CLASSES - 1)<br>            <br>            # Count pixels for each class<br>            for class_idx in range(NUM_CLASSES):<br>                class_counts[class_idx] += np.sum(mask == class_idx)<br>                <br>        except Exception as e:<br>            print(f&quot;Error processing {mask_path}: {e}&quot;)<br>            continue<br>    <br>    # Calculate weights <br>    total_pixels = np.sum(class_counts)<br>    class_frequencies = class_counts / (total_pixels + 1e-6)<br>    class_weights = 1.0 / (class_frequencies + 1e-6)<br>    <br>    # Normalize weights<br>    class_weights = class_weights / np.sum(class_weights) * NUM_CLASSES<br>    <br>    print(&quot;Class distribution:&quot;)<br>    for i, (count, weight) in enumerate(zip(class_counts, class_weights)):<br>        print(f&quot;  {GRAYSCALE_CLASS_MAP[i]}: {count:.0f} pixels, weight: {weight:.3f}&quot;)<br>    <br>    return class_weights.astype(np.float32)<br><br>class_weights_np = calculate_class_weights(all_masks)<br>class_weights = tf.constant(class_weights_np, dtype=tf.float32)<br>print(f&quot;Class weights tensor shape: {class_weights.shape}&quot;)<br>print(f&quot;Class weights tensor dtype: {class_weights.dtype}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*-6_UwTkSX_ss0KfDrId4WA.png" /></figure><h4>Loss Functions and Evaluation Metrics</h4><p>We define the following functions for computing the loss and evaluation:</p><ul><li>dice_coefficient: Computes the Dice score (F1) to measure overlap between predictions and ground truth masks.</li><li>iou_metric: Calculates Intersection-over-Union (IoU) to evaluate segmentation accuracy per class.</li><li>weighted_categorical_crossentropy: Adapts cross entropy loss with class weights to handle imbalanced data.</li><li>combined_loss: Hybrid loss merging weighted crossentropy and Dice loss for balanced optimization.</li></ul><pre>def dice_coefficient(y_true, y_pred, smooth=1e-6):<br>    &quot;&quot;&quot;Dice coefficient for multi-class segmentation&quot;&quot;&quot;<br>    y_true = tf.cast(y_true, tf.float32)<br>    y_pred = tf.cast(y_pred, tf.float32)<br>    y_pred = tf.nn.softmax(y_pred, axis=-1)<br>    intersection = tf.reduce_sum(y_true * y_pred, axis=[1, 2])<br>    union = tf.reduce_sum(y_true, axis=[1, 2]) + tf.reduce_sum(y_pred, axis=[1, 2])<br>    dice = tf.reduce_mean((2. * intersection + smooth) / (union + smooth))<br>    return dice<br><br>def iou_metric(y_true, y_pred, smooth=1e-6):<br>    &quot;&quot;&quot;IoU metric for multi-class segmentation&quot;&quot;&quot;<br>    y_true = tf.cast(y_true, tf.float32)<br>    y_pred = tf.cast(y_pred, tf.float32)<br>    y_pred = tf.nn.softmax(y_pred, axis=-1)<br>    intersection = tf.reduce_sum(y_true * y_pred, axis=[1, 2])<br>    union = tf.reduce_sum(y_true, axis=[1, 2]) + tf.reduce_sum(y_pred, axis=[1, 2]) - intersection<br>    iou = tf.reduce_mean((intersection + smooth) / (union + smooth))<br>    return iou<br><br>def weighted_categorical_crossentropy(y_true, y_pred):<br>    &quot;&quot;&quot;Weighted categorical crossentropy loss&quot;&quot;&quot;<br>    y_true = tf.cast(y_true, tf.float32)<br>    y_pred = tf.cast(y_pred, tf.float32)<br>    <br>    # Calculate class weights for each pixel<br>    weights = tf.reduce_sum(class_weights * y_true, axis=-1)<br>    <br>    # Calculate categorical crossentropy<br>    ce_loss = tf.keras.losses.categorical_crossentropy(y_true, y_pred)<br>    <br>    # Apply weights<br>    weighted_loss = ce_loss * weights<br>    <br>    return tf.reduce_mean(weighted_loss)<br><br>def combined_loss(y_true, y_pred, alpha=0.7):<br>    &quot;&quot;&quot;Combination of weighted categorical crossentropy and dice loss&quot;&quot;&quot;<br>    # Weighted categorical crossentropy<br>    weighted_ce = weighted_categorical_crossentropy(y_true, y_pred)<br>    <br>    # Dice loss<br>    dice_loss = 1.0 - dice_coefficient(y_true, y_pred)<br>    <br>    return alpha * weighted_ce + (1.0 - alpha) * dice_loss</pre><h4>The Data Generator Class</h4><p>We now define theFloodNetDataGenerator which will enable us to load and preprocess dataset images and their corresponding masks in batches during model training. The class definition and key methods are as follows:</p><ol><li><strong>Initialization: </strong>The __init__ method performs validation of image-mask path pairs to verify file existence, stores configuration parameters (including batch size, image dimensions, and augmentation flag), filters out invalid pairs, and outputs the total count of valid samples.</li><li><strong>Batch Loading: </strong>The __getitem__ method loads and normalizes images to a 0–1 range while processing their corresponding masks by converting 0–255 values into 0–9 class indices (with value clipping to ensure valid ranges) and transforming them into a one-hot encoded format. It handles corrupt files(to maintain robustness) by substituting them with zero arrays and optionally applies augmentations.</li><li><strong>Augmentation:</strong> The apply_augmentation method randomly performs geometric transformations with specified probabilities: horizontal and vertical flips each have a 50% chance of being applied, while 90-degree rotations occur with a 30% probability. Additionally, it maintains perfect synchronization between all image and mask transformations meaning that whenever an image is flipped or rotated, its corresponding mask undergoes the exact same geometric modification simultaneously. This ensure spatial correspondence is preserved throughout the augmentation process.</li></ol><p>The generator outputs batches of:</p><ul><li>Images: [batch_size, 256, 256, 3] float32 tensors (0-1 normalized).</li><li>Masks: [batch_size, 256, 256, num_classes] one-hot encoded float32 tensors.</li></ul><pre>class FloodNetDataGenerator(Sequence):<br>    def __init__(self, image_paths, mask_paths, batch_size=4, img_size=(256, 256), augment=False):<br>        self.image_paths = image_paths<br>        self.mask_paths = mask_paths<br>        self.batch_size = batch_size<br>        self.img_size = img_size<br>        self.augment = augment<br>        self.n_classes = NUM_CLASSES<br>        <br>        # Validate paths<br>        valid_pairs = []<br>        for img_path, mask_path in zip(image_paths, mask_paths):<br>            if os.path.exists(img_path) and os.path.exists(mask_path):<br>                valid_pairs.append((img_path, mask_path))<br>            else:<br>                print(f&quot;Warning: Missing files - {img_path} or {mask_path}&quot;)<br>        <br>        self.image_paths = [pair[0] for pair in valid_pairs]<br>        self.mask_paths = [pair[1] for pair in valid_pairs]<br>        print(f&quot;Valid pairs found: {len(self.image_paths)}&quot;)<br>        <br>    def __len__(self):<br>        return int(np.ceil(len(self.image_paths) / self.batch_size))<br>    <br>    def __getitem__(self, idx):<br>        batch_image_paths = self.image_paths[idx*self.batch_size:(idx+1)*self.batch_size]<br>        batch_mask_paths = self.mask_paths[idx*self.batch_size:(idx+1)*self.batch_size]<br>        <br>        images = []<br>        masks = []<br>        <br>        for img_path, mask_path in zip(batch_image_paths, batch_mask_paths):<br>            try:<br>                # Load and preprocess image<br>                img = load_img(img_path, target_size=self.img_size)<br>                img = img_to_array(img, dtype=np.float32) / 255.0<br>                <br>                # Load grayscale mask<br>                mask = load_img(mask_path, target_size=self.img_size, color_mode=&#39;grayscale&#39;)<br>                mask = img_to_array(mask, dtype=np.float32).squeeze()<br>                <br>                # Handle mask value mapping<br>                unique_values = np.unique(mask)<br>                <br>                # If values are scaled (0-255), map them back to 0-9<br>                if np.max(mask) &gt; 9:<br>                    mask = np.round(mask * 9 / 255).astype(np.int32)<br>                else:<br>                    mask = mask.astype(np.int32)<br>                <br>                # Ensure all values are within valid class range<br>                mask = np.clip(mask, 0, self.n_classes - 1)<br>                <br>                # Data augmentation<br>                if self.augment:<br>                    img, mask = self.apply_augmentation(img, mask)<br>                <br>                # Convert mask to one-hot encoding <br>                mask_categorical = tf.keras.utils.to_categorical(mask, num_classes=self.n_classes).astype(np.float32)<br>                <br>                images.append(img)<br>                masks.append(mask_categorical)<br>                <br>            except Exception as e:<br>                print(f&quot;Error loading {img_path} or {mask_path}: {e}&quot;)<br>                <br>                # Create dummy data to maintain batch size <br>                img = np.zeros((*self.img_size, 3), dtype=np.float32)<br>                mask = np.zeros((*self.img_size, self.n_classes), dtype=np.float32)<br>                images.append(img)<br>                masks.append(mask)<br>            <br>        return np.array(images, dtype=np.float32), np.array(masks, dtype=np.float32)<br>    <br>    def apply_augmentation(self, img, mask):<br>        &quot;&quot;&quot;Apply random augmentations&quot;&quot;&quot;<br>        # Random horizontal flip<br>        if np.random.rand() &gt; 0.5:<br>            img = np.fliplr(img)<br>            mask = np.fliplr(mask)<br>        <br>        # Random vertical flip  <br>        if np.random.rand() &gt; 0.5:<br>            img = np.flipud(img)<br>            mask = np.flipud(mask)<br>            <br>        # Random rotation<br>        if np.random.rand() &gt; 0.7:<br><br>            # 90, 180, 270 degrees<br>            k = np.random.randint(1, 4)  <br>            img = np.rot90(img, k)<br>            mask = np.rot90(mask, k)<br>        <br>        return img, mask</pre><h4><strong>Splitting the Data and Creating the Generator</strong></h4><pre>train_img_paths, val_img_paths, train_mask_paths, val_mask_paths = train_test_split(<br>    all_images, all_masks, test_size=0.2, random_state=42<br>)<br><br>train_gen = FloodNetDataGenerator(<br>    image_paths=train_img_paths,<br>    mask_paths=train_mask_paths,<br>    batch_size=4,<br>    img_size=(256, 256),<br>    augment=True<br>)<br><br>val_gen = FloodNetDataGenerator(<br>    image_paths=val_img_paths,<br>    mask_paths=val_mask_paths,<br>    batch_size=4,<br>    img_size=(256, 256),<br>    augment=False<br>)<br><br>print(f&quot;Training samples: {len(train_img_paths)}&quot;)<br>print(f&quot;Validation samples: {len(val_img_paths)}&quot;)<br>print(f&quot;Training batches: {len(train_gen)}&quot;)<br>print(f&quot;Validation batches: {len(val_gen)}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*jBzYkdkE6bItBy2Z-eeZsA.png" /></figure><p>Let us define a function visualize_grayscale_masks which samples and displays segmentation masks alongside their corresponding original images.</p><pre>def visualize_grayscale_masks(generator, n_samples=3):<br>    # Get a random batch from the generator<br>    random_batch_idx = random.randint(0, len(generator)-1)<br>    images, masks = generator[random_batch_idx]<br>    <br>    # Randomly select n_samples from the batch<br>    if len(images) &gt; n_samples:<br>        indices = random.sample(range(len(images)), n_samples)<br>        images = [images[i] for i in indices]<br>        masks = [masks[i] for i in indices]<br>    else:<br>        images = images[:n_samples]<br>        masks = masks[:n_samples]<br>    <br>    # Create colormap<br>    colors = [np.array(v) / 255 for v in CLASS_COLORS.values()]<br>    cmap = ListedColormap(colors)<br>    <br>    plt.figure(figsize=(20, 6 * n_samples))<br>    for i in range(len(images)):  <br>        # Original image<br>        plt.subplot(len(images), 3, i*3 + 1)<br>        plt.imshow(images[i])<br>        plt.title(f&quot;Random Sample {i+1} (Batch {random_batch_idx})&quot;)<br>        plt.axis(&#39;off&#39;)<br>        <br>        # Class mask<br>        plt.subplot(len(images), 3, i*3 + 2)<br>        mask_class = np.argmax(masks[i], axis=-1) if masks[i].ndim == 3 else masks[i]<br>        <br>        # Print actual values in mask<br>        print(f&quot;Sample {i+1} mask values:&quot;, np.unique(mask_class))<br>        <br>        plt.imshow(mask_class, cmap=cmap, vmin=0, vmax=NUM_CLASSES-1)<br>        plt.title(&quot;Class Mask&quot;)<br>        plt.axis(&#39;off&#39;)<br>        <br>        # Overlay<br>        plt.subplot(len(images), 3, i*3 + 3)<br>        plt.imshow(images[i])<br>        plt.imshow(mask_class, cmap=cmap, alpha=0.5, vmin=0, vmax=NUM_CLASSES-1)<br>        plt.title(&quot;Overlay&quot;)<br>        plt.axis(&#39;off&#39;)<br>    <br>    # Create legend<br>    patches = []<br>    for class_idx, class_name in GRAYSCALE_CLASS_MAP.items():<br>        color = np.array(CLASS_COLORS[class_idx]) / 255<br>        patches.append(plt.plot([], [], marker=&#39;o&#39;, ms=10, ls=&quot;&quot;, <br>                              mec=None, color=color, <br>                              label=f&quot;{class_idx}: {class_name}&quot;)[0])<br>    <br>    plt.legend(handles=patches, bbox_to_anchor=(1.05, 1), <br>               loc=&#39;upper left&#39;, borderaxespad=0., ncol=1)<br>    plt.tight_layout()<br>    plt.show()<br><br>visualize_grayscale_masks(train_gen, n_samples=4)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*rvshFpONt_t5qd_CPEwikw.png" /></figure><h4>Building the model</h4><p>We will implement a U-Net style architecture with a pretrained EfficientNetB4 backbone as its encoder, combined with a custom decoder path for semantic segmentation tasks. The use of the EfficientNetB4 will also enable us to leverage transfer learning by using EfficientNetB4.</p><p>Firstly, we take the complete EfficientNetB4 architecture (excluding its original classification head) and use it to process the input image through its deep convolutional layers. We strategically extract intermediate feature maps at four different scales from the EfficientNet backbone to serve as skip connections. These are taken from specific activation layers corresponding to different resolution levels (128x128, 64x64, 32x32, and 16x16 feature maps).</p><p>The decoder path will then reconstruct the segmentation mask through a series of upsampling blocks. Each block will first increase the spatial resolution using bilinear interpolation, then concatenate these upsampled features with the corresponding skip connection from the encoder path. Following each concatenation, two convolutional layers with ReLU activation refine the combined features.</p><p>After that, we have the final upsampling layer that scales the 128x128 feature maps up to the original 256x256 input resolution. Then the model will conclude with a 1x1 convolutional layer that produces the final segmentation mask, using softmax activation to generate class probabilities for each pixel.</p><pre>def efficientnetb4_unet(input_shape=(256, 256, 3), num_classes=10):<br>   <br>    base_model = EfficientNetB4(input_shape=input_shape, include_top=False, weights=&#39;imagenet&#39;)<br><br>    # Skip connection layers from the B4 architecture<br>    skip_connection_names = [<br>        &quot;block2a_expand_activation&quot;, # 128x128<br>        &quot;block3a_expand_activation&quot;, # 64x64<br>        &quot;block4a_expand_activation&quot;, # 32x32<br>        &quot;block6a_expand_activation&quot;, # 16x16<br>    ]<br>    encoder_outputs = [base_model.get_layer(name).output for name in skip_connection_names]<br>    bottleneck = base_model.get_layer(&quot;top_activation&quot;).output # 8x8<br><br>    # Decoder Path<br>    x = UpSampling2D(size=(2, 2), interpolation=&quot;bilinear&quot;)(bottleneck)<br>    x = Concatenate()([x, encoder_outputs[3]])<br>    x = layers.Conv2D(160, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br>    x = layers.Conv2D(160, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br><br>    x = UpSampling2D(size=(2, 2), interpolation=&quot;bilinear&quot;)(x)<br>    x = Concatenate()([x, encoder_outputs[2]])<br>    x = layers.Conv2D(56, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br>    x = layers.Conv2D(56, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br><br>    x = UpSampling2D(size=(2, 2), interpolation=&quot;bilinear&quot;)(x)<br>    x = Concatenate()([x, encoder_outputs[1]])<br>    x = layers.Conv2D(32, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br>    x = layers.Conv2D(32, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br><br>    x = UpSampling2D(size=(2, 2), interpolation=&quot;bilinear&quot;)(x)<br>    x = Concatenate()([x, encoder_outputs[0]])<br>    x = layers.Conv2D(24, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br>    x = layers.Conv2D(24, 3, activation=&#39;relu&#39;, padding=&#39;same&#39;)(x)<br><br>    #final upsampling layer<br>    x = UpSampling2D(size=(2, 2), interpolation=&quot;bilinear&quot;)(x)<br><br>    # Final output layer<br>    outputs = layers.Conv2D(num_classes, 1, activation=&#39;softmax&#39;, dtype=&#39;float32&#39;)(x)<br><br>    return models.Model(inputs=base_model.input, outputs=outputs)<br><br><br># create the model<br>model = efficientnetb4_unet(input_shape=(256, 256, 3), num_classes=NUM_CLASSES)</pre><h4>Training the model</h4><pre>model.compile(<br>    optimizer=Adam(learning_rate=1e-4),<br>    loss=combined_loss,<br>    metrics=[&#39;accuracy&#39;, dice_coefficient, iou_metric]<br>)<br><br>model.summary()<br><br>callbacks = [<br>    ModelCheckpoint(&#39;best_efficientnetb4_unet_model.h5&#39;, monitor=&#39;val_dice_coefficient&#39;, save_best_only=True, mode=&#39;max&#39;, verbose=1),<br>    ReduceLROnPlateau(monitor=&#39;val_loss&#39;, factor=0.3, patience=7, min_lr=1e-7, verbose=1)<br>]<br><br># Training<br>history = model.fit(<br>    train_gen,<br>    validation_data=val_gen,<br>    epochs=70,<br>    callbacks=callbacks,<br>    verbose=1<br>)</pre><h4>Model Evaluation</h4><pre>val_loss, val_acc, val_dice, val_iou = model.evaluate(val_gen, verbose=1)<br>print(f&quot;Validation Loss: {val_loss:.4f}&quot;)<br>print(f&quot;Validation Accuracy: {val_acc:.4f}&quot;)<br>print(f&quot;Validation Dice Coefficient: {val_dice:.4f}&quot;)<br>print(f&quot;Validation IoU: {val_iou:.4f}&quot;)</pre><p>After training for 60 epochs, the model achieves a reasonable pixel accuracy of 77.69%. However, the model may struggle with segmentation quality due to relatively low dice coefficient and IoU.</p><pre>plt.figure(figsize=(18, 6))<br>plt.subplot(1, 3, 1)<br>plt.plot(history.history[&#39;accuracy&#39;], label=&#39;Train Accuracy&#39;)<br>plt.plot(history.history[&#39;val_accuracy&#39;], label=&#39;Val Accuracy&#39;)<br>plt.title(&#39;Model Accuracy&#39;)<br>plt.xlabel(&#39;Epoch&#39;); plt.ylabel(&#39;Accuracy&#39;); plt.legend(); plt.grid(True)<br>plt.subplot(1, 3, 2)<br>plt.plot(history.history[&#39;loss&#39;], label=&#39;Train Loss&#39;)<br>plt.plot(history.history[&#39;val_loss&#39;], label=&#39;Val Loss&#39;)<br>plt.title(&#39;Model Loss&#39;)<br>plt.xlabel(&#39;Epoch&#39;); plt.ylabel(&#39;Loss&#39;); plt.legend(); plt.grid(True)<br>plt.subplot(1, 3, 3)<br>plt.plot(history.history[&#39;dice_coefficient&#39;], label=&#39;Train Dice&#39;)<br>plt.plot(history.history[&#39;val_dice_coefficient&#39;], label=&#39;Val Dice&#39;)<br>plt.title(&#39;Dice Coefficient&#39;)<br>plt.xlabel(&#39;Epoch&#39;); plt.ylabel(&#39;Dice Score&#39;); plt.legend(); plt.grid(True)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*seHmQLriD0bvYlsOJBkkfQ.png" /></figure><p>Now let us define a function to visualize the predictions. The function will take a trained model and validation generator, then display a side-by-side comparison of original images, their true segmentation masks, and the model’s predicted masks for a specified number of samples, using a custom color map to visually distinguish between classes.</p><pre>def visualize_predictions(model, val_gen, num_samples=3):<br>    images, true_masks = val_gen[0]<br>    pred_masks = model.predict(images)<br>    <br>    colors = [np.array(v)/255.0 for v in CLASS_COLORS.values()]<br>    cmap = ListedColormap(colors)<br>    <br>    for i in range(min(num_samples, len(images))):<br>        plt.figure(figsize=(15, 5))<br>        <br>        # Original image<br>        plt.subplot(1, 3, 1)<br>        plt.imshow(images[i])<br>        plt.title(&#39;Original Image&#39;)<br>        plt.axis(&#39;off&#39;)<br>        <br>        # True mask<br>        plt.subplot(1, 3, 2)<br>        true_mask_vis = np.argmax(true_masks[i], axis=-1)<br>        plt.imshow(true_mask_vis, cmap=cmap, vmin=0, vmax=NUM_CLASSES-1)<br>        plt.title(&#39;True Mask&#39;)<br>        plt.axis(&#39;off&#39;)<br>        <br>        # Predicted mask<br>        plt.subplot(1, 3, 3)<br>        pred_mask_vis = np.argmax(pred_masks[i], axis=-1)<br>        plt.imshow(pred_mask_vis, cmap=cmap, vmin=0, vmax=NUM_CLASSES-1)<br>        plt.title(&#39;Predicted Mask&#39;)<br>        plt.axis(&#39;off&#39;)<br>        <br>        plt.tight_layout()<br>        plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6hIMpll7ws7ha3I9rLw3bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkt3vgG0r64qGzOJhhCVKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-8VlMNlk-CMFaJBDxKBl6w.png" /></figure><h4>Final Thoughts</h4><p>This work demonstrates the effectiveness of a U-Net with EfficientNetB4 backbone for segmenting flood-affected areas in aerial imagery, achieving promising accuracy while revealing opportunities for improvement through better handling of class imbalance and refined metrics. By addressing these limitations, the model could become even more reliable for real-world emergency response scenarios.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85ac7d7c7d3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Object Tracking in Video: Using YOLO and DeepSORT for Object Tracking]]></title>
            <link>https://medium.com/@tony_chisenga/multi-object-tracking-in-video-using-yolo-and-deepsort-for-object-tracking-016c2f84267d?source=rss-4f29fe44b012------2</link>
            <guid isPermaLink="false">https://medium.com/p/016c2f84267d</guid>
            <category><![CDATA[yolo]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tony Chisenga]]></dc:creator>
            <pubDate>Tue, 12 Aug 2025 09:55:53 GMT</pubDate>
            <atom:updated>2025-08-12T09:55:53.501Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyq5c-WiNjPk9vSyyOCS-w.png" /></figure><p>This project will develop a multi-object tracking system using YOLO for detection and DeepSORT for tracking on the Polish 12k road traffic dataset.</p><h4>Training the Model</h4><p>We train a YOLOv11 model using a publicly available dataset from <a href="https://www.kaggle.com/datasets/mikoajkoek/traffic-road-object-detection-polish-12k/data">Kaggle</a>. This dataset provides annotated images of Polish roads, created specifically for object detection tasks. It features a total of eleven object classes, capturing a wide variety of scenarios like different road types and day and night lighting conditions.</p><pre>!pip install ultralytics --quiet</pre><pre>import os<br>import cv2<br>import yaml<br>import random<br>import matplotlib.pyplot as plt<br>from google.colab.patches import cv2_imshow</pre><pre>data_dir = &#39;/kaggle/input/traffic-road-object-detection-polish-12k/road_detection/road_detection&#39;</pre><p>We will define a function to print the structure of the dataset directory.</p><pre>def print_directory_structure(data_dir):<br>    &quot;&quot;&quot;<br>    Prints the directory structure<br>    &quot;&quot;&quot;<br>    for root, dirs, files in os.walk(data_dir):<br>        level = root.replace(data_dir, &#39;&#39;).count(os.sep)<br>        indent = &#39;    &#39; * level<br>        print(f&quot;{indent}{os.path.basename(root)}/&quot;)<br><br>print_directory_structure(data_dir)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*kHR5T7u6XGDJo3PEGyTmBQ.png" /></figure><p>Now, we will create the data.yaml file so that we can specify the locations of our training and validation datasets and define the class names so that the ultralytics YOLO training script can correctly access and interpret the data.</p><pre>yaml_file_path = os.path.join(&#39;/kaggle/working&#39;, &#39;data.yaml&#39;)<br><br># Define the class names(the order is crucial)<br>class_names = [<br>    &#39;car&#39;,<br>    &#39;different-traffic-sign&#39;,<br>    &#39;green-traffic-light&#39;,<br>    &#39;motorcycle&#39;,<br>    &#39;pedestrian&#39;,<br>    &#39;pedestrian-crossing&#39;,<br>    &#39;prohibition-sign&#39;,<br>    &#39;red-traffic-light&#39;,<br>    &#39;speed-limit-sign&#39;,<br>    &#39;truck&#39;,<br>    &#39;warning-sign&#39;<br>]<br><br>yaml_data = {<br>    &#39;path&#39;: data_dir, #points to the root dataset<br>    &#39;train&#39;: &#39;train/images&#39;,<br>    &#39;val&#39;: &#39;valid/images&#39;,<br>    &#39;test&#39;: &#39;test/images&#39;,<br>    &#39;nc&#39;: len(class_names),<br>    &#39;names&#39;: class_names<br>}<br><br>if not os.path.exists(yaml_file_path):<br><br>    # Write to the data.yaml <br>    with open(yaml_file_path, &#39;w&#39;) as f:<br>        yaml.dump(yaml_data, f, sort_keys=False)<br>    print(f&quot;data.yaml file created successfully at: {yaml_file_path}&quot;)<br>else:<br>    print(f&quot;data.yaml file already exists at: {yaml_file_path}&quot;)</pre><p><strong>Let us have a look at the data</strong></p><pre>splits = [&#39;train&#39;, &#39;valid&#39;,&#39;test&#39;]<br><br>for split in splits:<br>    images_path = os.path.join(data_dir, split, &#39;images&#39;)<br>    labels_path = os.path.join(data_dir, split, &#39;labels&#39;)<br><br>    num_images = len(os.listdir(images_path))<br>    num_labels = len(os.listdir(labels_path))<br><br>    print(f&quot;{split.capitalize()} dataset:&quot;)<br>    print(f&quot;  Images: {num_images}&quot;)<br>    print(f&quot;  Labels: {num_labels}&quot;)<br>    print(f&quot;  Ratio (images/labels): {num_images/num_labels:.2f}&quot; if num_labels &gt; 0 else &quot;  No labels found&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*scIUcSdnQHuTL7Wu0q9Fkg.png" /></figure><p>We define a function that will help us to verify that the annotations align correctly with the images. The plot_images_with_boxes reads labeled images and draws bounding boxes around detected objects. Here is how it works:</p><p><strong>Loading and preparing the images: </strong>The function starts by selecting random images from a specified dataset split. Each chosen image is loaded using OpenCV, converted to RGB for correct color display in Matplotlib, and then its height (h) and width (w) are retrieved in pixels, which are essential for translating normalized bounding box coordinates into actual pixel positions.</p><p><strong>Parsing the annotation files: </strong>Each image has a corresponding .txt file (in YOLO format) containing object annotations. Every line in this file describes one detected object with five values(<em>&lt;class_id&gt; &lt;x_center&gt; &lt;y_center&gt; &lt;width&gt; &lt;height&gt;). </em>These coordinates are normalized, meaning they range between 0 and 1, relative to the image dimensions.</p><p><strong>Converting normalized coordinates to pixel values: </strong>To draw the bounding boxes, the function first converts these normalized values into pixel coordinates:</p><ul><li>Center Points (x_center, y_center) → Multiplied by image width (w) and height (h) to get the actual center position in pixels.</li><li>Box Dimensions (width, height) → Also scaled to pixel values by multiplying by w and h.</li></ul><p><strong>Calculating bounding box corners: </strong>YOLO stores box coordinates as center points + dimensions, but OpenCV’s rectangle() function requires top-left and bottom-right corners. The conversion is done with simple arithmetic:</p><ul><li>Top-left corner (x1, y1) = (x_center - width/2, y_center - height/2)</li><li>Bottom-right corner (x2, y2) = (x_center + width/2, y_center + height/2)</li></ul><p>This ensures the box is centered correctly around the object.</p><p>Finally, bounding boxes are drawn around detected objects and corresponding class labels are added.</p><pre>def plot_images_with_boxes(base_path, split, num_samples=3):<br>    images_path = os.path.join(base_path, split, &#39;images&#39;)<br>    labels_path = os.path.join(base_path, split, &#39;labels&#39;)<br><br>    # Get list of image files<br>    image_files = [f for f in os.listdir(images_path) if f.endswith(&#39;.jpg&#39;)]<br><br>    # Randomly select samples<br>    selected_images = random.sample(image_files, min(num_samples, len(image_files)))<br><br>    for img_file in selected_images:<br>        # Read image<br>        img_path = os.path.join(images_path, img_file)<br>        image = cv2.imread(img_path)<br>        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)<br>        h, w = image.shape[:2]<br><br>        # Read corresponding label file<br>        label_path = os.path.join(labels_path, img_file.replace(&#39;.jpg&#39;, &#39;.txt&#39;))<br><br>        if os.path.exists(label_path):<br>            with open(label_path, &#39;r&#39;) as f:<br>                lines = f.readlines()<br><br>            # Draw bounding boxes<br>            for line in lines:<br>                parts = line.strip().split()<br>                if len(parts) == 5:<br>                    class_id, x_center, y_center, width, height = map(float, parts)<br><br>                    # Convert from normalized coordinates to pixel coordinates<br>                    x_center *= w<br>                    y_center *= h<br>                    width *= w<br>                    height *= h<br><br>                    # Calculate box coordinates<br>                    x1 = int(x_center - width/2)<br>                    y1 = int(y_center - height/2)<br>                    x2 = int(x_center + width/2)<br>                    y2 = int(y_center + height/2)<br><br>                    # Draw green rectangle<br>                    color = (0, 255, 0)  <br>                    thickness = 2<br>                    cv2.rectangle(image, (x1, y1), (x2, y2), color, thickness)<br><br>                    # Add class label<br>                    class_name = class_names[int(class_id)] if int(class_id) &lt; len(class_names) else str(class_id)<br>                    cv2.putText(image, class_name, (x1, y1-10),<br>                                cv2.FONT_HERSHEY_SIMPLEX, 0.7, color, thickness)<br><br>        # Display image<br>        plt.figure(figsize=(10, 8))<br>        plt.imshow(image)<br>        plt.title(f&quot;{split.capitalize()} - {img_file}&quot;)<br>        plt.axis(&#39;off&#39;)<br>        plt.show()<br><br>#display samples from each split<br>for split in [&#39;train&#39;, &#39;valid&#39;,&#39;test&#39;]:<br>    plot_images_with_boxes(data_dir, split, num_samples=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*wzCnzHcFQ21Ld2YNT8jt9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCkHHt0aiaeJv1TSH-w4mg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*xWE2COWLeh6sUHUg5pzqTw.png" /></figure><p>We now train the model for 50 epochs.</p><pre>!yolo task=detect mode=train model=yolo11m.pt data=&quot;{yaml_file_path}&quot; epochs=50 imgsz=640</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*acdRQutZ-HAR7UADZwtDWw.png" /></figure><p>When training is done, model weights are saved in runs/detect/train/weights/. The weights include the best.pt, which contains the model state from the epoch with the highest validation performance, and the last.pt, which contains the model state from the final epoch of training. We will use best.pt so that we can load the checkpoint that achieved the best results during training.</p><h4>Object Tracking</h4><p>This part of the project will be organized as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*_2FNB54zHF8vED21StqVGQ.png" /></figure><p><strong>models: </strong>This folder contains best.pt, which is the model state from the epoch with the highest validation performance, downloaded from our trained YOLO model.</p><p><strong>utils: </strong>One of the files in this folder is __init__.py, which ensures that the defined classes and other components are accessible outside the utils folder. Below is the code in __init__.py:</p><pre>from .object_tracker import ObjectTracker</pre><p>The other file is the object_tracker.py. This is where the ObjectTracker class is defined. The purpose is to detect vehicles in a video feed, track their movement, and estimate their speeds in real time.</p><p>First, we define PIXELS_PER_METER, a calibration value that converts pixel distances into real-world meters. Next, we set speed thresholds to distinguish between very slow and normal speeds. The bounding box color for each vehicle is then determined by the speed category it falls into.</p><p><strong>The ObjectTracker Class:</strong> This class encapsulates the main functionality. Upon initialization, it loads the YOLO model, creates a DeepSort tracker with parameters that control how long tracks are retained and how many frames are required before a track is confirmed, and sets up an empty track_history dictionary to record each tracked object’s previous position — a crucial step for calculating movement and speed.</p><p><strong>Processing a Frame: </strong>The process_frame method takes in a video frame and the frame rate (fps) and returns a copy of the frame with bounding boxes, colors, and labels applied.</p><p><strong>Detecting Vehicles:</strong> Although the model was originally trained to recognize 11 classes, our focus is on fast-moving objects. Therefore, we filter the detections to only keep our target classes: &#39;car&#39;, &#39;truck&#39;, and &#39;motorcycle&#39;. When the model returns a set of detections with coordinates, class IDs, and confidence scores, only those matching the target classes are retained. Each retained detection is then converted into a format that DeepSort understands — namely, bounding box coordinates, confidence, and class ID.</p><p><strong>Tracking Across Frames:</strong> DeepSort’s update_tracks method is called with the detections, assigning each one to an existing track when possible or starting a new track if necessary. This ensures that the same vehicle retains the same track_id across frames. The time_increment, representing the time between frames, is then calculated using the provided fps.</p><p><strong>Calculating Speed and Categorizing Vehicles:</strong> For each confirmed track, the current bounding box center is determined. If the object was also detected in the previous frame, the pixel distance it has moved is calculated and converted into meters using PIXELS_PER_METER. The speed in meters per second is then obtained by dividing the real-world distance by time_increment and subsequently converted to km/h. Based on the defined thresholds, the vehicle is categorized as very slow (yellow box), normal (green box), or over-speeding (red box). If the object has no previous position (i.e., it is appearing for the first time), it is assigned a default white box with an “N/A” speed. Finally, the object’s current center is stored in track_history for use in the next frame’s calculation.</p><pre>import cv2<br>import numpy as np<br>from ultralytics import YOLO<br>from deep_sort_realtime.deepsort_tracker import DeepSort<br><br># configurable parameters<br>PIXELS_PER_METER = 30  <br>VERY_SLOW_THRESHOLD = 10 <br>NORMAL_SPEED_THRESHOLD = 50<br><br>#colors for each speed category(BGR)<br>COLOR_VERY_SLOW = (0, 255, 255)<br>COLOR_NORMAL = (0, 255, 0)   <br>COLOR_OVER_SPEEDING = (0, 0, 255)<br>COLOR_DEFAULT = (255, 255, 255)<br><br><br>class ObjectTracker:<br>    def __init__(self, model_path=&#39;models/best.pt&#39;):<br>        self.model = YOLO(model_path)<br>        self.tracker = DeepSort(max_age=30, n_init=3)<br>        self.track_history = {}<br><br>    def process_frame(self, frame, fps):<br>        target_classes = [&#39;car&#39;, &#39;truck&#39;, &#39;motorcycle&#39;]<br>        results = self.model(frame, stream=True, conf=0.3)<br>        <br>        detections = []<br>        for result in results:<br>            boxes = result.boxes<br>            for box in boxes:<br>                class_id = int(box.cls[0])<br>                class_name = self.model.names.get(class_id)<br>                if class_name not in target_classes:<br>                    continue<br>                x1, y1, x2, y2 = box.xyxy[0]<br>                conf = box.conf[0]<br>                w, h = x2 - x1, y2 - y1<br>                detections.append(([int(x1), int(y1), int(w), int(h)], float(conf), class_id))<br><br>        tracks = self.tracker.update_tracks(detections, frame=frame)<br>        time_increment = 1.0 / fps<br>        frame_h, frame_w = frame.shape[:2]<br><br>        for track in tracks:<br>            if not track.is_confirmed():<br>                continue<br>            track_id = track.track_id<br>            ltrb = track.to_ltrb()<br>            class_id = track.get_det_class()<br>            class_name = self.model.names.get(class_id, &#39;Unknown&#39;)<br>            x1, y1, x2, y2 = int(ltrb[0]), int(ltrb[1]), int(ltrb[2]), int(ltrb[3])<br>            <br>            # speed categorization<br>            <br>            current_center = (int((x1 + x2) / 2), int((y1 + y2) / 2))<br>            speed_kmh = 0<br>            <br>            # Default color and label<br>            box_color = COLOR_DEFAULT<br>            speed_category = &quot;N/A&quot;<br><br>            if track_id in self.track_history:<br>                prev_center = self.track_history[track_id]<br>                pixel_distance = np.linalg.norm(np.array(current_center) - np.array(prev_center))<br>                real_world_distance = pixel_distance / PIXELS_PER_METER<br>                speed_mps = real_world_distance / time_increment<br>                speed_kmh = speed_mps * 3.6<br>              <br>                if speed_kmh &lt; VERY_SLOW_THRESHOLD:<br>                    speed_category = &quot;Very Slow&quot;<br>                    box_color = COLOR_VERY_SLOW<br>                elif speed_kmh &lt; NORMAL_SPEED_THRESHOLD:<br>                    speed_category = &quot;Normal&quot;<br>                    box_color = COLOR_NORMAL<br>                else:<br>                    speed_category = &quot;Over-speeding&quot;<br>                    box_color = COLOR_OVER_SPEEDING<br>            <br>            self.track_history[track_id] = current_center<br>          <br>            # apply padding on bboxes<br>            padding_percent = 0.05<br>            w, h = x2 - x1, y2 - y1<br>            padding_w, padding_h = int(w * padding_percent), int(h * padding_percent)<br>            new_x1, new_y1 = max(0, x1 - padding_w), max(0, y1 - padding_h)<br>            new_x2, new_y2 = min(frame_w, x2 + padding_w), min(frame_h, y2 + padding_h)<br><br>            #final label with ID, speed, and category<br>            label = f&quot;ID:{track_id} {int(speed_kmh)}km/h - {speed_category}&quot;<br><br>            # Draw bbox and label<br>            cv2.rectangle(frame, (new_x1, new_y1), (new_x2, new_y2), box_color, 2)<br>            cv2.putText(frame, label, (new_x1, new_y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.7, box_color, 2)<br>            <br>        return frame</pre><p><strong>input_videos:</strong> This folder contains the short video clips used as inputs. The processed outputs are saved in the runs/track folder.</p><p><strong>inference.py: </strong>This script serves as the entry point for running the complete vehicle detection, tracking, and speed estimation pipeline. It handles reading an input video file, processing it frame by frame through the tracker, writing the annotated frames to a new output video file, and managing output directories to keep results organized.</p><p>We define a function called get_next_run_dir that creates and returns a new directory for saving output runs by checking for existing folders named &#39;exp&#39;, &#39;exp2&#39;, &#39;exp3&#39;, and so forth. It increments the number until it finds a folder name that doesn’t already exist, then creates and returns that directory to prevent overwriting previous results. Next, the object tracker is initialized with our trained model, and paths for the input video and output directory are set up. The video’s properties, such as frame size and FPS, are retrieved, and a video writer is prepared to save the processed frames as the video is processed frame by frame through the tracker, with annotated frames written to the output.</p><pre>import cv2<br>from pathlib import Path<br>from tqdm import tqdm <br>from utils import ObjectTracker<br><br>def get_next_run_dir(base_dir=&#39;runs/track&#39;):<br>    base_path = Path(base_dir)<br>    base_path.mkdir(parents=True, exist_ok=True) <br>    n = 1<br>    while True:<br>        run_name = f&quot;exp{n if n &gt; 1 else &#39;&#39;}&quot;<br>        run_path = base_path / run_name<br>        if not run_path.exists():<br>            run_path.mkdir()<br>            return run_path<br>        n += 1<br><br># initialization<br>tracker = ObjectTracker(model_path=&#39;models/best.pt&#39;)<br>video_path = &#39;input_videos/video_1.mp4&#39;<br>output_dir = get_next_run_dir()<br>video_filename = Path(video_path).name<br>output_path = output_dir / video_filename<br><br># setup video capture and writer<br>cap = cv2.VideoCapture(video_path)<br>frame_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))<br>frame_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))<br>fps = int(cap.get(cv2.CAP_PROP_FPS))<br>fourcc = cv2.VideoWriter_fourcc(*&#39;mp4v&#39;)<br>out = cv2.VideoWriter(str(output_path), fourcc, fps, (frame_width, frame_height))<br><br>total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))<br>progress_bar = tqdm(total=total_frames, desc=f&quot;Processing {video_filename}&quot;)<br><br>#progress loop<br>while cap.isOpened():<br>    success, frame = cap.read()<br>    if not success:<br>        break<br><br>    processed_frame = tracker.process_frame(frame, fps)<br>    out.write(processed_frame)<br><br>    #update the progress bar by one step<br>    progress_bar.update(1)<br><br><br><br>#cleanup<br>progress_bar.close()<br>cap.release()<br>out.release()<br>cv2.destroyAllWindows()<br>print(f&quot;Processing complete. Video saved to {output_path}&quot;)</pre><p><strong>Conclusion</strong></p><p>This project successfully integrates vehicle detection, tracking, and speed estimation into a video processing pipeline. The system processes input videos to generate annotated outputs that clearly visualize tracked vehicles along with their speed categories.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=016c2f84267d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Facial Emotion Recognition with a Convolutional Neural Network(CNN)]]></title>
            <link>https://medium.com/@tony_chisenga/facial-emotion-recognition-with-a-convolutional-neural-network-cnn-d5d2004053d8?source=rss-4f29fe44b012------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5d2004053d8</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[facial-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Tony Chisenga]]></dc:creator>
            <pubDate>Thu, 31 Jul 2025 17:38:58 GMT</pubDate>
            <atom:updated>2025-07-31T17:38:58.295Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*vgk1jEsKSe0qnuDj6RpGkw.png" /></figure><p><strong>Description:</strong> This is a classification problem that involves training a Convolutional Neural Network (CNN) to recognize facial emotions across seven categories: anger, fear, disgust, happiness, sadness, surprise, and neutral. The model is trained on the FER-2013 dataset, which contains 48×48 pixel grayscale images of faces.</p><pre>import os<br>import cv2<br>import torch<br>import torch.nn as nn<br>import numpy as np<br>from glob import glob<br>from PIL import Image<br>import matplotlib.pyplot as plt<br>from collections import defaultdict<br>from torchvision import transforms<br>from torch.utils.data import Dataset, DataLoader<br>from sklearn.preprocessing import LabelEncoder<br><br>base_dir = &#39;/kaggle/input/fer2013&#39;</pre><p>We define a function to to display the folder structure of the dataset directory in a hierarchical format. The function recursively scans through all subdirectories, skipping hidden folders, and prints each level with increasing indentation to visually represent the nesting depth.</p><pre>def show_directory_structure(root_dir):<br>    if not os.path.exists(root_dir):<br>        print(f&quot;Error: Directory &#39;{root_dir}&#39; does not exist.&quot;)<br>        return<br><br>    print(f&quot;Directory structure of: {root_dir}\n{&#39;-&#39;*40}&quot;)<br>    <br>    for root, dirs, files in os.walk(root_dir):<br><br>        # Skip hidden directories<br>        dirs[:] = [d for d in dirs if not d.startswith(&#39;.&#39;)]<br>        <br>        level = root.replace(root_dir, &#39;&#39;).count(os.sep)<br>        indent = &#39; &#39; * 4 * level<br>        print(f&quot;{indent}{os.path.basename(root)}/&quot;)<br>        <br>        sub_indent = &#39; &#39; * 4 * (level + 1)<br>        for d in dirs:<br>            print(f&quot;{sub_indent}{d}/&quot;)<br><br>show_directory_structure(base_dir)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*sZ8pjoeXPG20GRbSoZf4hw.png" /></figure><h4>Plotting class Distributions</h4><p>We define a function that creates a bar chart to visualize the distribution of emotion classes in a dataset.</p><pre>def plot_distribution(counts, title):<br>    emotions = list(counts.keys())<br>    values = list(counts.values())<br>    <br>    plt.figure(figsize=(10, 5))<br>    bars = plt.bar(emotions, values)<br>    plt.title(title)<br>    plt.xlabel(&#39;Emotion Class&#39;)<br>    plt.ylabel(&#39;Number of Images&#39;)<br>    plt.xticks(rotation=45)<br>    <br>    # Add counts on top of bars<br>    for bar in bars:<br>        height = bar.get_height()<br>        plt.text(bar.get_x() + bar.get_width()/2., height,<br>                f&#39;{int(height)}&#39;,<br>                ha=&#39;center&#39;, va=&#39;bottom&#39;)<br>    <br>    plt.tight_layout()<br>    plt.show()<br><br>plot_distribution(train_counts, &#39;Training Set Class Distribution&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*27hulEVgZj-rLDYN1KhPSA.png" /></figure><h4>Displaying Sample Images from the Dataset</h4><p>Let us define a function that creates a visual grid of sample images from the dataset displaying multiple examples for each emotion category. It takes a directory path and shows a configurable number of randomly selected images per emotion (default: 5). For each emotion folder, it randomly selects image files (without replacement) up to the requested sample count.</p><pre>def display_samples(directory, samples_per_class=5):<br>    plt.figure(figsize=(15, 10))<br>    emotions = sorted(os.listdir(directory))<br>    <br>    for i, emotion in enumerate(emotions):<br>        emotion_dir = os.path.join(directory, emotion)<br>        if not os.path.isdir(emotion_dir):<br>            continue<br>            <br>        # Get sample images<br>        image_files = glob(os.path.join(emotion_dir, &#39;*&#39;))<br>        sample_files = np.random.choice(image_files, min(samples_per_class, len(image_files)), replace=False)<br>        <br>        # Display samples<br>        for j, img_path in enumerate(sample_files):<br>            ax = plt.subplot(len(emotions), samples_per_class, i * samples_per_class + j + 1)<br>            img = plt.imread(img_path)<br>            plt.imshow(img, cmap=&#39;gray&#39;)<br>            plt.axis(&#39;off&#39;)<br>            <br>            if j == 0:<br>                plt.title(emotion, fontsize=8, pad=2)<br>    <br>    plt.suptitle(f&#39;Sample Images from {os.path.basename(directory)} Directory&#39;, y=1.02)<br>    plt.tight_layout()<br>    plt.show()<br><br>display_samples(train_dir)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*hLOEbnyfdCFqV3dPidDHRQ.png" /></figure><h4>The Dataset Class</h4><p>We define a custom Dataset class to efficiently load, preprocess, and manage the FER2013 dataset. This organizes the dataset in a way that PyTorch can efficiently use for training models.</p><p><strong>Core Methods of the Dataset Class</strong></p><p>__len__(): This method returns the exact count of images in the dataset.</p><p>__getitem__(idx): When the model requests for the <em>n-th</em> image, this method does the following:</p><ol><li>Fetches the image, reading it in grayscale.</li><li>Adjusts its shape, adding a single “channel” dimension (turning a 2D [height, width] image into [height, width,1] —a format PyTorch expects).</li><li>Transforms it: if the augmentations have been specified, they are applied here. Otherwise, it simply converts the image to a tensor.</li><li>Finally, it returns the image and its label.</li></ol><p><strong>Helper Methods</strong></p><p>show_sample(idx): This method utilizes the __getitem__ to load the image, label pair of a given idx and display it in grayscale.</p><p>get_class_distribution(): This method counts how many images belong to each class. It helps to check if the dataset is balanced.</p><pre>class FER2013Dataset(Dataset):<br>    def __init__(self, root_dir=base_dir, mode=&#39;train&#39;, transform=None):<br>        &quot;&quot;&quot;<br>        Args:<br>            root_dir (str): Path to dataset root directory<br>            mode (str): &#39;train&#39;, &#39;test&#39;, or &#39;val&#39;<br>            transform (callable): Optional transform to be applied<br>        &quot;&quot;&quot;<br>        self.root_dir = root_dir<br>        self.mode = mode<br>        self.transform = transform<br>        <br>        # Get all image paths and labels<br>        self.image_paths = []<br>        self.labels = []<br>        <br>        class_folders = sorted(os.listdir(os.path.join(root_dir, mode)))<br>        self.classes = class_folders<br>        self.label_encoder = LabelEncoder()<br>        self.label_encoder.fit(class_folders)<br>        <br>        for class_name in class_folders:<br>            class_dir = os.path.join(root_dir, mode, class_name)<br>            for img_path in glob(os.path.join(class_dir, &#39;*&#39;)):<br>                self.image_paths.append(img_path)<br>                self.labels.append(class_name)<br>        <br>        # Convert labels to numerical values<br>        self.labels = self.label_encoder.transform(self.labels)<br>        <br>    def __len__(self):<br>        return len(self.image_paths)<br>    <br>    def __getitem__(self, idx):<br>        img_path = self.image_paths[idx]<br>        label = self.labels[idx]<br>        <br>        # Load image as grayscale<br>        image = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)<br>        <br>        # Add channel dimension (H,W,1)<br>        image = np.expand_dims(image, axis=-1)  <br>        <br>        # Convert to PIL Image for transforms<br>        image = Image.fromarray(image.squeeze())<br>        <br>        if self.transform:<br>            image = self.transform(image)<br>        else:<br>            # Default transform if none provided<br>            image = transforms.ToTensor()(image)<br>        <br>        return image, label<br>    <br>    def show_sample(self, idx):<br>        image, label = self.__getitem__(idx)<br>        plt.imshow(image.squeeze(), cmap=&#39;gray&#39;)<br>        plt.title(f&#39;Class: {self.label_encoder.inverse_transform([label])[0]}&#39;)<br>        plt.axis(&#39;off&#39;)<br>        plt.show()<br>        <br>    def get_class_distribution(self):<br>        unique, counts = np.unique(self.labels, return_counts=True)<br>        return dict(zip(self.label_encoder.inverse_transform(unique), counts))</pre><pre># Define transforms<br>train_transform = transforms.Compose([<br>    transforms.Resize(size=(64,64)),<br>    transforms.RandomHorizontalFlip(),<br>    transforms.RandomRotation(10),<br>    transforms.RandomAffine(0, translate=(0.1, 0.1)),<br>    transforms.ToTensor(),<br>    transforms.Normalize(mean=[0.5], std=[0.5]) <br>])<br><br>test_transform = transforms.Compose([<br>    transforms.Resize(size=(64,64)),<br>    transforms.ToTensor(),<br>    transforms.Normalize(mean=[0.5], std=[0.5])<br>])<br><br><br># Create datasets<br>train_dataset = FER2013Dataset(mode=&#39;train&#39;, transform=train_transform)<br>test_dataset = FER2013Dataset(mode=&#39;test&#39;, transform=test_transform)<br><br># Dataloaders<br>train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)<br>test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)<br><br># Visualize a sample<br>train_dataset.show_sample(0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*4XLjomwv7T8tsWA39AVxqQ.png" /></figure><h4>Defining the baseline model(a CNN)</h4><p>Since our transform resizes images to 64x64, the model architecture is designed to process these 64x64 grayscale facial expression images and classify them into 7 emotion categories.</p><h4>1. Feature Extraction Layers</h4><p><strong>First Conv Layer (conv1):</strong></p><ul><li>Input: 1 channel (grayscale image)</li><li>Creates 32 feature maps using 3x3 filters</li><li>Padding=1 maintains spatial dimensions (64x64)</li><li>Math: (64–3+2*1)/1 + 1 = 64 (same size)</li></ul><p><strong>First Pooling:</strong></p><ul><li>MaxPool2d(2,2) reduces dimensions by half (64→32)</li><li>Now we have (batch, 32, 32, 32)</li></ul><p><strong>Second Conv Layer (conv2):</strong></p><ul><li>Takes 32 feature maps as input</li><li>Produces 64 feature maps</li><li>Kernel size remains 3x3 with padding=1</li><li>After pooling: (64, 16, 16)</li></ul><p><strong>Third Conv Layer (conv3):</strong></p><ul><li>Input: 64 feature maps</li><li>Output: 128 feature maps</li><li>Same 3x3 kernel/padding</li><li>Final pooled size: (128, 8, 8)</li></ul><h4>2. Transition to Classification</h4><p><strong>Flattening:</strong></p><ul><li>The (128,8,8) feature maps become one long vector(128 channels × 8 width × 8 height = 8192 numbers)</li><li>Like unraveling all the detected patterns into a list</li></ul><p><strong>First Fully Connected Layer (fc1):</strong></p><ul><li>Takes 8192 inputs</li><li>Outputs 256 values</li><li>Dropout (0.5) randomly ignores half the neurons during training to prevent overfitting</li></ul><p><strong>Final Classification Layer (fc2):</strong></p><ul><li>Takes 256 inputs</li><li>Outputs 7 values (one per emotion class)</li><li>No activation function because we’ll use CrossEntropyLoss which includes SoftMax</li></ul><h4>Data Transformation process summary</h4><p>The initial size of the images is 48x48. A transform resized them to 64x64 pixels. These resized images take the following path:</p><ol><li>Starts as (1,64,64) — one grayscale image</li><li>After conv1+pooling: (32,32,32) — 32 feature maps at half resolution</li><li>After conv2+pooling: (64,16,16) — more features, quarter resolution</li><li>After conv3+pooling: (128,8,8) — rich features at 1/8th resolution</li><li>Flattened to 8192 numbers</li><li>Condensed to 256 meaningful measurements</li><li>Finally evaluated as 7 emotion probabilities</li></ol><pre>class BaselineCNN(nn.Module):<br>    def __init__(self):<br>        super().__init__()<br>        # Feature extraction<br>        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=1)  <br>        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)  <br>        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding=1)<br>        <br>        # Classifier<br>        self.fc1 = nn.Linear(128 * 8 * 8, 256)  <br>        self.fc2 = nn.Linear(256, 7)  <br><br>        # Helpers<br>        self.pool = nn.MaxPool2d(2, 2) <br>        self.dropout = nn.Dropout(0.5)<br><br>    def forward(self, x):<br>        # Feature extraction<br>        x = self.pool(nn.ReLU()(self.conv1(x))) <br>        x = self.pool(nn.ReLU()(self.conv2(x)))  <br>        x = self.pool(nn.ReLU()(self.conv3(x))) <br>        <br>        # Classifier<br>        x = x.view(x.size(0), -1)  # Flatten<br>        x = self.dropout(nn.ReLU()(self.fc1(x)))<br>        x = self.fc2(x) <br>        return x</pre><h3>Model Training</h3><pre>device = &#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;<br>model0 = BaselineCNN()<br><br># You can pick asample from the dataloader to test your model on a single sample<br>image_batch, label_batch = next(iter(train_loader))<br>image_batch.shape, label_batch.shape<br><br>model0(image_batch).to(device)</pre><pre>loss_fn = nn.CrossEntropyLoss()<br>optimizer = torch.optim.Adam(params=model0.parameters(),lr=1e-3)<br><br>from tqdm.auto import tqdm<br><br>def train_step(model: torch.nn.Module,<br>               dataloader: torch.utils.data.DataLoader,<br>               loss_fn: torch.nn.Module,<br>               optimizer: torch.optim.Optimizer,<br>               device: torch.device = device):<br>    model.train()<br>    train_loss, train_acc = 0, 0<br>    <br>    for batch, (X, y) in enumerate(tqdm(dataloader)):<br>        X, y = X.to(device), y.to(device)<br>        <br>        # Forward pass<br>        y_pred = model(X)<br>        loss = loss_fn(y_pred, y)<br>        train_loss += loss.item() <br>        <br>        # Backpropagation<br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br>        <br>        # Calculate accuracy<br>        y_pred_class = torch.argmax(y_pred, dim=1)  <br>        train_acc += (y_pred_class == y).sum().item() / len(y_pred)<br>    <br>    # Average loss and accuracy per batch<br>    train_loss = train_loss / len(dataloader)<br>    train_acc = train_acc / len(dataloader)<br>    return train_loss, train_acc<br><br><br>def test_step(model: torch.nn.Module,<br>               dataloader: torch.utils.data.DataLoader,<br>               loss_fn:torch.nn.Module,<br>               device:torch.device=device):<br>  test_loss, test_acc = 0,0<br>  model.eval()<br><br>  with torch.inference_mode():<br>    for batch, (X,y) in enumerate(dataloader):<br>      X,y = X.to(device), y.to(device)<br>      y_pred = model(X)<br>      loss = loss_fn(y_pred,y)<br>      test_loss += loss.item()<br>      y_pred_class = torch.argmax(torch.softmax(y_pred,dim=1),dim=1)<br>      test_acc += (y_pred_class==y).sum().item()/len(y_pred)<br><br>    test_loss = test_loss / len(dataloader)<br>    test_acc = test_acc / len(dataloader)<br>    return test_loss, test_acc<br><br>def train(<br>    model: torch.nn.Module,<br>    train_dataloader: torch.utils.data.DataLoader,<br>    test_dataloader: torch.utils.data.DataLoader,<br>    optimizer: torch.optim.Optimizer,<br>    loss_fn: torch.nn.Module,<br>    epochs: int,<br>    device: torch.device,<br>    scheduler: torch.optim.lr_scheduler = None <br>):<br>    results = {&#39;train_loss&#39;: [], &#39;train_acc&#39;: [], &#39;test_loss&#39;: [], &#39;test_acc&#39;: []}<br>    <br>    for epoch in range(epochs):<br>        train_loss, train_acc = train_step(model, train_dataloader, loss_fn, optimizer, device)<br>        test_loss, test_acc = test_step(model, test_dataloader, loss_fn, device)<br>        <br>        if scheduler:<br>            scheduler.step(test_acc)  <br>        <br>        print(f&quot;Epoch: {epoch+1} | Train Loss: {train_loss:.4f} | Train Acc: {train_acc:.4f} | Test Acc: {test_acc:.4f} | LR: {optimizer.param_groups[0][&#39;lr&#39;]:.2e}&quot;)<br>        <br>        results[&#39;train_loss&#39;].append(train_loss)<br>        results[&#39;train_acc&#39;].append(train_acc)<br>        results[&#39;test_loss&#39;].append(test_loss)<br>        results[&#39;test_acc&#39;].append(test_acc)<br>    <br>    return results<br><br>def plot_loss_curves(results):<br>  &quot;&quot;&quot;Plot training curves of a results dictionary.&quot;&quot;&quot;<br>  loss = results[&#39;train_loss&#39;]<br>  test_loss = results[&#39;test_loss&#39;]<br><br>  train_acc = results[&#39;train_acc&#39;]<br>  test_acc = results[&#39;test_acc&#39;]<br>  epochs = range(len(results[&#39;train_loss&#39;]))<br><br>  plt.figure(figsize=(15,7))<br><br>  #plot the loss<br>  plt.subplot(1,2,1)<br>  plt.plot(epochs,loss,label=&#39;train_loss&#39;)<br>  plt.plot(epochs,test_loss,label=&#39;test_loss&#39;)<br>  plt.title(&#39;Loss&#39;)<br>  plt.xlabel(&#39;Epochs&#39;)<br>  plt.legend()<br><br>  #plot accuracy<br>  plt.subplot(1,2,2)<br>  plt.plot(epochs,train_acc,label=&#39;train_accuracy&#39;)<br>  plt.plot(epochs,test_acc,label=&#39;test_accuracy&#39;)<br>  plt.title(&#39;Accuracy&#39;)<br>  plt.xlabel(&#39;Epochs&#39;)<br>  plt.legend()<br><br><br>torch.manual_seed(42)<br>torch.cuda.manual_seed(42)<br><br>from timeit import default_timer as timer<br><br>start_time = timer()<br><br>model0_results =  train(model=model0,<br>          train_dataloader= train_loader,<br>          test_dataloader=test_loader,<br>          optimizer=optimizer,<br>          loss_fn=loss_fn,<br>          epochs =3,<br>          device=device)<br><br>end_time = timer()<br>print(f&quot;Total training time: {end_time-start_time:.3f} seconds&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*R1YrY5CQvmqzQPam_jPaYg.png" /></figure><h4>Adding Residual Blocks and Batch Normalization to the <strong>CNN</strong></h4><p>A residual block is a very useful component for training very deep networks. It helps solve the vanishing gradient problem by enabling the model to learn residuals — that is, the differences between the input and the desired output — instead of learning a full transformation directly. In a typical setup, a standard convolutional block applies a series of operations like convolution, batch normalization, and a ReLU activation to the input. What makes a residual block unique is the addition of a skip connection that bypasses these transformations and adds the original input directly to the output of the block. This shortcut connection allows gradients to pass through the network more easily during training.</p><p><strong>ResidualBlock Class : </strong>This class handles two main scenarios. When the input and output channels are equal, the skip connection is a simple identity — the input is passed directly to the output without any change. The residual path in this case consists of two 3×3 convolutional layers, each followed by batch normalization, with a ReLU activation after the first convolution. The final output is obtained by adding this residual path to the original input and applying a ReLU activation.</p><p>When the input and output channels differ, the skip connection uses a 1×1 convolution followed by batch normalization to adjust the dimensions. This ensures that the shapes match and the addition operation remains valid.</p><pre>class ResidualBlock(nn.Module):<br>    def __init__(self, in_channels, out_channels):<br>        super().__init__()<br>        self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)<br>        self.bn1 = nn.BatchNorm2d(out_channels)<br>        self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1)<br>        self.bn2 = nn.BatchNorm2d(out_channels)<br><br>        # Identity if in_channels == out_channels<br>        self.shortcut = nn.Sequential()  <br>        if in_channels != out_channels:<br>            self.shortcut = nn.Sequential(<br>                nn.Conv2d(in_channels, out_channels, kernel_size=1),<br>                nn.BatchNorm2d(out_channels)<br>            )<br><br>    def forward(self, x):<br>        out = nn.ReLU()(self.bn1(self.conv1(x)))<br>        out = self.bn2(self.conv2(out))<br>        out += self.shortcut(x)  #residual connection<br>        return nn.ReLU()(out)</pre><p><strong>EmotionsCNN Class: </strong>This class integrates residual blocks into a complete model. The data flows through the network as follows:</p><ol><li>Starts as (1, 64, 64) – A single-channel (grayscale) input image.</li><li>After conv1 + pooling: Expands to (32, 32, 32) – 32 feature maps at half resolution (due to pooling).</li><li>After res1 + pooling: Increases to (64, 16, 16) – 64 deeper feature maps at quarter resolution.</li><li>After res2 + pooling: Grows to (128, 8, 8) – 128 high-level features at 1/8th resolution.</li><li>After final conv4 + pooling: Expands to (256, 4, 4) – 256 rich semantic features at 1/16th resolution.</li><li>Flattened to 4096 values (256 x 4 x 4).</li><li>Condensed to 512 meaningful features in the first fully connected layer.</li><li>Finally classified into 7 emotion probabilities (angry, disgust, fear, happy, sad, surprise, neutral).</li></ol><pre>class EmotionsCNN(nn.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=1)<br>        self.bn1 = nn.BatchNorm2d(32)<br>        self.res1 = ResidualBlock(32, 64)<br>        self.res2 = ResidualBlock(64, 128)<br>        self.conv4 = nn.Conv2d(128, 256, kernel_size=3, padding=1)<br>        self.bn4 = nn.BatchNorm2d(256)<br>        self.pool = nn.MaxPool2d(2, 2)<br>        self.fc1 = nn.Linear(256 * 4 * 4, 512)<br>        self.bn_fc = nn.BatchNorm1d(512)<br>        self.dropout = nn.Dropout(0.5)<br>        self.fc2 = nn.Linear(512, 7)<br><br>    def forward(self, x):<br>        x = self.pool(nn.ReLU()(self.bn1(self.conv1(x))))<br>        x = self.pool(self.res1(x))<br>        x = self.pool(self.res2(x))<br>        x = self.pool(nn.ReLU()(self.bn4(self.conv4(x))))<br>        x = x.view(x.size(0), -1)<br>        x = self.dropout(nn.ReLU()(self.bn_fc(self.fc1(x))))<br>        return self.fc2(x)<br><br><br>model2 = EmotionsCNN().to(device)<br>optimizer2 = torch.optim.AdamW(model2.parameters(), lr=3e-4, weight_decay=1e-4)<br>scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer2, mode=&#39;max&#39;, patience=2, factor=0.5)<br><br>model2_results = train(<br>    model=model2,<br>    train_dataloader=train_loader,<br>    test_dataloader=test_loader,<br>    optimizer=optimizer2, <br>    loss_fn=loss_fn,<br>    epochs=70,<br>    device=device,<br>    scheduler=scheduler<br>)<br><br><br>plot_loss_curves(model2_results)</pre><p>Below are the plots of the losses and accuracies for 70 epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XD89VTs2Wu_ZmlC2neTAiA.png" /></figure><h4>Final Thoughts</h4><p>We have demonstrated that a CNN enhanced with residual blocks can successfully recognize emotions from facial images. While the model performed well, exploring more advanced architectures or training techniques could further improve accuracy. There’s still room for improvement.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5d2004053d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>