<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yalcinselcuk on Medium]]></title>
        <description><![CDATA[Stories by Yalcinselcuk on Medium]]></description>
        <link>https://medium.com/@yalcinselcuk0?source=rss-21bd9cde6b10------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*SAA6PFSzsZuuLjG9</url>
            <title>Stories by Yalcinselcuk on Medium</title>
            <link>https://medium.com/@yalcinselcuk0?source=rss-21bd9cde6b10------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:24:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yalcinselcuk0/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Convolutional Neural Networks: AlexNet, U-Net, ResNet and DenseNet Architectures]]></title>
            <link>https://medium.com/@yalcinselcuk0/convolutional-neural-networks-alexnet-u-net-resnet-and-densenet-architectures-66d37fe88a88?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/66d37fe88a88</guid>
            <category><![CDATA[resnet]]></category>
            <category><![CDATA[densenet]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[unet]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Sun, 19 Oct 2025 21:14:38 GMT</pubDate>
            <atom:updated>2025-10-19T21:14:38.773Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’ve ever wondered how <strong>machines</strong> can <strong>recognise</strong> <strong>cats</strong> in photos, <strong>detect</strong> <strong>tumours</strong> in scans, or even make <strong>sense</strong> <strong>of</strong> <strong>videos</strong>, the secret often lies in <strong>convolutional neural networks (CNNs)</strong>. These architectures are the workhorses of computer vision, and over the years, they’ve gone through an exciting evolution.</p><p>It all started with <strong>AlexNet</strong> in 2012, which shocked the world by dominating the <strong>ImageNet</strong> competition and <strong>demonstrating</strong> the true <strong>power</strong> <strong>of</strong> <strong>deep</strong> <strong>learning</strong>. Then came <strong>VGGNet</strong>, proving that simply <strong>stacking</strong> <strong>more</strong> <strong>layers</strong> <strong>could</strong> <strong>boost</strong> <strong>accuracy</strong>; simplicity turned out to be a strength. <strong>However</strong>, as <strong>networks</strong> <strong>became</strong> <strong>deeper</strong>, problems such as <strong>vanishing gradients</strong> <strong>began</strong> to <strong>emerge</strong>. That’s where <strong>ResNet</strong> changed the game with <strong>skip connections</strong>, allowing <strong>signals</strong> to <strong>flow</strong> more <strong>smoothly</strong>. <strong>DenseNet</strong> <strong>took</strong> the <strong>idea</strong> even <strong>further</strong> by <strong>connecting</strong> <strong>every</strong> <strong>layer to every other</strong>, <strong>maximising</strong> feature <strong>reuse</strong> and <strong>efficiency</strong>.</p><p><strong>Meanwhile</strong>, <strong>not</strong> <strong>all</strong> tasks are about classification. <strong>U-Net</strong> opened new horizons<strong> in medical imaging and segmentation</strong>, where <strong>every</strong> <strong>pixel</strong> <strong>matters</strong>.</p><p>In this article, we’ll walk through these architectures step by step, exploring what makes each one unique and how they’ve shaped the AI systems we use today. First, let&#39;s start with the structure of a Convolutional Neural Network.</p><h4><strong>Convolutional Neural Network</strong></h4><p>Convolutional Neural Networks (CNNs) are a specialised type of deep neural network, primarily designed for processing structured grid data such as images. Their architecture is composed of three main components: feature extraction, classification and Probabilistic Distribution. Through layered operations involving convolution and pooling, CNNs are capable of learning hierarchical patterns from raw image data, making them highly effective in computer vision tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XJLImHX81Nwc3rrpdIQq8Q.png" /><figcaption><strong>Architecture of Convolutional Neural Network</strong></figcaption></figure><p>The overall architecture of <strong>CNNs</strong> consists of <strong>three</strong> <strong>main</strong> <strong>parts</strong>: <strong>feature</strong> <strong>extractors, </strong>a <strong>classifier and a probabilistic distribution</strong>.</p><p><strong>Feature Extraction: </strong>In the feature extraction layers, <strong>each layer</strong> of the network <strong>receives</strong> the <strong>output</strong> from its <strong>immediate</strong> <strong>previous</strong> <strong>layer</strong> as its input, and <strong>passes</strong> <strong>its output as the input to the next layer. </strong>The CNN architecture consists of a combination of three types of layers: <strong>convolution, max-pooling, and classification.</strong> There are two types of layers in the low and middle levels of the network: convolutional layers and max-pooling layers. The even-numbered layers are for convolutions, and the odd-numbered layers are for max-pooling operations. The output nodes of the convolution and max-pooling layers are grouped into a 2D plane, known as feature mapping. Each plane of a layer is usually derived from the combination of one or more planes of previous layers. The nodes of a plane are connected to a small region of each connected plane of the previous layer. Each node in the convolution layer extracts features from the input images through convolution operations on the input nodes.</p><p>As seen below, the <strong>convolution</strong> and <strong>pooling</strong> <strong>layers</strong> work together to <strong>create</strong> a <strong>compact</strong> <strong>representation</strong> of the <strong>input</strong> <strong>image</strong>. The objective is to <strong>reduce</strong> the <strong>spatial</strong> <strong>dimensions</strong> while <strong>preserving</strong> <strong>essential</strong> <strong>features</strong>, effectively <strong>lowering</strong> the <strong>computational</strong> <strong>load</strong> on the neural network. <strong>Without</strong> this step, redundant or irrelevant information <strong>would</strong> <strong>increase</strong> the <strong>complexity</strong> of the <strong>model</strong>, making <strong>learning</strong> <strong>less</strong> <strong>efficient</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*ezovpkymS1osf_9EGmmSyw.png" /></figure><p><strong>Classification:</strong> The <strong>output</strong> <strong>of the last</strong> <strong>layer</strong> of the CNN is <strong>used</strong> as the <strong>input</strong> to a <strong>fully connected network</strong>, which is called the <strong>classification</strong> <strong>layer</strong>. Feed-forward neural networks have been <strong>used</strong> <strong>as</strong> <strong>the classification layer</strong>, as they have better performance. <strong>In the classification layer, the desired number of features is selected as inputs with respect to the dimension of the weight matrix of the final neural network. However</strong>, the <strong>fully</strong> <strong>connected</strong> <strong>layers</strong> are <strong>expensive</strong> in terms of network or learning parameters. Nowadays, there are several new techniques, including average pooling and global average pooling, that are used as an alternative to fully-connected networks.</p><p><strong>Probabilistic Distribution: The score of the respective class</strong> is <strong>calculated</strong> in the top classification layer<strong> using a softmax layer, which ranges output 0 to 1.</strong> Based on the highest score, the classifier gives output for the corresponding classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*pNr3U1axGJi0S-XdIo7oHg.png" /></figure><p>In summary, <strong>CNNs</strong> <strong>operate by extracting meaningful spatial features from input images</strong> through successive convolution and pooling layers, and then classifying them using a fully connected layer or its alternatives. This hierarchical structure enables <strong>CNNs</strong> to efficiently <strong>learn</strong> <strong>complex</strong> <strong>patterns</strong> <strong>while reducing computational cost</strong>, making them a powerful tool for modern image classification tasks.</p><h3>AlexNet</h3><p><strong>AlexNet</strong> is a deep convolutional neural network designed for image classification tasks. It <strong>works</strong> in a <strong>similar</strong> way <strong>to</strong> <strong>classical</strong> <strong>CNNs</strong> by using convolution and pooling layers to extract features from images, followed by fully connected layers for classification. <strong>However</strong>, <strong>AlexNet</strong> is deeper and <strong>includes</strong> important improvements like <strong>ReLU</strong> <strong>activation</strong>, <strong>dropout to reduce overfitting, and training on GPUs for faster performance.</strong> These changes made it much <strong>more</strong> <strong>powerful</strong> <strong>than</strong> <strong>earlier</strong> <strong>CNN</strong> <strong>models</strong> and <strong>allowed</strong> it to <strong>recognise a large number of object categories</strong> <strong>accurately</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*yNIgBl9jmpx3XJGpPCVpkQ.png" /><figcaption><strong>Architecture of AlexNet</strong></figcaption></figure><p><strong>AlexNet</strong> <strong>consists</strong> <strong>of</strong> <strong>five convolutional layers followed by three fully connected layers.</strong> One notable feature of its design is that the network was <strong>split into two parts</strong>, each handled by a separate GPU. This was done because, at the time, a <strong>single GPU did not have enough memory to train the full model.</strong> By <strong>dividing</strong> the <strong>network</strong> into <strong>two</strong> <strong>parallel</strong> <strong>branches</strong>, the model could be trained <strong>more efficiently using the hardware available.</strong></p><p>In later versions or adaptations, the final fully connected layer (called the “projection” layer) can be replaced with a smaller one. If the rest of the model is kept frozen and only this last layer is fine-tuned, it becomes possible to create a new vision model with <strong>much lower computational cost</strong> compared to training from scratch.</p><p>A key conclusion from AlexNet was that the depth of the network had been instrumental to its performance. That depth generated <strong>a large number of parameters,</strong> <strong>making</strong> <strong>training</strong> either impractically <strong>slow</strong> or <strong>impossible</strong> if performed <strong>on a CPU.</strong> By training on a GPU, training time could become practical. Still, high-end GPUs of the time were limited to 3GB of memory, not enough to train AlexNet.</p><h3>U-Net</h3><p><strong>U-Net</strong> is a type of neural network primarily <strong>designed</strong> <strong>for</strong> <strong>image</strong> <strong>segmentation</strong>, <strong>which</strong> involves <strong>dividing</strong> an <strong>image</strong> <strong>into</strong> <strong>meaningful</strong> <strong>regions</strong> <strong>to</strong> <strong>detect</strong> <strong>specific</strong> <strong>objects</strong>, such as <strong>isolating a tumour from surrounding healthy</strong> <strong>tissue</strong> in medical scans. Its name comes from the U-shaped structure of its architecture. <strong>U-Net</strong> is especially <strong>popular</strong> in<strong> medical imaging</strong> due to its strong performance, even when only a limited amount of labelled training data is available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O9py422pFyMcOjEg_AjHoA.jpeg" /><figcaption><strong>U-Net Architecture</strong></figcaption></figure><h4>Key Components and Function of U-Net</h4><p><strong>U-Net </strong>is built around <strong>three</strong> <strong>key</strong> <strong>components</strong>: an <strong>encoder</strong>, a <strong>decoder</strong>, and <strong>skip connections.</strong> Together, they enable the network to capture both global context and fine details, making it highly effective for precise image segmentation tasks.</p><h4><strong>Encoder</strong></h4><p>The <strong>encoder</strong> is the first part of the U-Net architecture, <strong>responsible</strong> <strong>for gradually reducing the spatial dimensions of the input </strong>image while <strong>capturing</strong> <strong>its</strong> <strong>essential</strong> <strong>features</strong> <strong>and</strong> <strong>context</strong>. It operates in <strong>multiple stages</strong>, where<strong> each stage</strong> typically <strong>includes</strong> <strong>two convolutional layers with small 3×3 filters</strong> that scan the image to extract local patterns. <strong>These layers</strong> <strong>use</strong> the <strong>ReLU</strong> <strong>activation</strong> <strong>function</strong> to introduce non-linearity, enabling <strong>the model to learn more complex representations. After the convolutions, a 2×2 max pooling layer is applied to downsample the feature maps,</strong> reducing their size but retaining the most important information.<strong> This process helps the network focus on larger, more meaningful features in the image.</strong></p><p><strong>At the end of the encoder, a bottleneck layer connects it to the decoder; </strong>this is <strong>where</strong> the <strong>feature</strong> <strong>maps</strong> are <strong>most compressed</strong>, storing the most abstract and informative <strong>representation of the input.</strong></p><h4>Decoder</h4><p>The <strong>decoder</strong> is the second main component of the U-Net architecture, <strong>responsible for gradually increasing the spatial resolution of the feature maps to reconstruct the original image size.</strong> It <strong>does</strong> this <strong>through</strong> <strong>upsampling</strong> operations, commonly using transposed convolutional layers <strong>that expand the feature maps step by step</strong>. <strong>At each stage, the decoder typically applies two convolutional layers</strong> with 3×3 filters and ReLU activations, similar to the encoder, to refine and clean up the feature representations. <strong>This process allows the network to rebuild detailed spatial information and produce precise segmentation outputs</strong>.</p><h4>Skip connections (Bridge)</h4><p>The <strong>skip connection</strong> is the third main component of the U-Net architecture, <strong>linking corresponding layers of the encoder and decoder.</strong> At each stage, <strong>it</strong> <strong>directly</strong> <strong>transfers feature maps from the encoder to the decoder, allowing the network to combine low-level spatial details with high-level semantic information. </strong>This <strong>helps</strong> the <strong>model</strong> <strong>recover</strong> fine-grained structures that <strong>may have been lost during downsampling, leading to more accurate and precise segmentation results.</strong></p><h4>How U-Net work step by step?</h4><p><strong>Input Image: An </strong>Original grayscale or RGB image is fed into the network</p><p><strong>↓</strong></p><p><strong>Feature Extraction (Encoder): </strong>Convolutions + max pooling reduce spatial size and extract features)</p><p><strong>↓</strong></p><p><strong>Bottleneck: </strong>Most compressed representation; captures high-level features</p><p><strong>↓</strong></p><p><strong>Reconstruction (Decoder): </strong>Upsampling restores image size; reconstructs spatial structure<br><strong> ↓</strong></p><p><strong>Skip Connections: </strong>Transfers detailed features from encoder to decoder for precision</p><p><strong>↓</strong></p><p><strong>Final Prediction: </strong>1×1 convolution maps each pixel to a class; output = input size</p><p>However, the limitations of the U-Net are very obvious. Firstly, the U-Net model does not have good generalisation ability when the convolution layers increase. Secondly, the size of the dataset and the quality of the images have a significant impact on the training network, but in most cases, the available medical datasets are limited. Finally, the U-Net network structure can only be predicted at a single scale and cannot respond well to changes in scale.</p><h3>ResNet (Residual Networks)</h3><p>ResNet (<strong>Residual Network</strong>) is a deep convolutional neural network architecture known for introducing <strong>skip connections</strong>, which help address the training difficulties in very deep networks. In traditional CNNs, as more layers are added, learning becomes harder, and performance can degrade due to <strong>vanishing gradients.</strong> ResNet solves this by <strong>adding the input of a layer directly to its output</strong>, creating shortcut paths that allow gradients to flow more easily during backpropagation. This design enables the network to <strong>learn residual functions,</strong> focusing on what’s changed rather than re-learning everything, making it possible to train much deeper models effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*159Iso2wFRG2ukGClEMZKg.png" /><figcaption><strong>ResNet Architecture</strong></figcaption></figure><h3>Key Features of ResNet</h3><p><strong>Residual Block:</strong> Instead of learning a direct mapping from input to output, a <strong>residual block</strong> learns a residual function, that is, what needs to be added to the input to reach the desired output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*oKpyje-Lx3kIkodjWm73ZQ.png" /></figure><p>Here, <strong>F(x)</strong> represents the learned residual mapping, and x is the input that is directly added to the block’s output.</p><p><strong>Shortcut (Skip) Connections: </strong>These connections <strong>“skip” one or more layers</strong>, allowing the original input to <strong>bypass</strong> <strong>intermediate</strong> <strong>transformations</strong>. They enable the network to <strong>preserve</strong> <strong>essential</strong> <strong>information</strong> and<strong> improve gradient flow during training.</strong></p><p><strong>Identity Mapping: </strong>When the<strong> input x</strong> and<strong> the output</strong> <strong>of the residual block </strong>have the <strong>same dimensions</strong>, <strong>the shortcut connection</strong> becomes <strong>an identity mapping</strong><em>, </em>meaning the input is added directly to the transformed features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*oKpyje-Lx3kIkodjWm73ZQ.png" /></figure><p><strong>Projection Shortcuts: </strong>If the <strong>input</strong> and <strong>output</strong> <strong>dimensions</strong> differ, a <strong>projection shortcut</strong> is used to align them. Typically, this involves a <strong>1×1 convolution</strong> to project the input into the same dimensional space as the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*B9uxvpZedIcQjqWqPS-_gA.png" /></figure><p><strong>Ws:</strong> The projection matrix (or 1×1 convolution kernel).</p><h4>How does ResNet work step by step?</h4><p><strong>Input Image: </strong>Data enters the network</p><p><strong>↓</strong></p><p><strong>Pre-processing: </strong>The input is processed by initial convolutional layers.</p><p><strong>↓</strong></p><p><strong>Residual Blocks: </strong>The main body of the network consists of stacked residual blocks. Within each block:</p><ul><li>The input passes through a few layers (e.g., convolutions, batch normalisation, activation functions).</li><li>The original input is added to the output of these layers via a shortcut connection.</li></ul><p><strong>↓</strong></p><p><strong>Gradual Reduction in Spatial Dimensions: </strong>Later residual blocks may reduce the spatial dimensions (height and width) of the feature maps, often by using a stride of 2 in a convolutional layer within the block or the shortcut.</p><p><strong>↓</strong></p><p><strong>Fully Connected Layer: </strong>After the convolutional and residual blocks, the output is flattened and passed through a fully connected layer for final classification.</p><h4>DenseNet</h4><p><strong>DenseNet (Densely Connected Convolutional Network)</strong> is an advanced deep learning architecture that builds upon the ideas introduced by <strong>ResNet</strong>. While ResNet connects layers using shortcut paths to help gradients flow and prevent vanishing, DenseNet takes this concept further it <strong>directly connecting every layer to all subsequent layers</strong> in a feed-forward manner.</p><p>This dense connectivity ensures that each layer receives the <strong>collective knowledge</strong> (feature maps) of all previous layers, promoting <strong>feature reuse</strong>, improving <strong>information flow</strong>, and significantly reducing the number of parameters compared to equally deep ResNet models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*CUnaXk309whSeFWYYno_Pg.png" /><figcaption><strong>DensNet Architecture</strong></figcaption></figure><h4>Key Features of ResNet</h4><p><strong>Dense Block: </strong>Each <strong>Dense Block</strong> is the core unit of DenseNet. Instead of adding outputs like in ResNet, <strong>DenseNet concatenates</strong> them, meaning every layer receives feature maps from <em>all preceding layers</em> and passes its own output to all subsequent layers. This dense connectivity encourages <strong>feature reuse</strong>, improves <strong>gradient flow</strong>, and makes the model more parameter-efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*rBWeS0ryMfMVZ2SLRu7xCQ.png" /></figure><ul><li>x_l: output of the l.th layer</li><li>H_l(.): composite function (BatchNorm → ReLU → Conv)</li><li>[x_0, x_1, …, x_{l-1}]: concatenation of feature maps from all previous layers</li></ul><p><strong>Bottleneck Layer (1×1 Convolution): </strong>To prevent the number of feature maps from growing too large (since all previous outputs are concatenated), DenseNet introduces <strong>Bottleneck Layers</strong>. They use a <strong>1×1 convolution</strong> before the main <strong>3×3 convolution</strong> to reduce the number of input channels, improving computational efficiency without losing much information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*3LyhPnNUfNlTPW_Br5kwrg.png" /></figure><p><strong>Transition Layer: </strong>Transition layers are placed between Dense Blocks to <strong>compress</strong> and <strong>downsample</strong> feature maps. They help control model size and computational cost. Each transition layer consists of:</p><ul><li>Batch Normalization</li><li>1×1 Convolution (for compression)</li><li>2×2 Average Pooling (for spatial downsampling)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*rBWeS0ryMfMVZ2SLRu7xCQ.png" /></figure><p><strong>Growth Rate (k): </strong>The <strong>growth rate</strong> defines how many new feature maps each layer contributes to the network. If a Dense Block has an input of m_0 feature maps and contains L layers, then the total number of output feature maps becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*04Vc4IsIyc6_XxZwGE-q7Q.png" /></figure><p><strong>Global Average Pooling (GAP) and Fully Connected Layer: </strong>After the last Dense Block, a <strong>Global Average Pooling (GAP)</strong> layer aggregates spatial information from all feature maps into a single vector. Then, a <strong>fully connected (FC)</strong> layer produces the final class probabilities using <strong>Softmax</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/188/1*Czbo-35iT4Ynlej4S3Oiqg.png" /></figure><h4>How does DenseNet work step by step?</h4><p><strong>Input Image:</strong> The raw image is fed into the network as input data.</p><p>↓</p><p><strong>Initial Convolution &amp; Pooling:</strong> The image passes through an initial convolution and pooling layer to extract basic low-level features (like edges, colors, textures).</p><p>↓</p><p><strong>Dense Block 1:</strong> The first dense block contains multiple layers (e.g., Conv + BN + ReLU).</p><p>Each new layer receives the feature maps from <strong>all previous layers</strong> within the block (feature reuse), improving gradient flow and efficiency.</p><p>↓</p><p><strong>Transition Layer 1:</strong> After each dense block, a transition layer (1×1 convolution + 2×2 average pooling) reduces the number and size of feature maps, controlling model complexity.</p><p>↓</p><p><strong>Dense Block 2 &amp; 3:</strong> The same pattern continues — multiple dense blocks stacked sequentially, where each layer within a block connects to all earlier ones.</p><p>This allows later layers to access both low- and high-level features directly.</p><p>↓</p><p><strong>Final Pooling Layer:</strong> Global average pooling is applied to reduce spatial dimensions to a single vector representing the entire image.</p><p>↓</p><p><strong>Fully Connected Layer (FC):</strong> The pooled features are flattened and passed through a fully connected layer to perform classification.</p><p>↓</p><p><strong>Output Layer:</strong> A softmax activation produces the final class probabilities (e.g., 17 categories in the example).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vezSbTSyMUv9xjt-FkKSSA.png" /><figcaption><strong>CNN vs ResNet and DenseNet</strong></figcaption></figure><p><strong>In conclusion,</strong> the evolution of convolutional neural networks has been marked by continuous innovation and specialisation across architectures. <strong>AlexNet</strong> demonstrated the immense power of deep learning by introducing ReLU activations and GPU training, setting the foundation for large-scale image recognition. <strong>U-Net</strong> extended this progress to <strong>medical imaging,</strong> <strong>using encoder–decoder</strong> structures and skip connections to achieve pixel-level segmentation. <strong>ResNet </strong>tackled the vanishing gradient problem through residual learning, allowing networks to scale to unprecedented depths without performance degradation. <strong>DenseNet</strong> further enhanced feature propagation and efficiency by connecting each layer to all others, promoting maximum information reuse.</p><h4>Python Code for CNN</h4><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>import torchvision<br>import torchvision.transforms as transforms<br>import torch.nn.functional as F<br><br># 1. Load CIFAR-10 dataset<br>transform = transforms.Compose([<br>    transforms.ToTensor(),<br>    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>])<br><br>trainset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=True,<br>                                        download=True, transform=transform)<br>trainloader = torch.utils.data.DataLoader(trainset, batch_size=64,<br>                                          shuffle=True, num_workers=2)<br><br>testset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=False,<br>                                       download=True, transform=transform)<br>testloader = torch.utils.data.DataLoader(testset, batch_size=64,<br>                                         shuffle=False, num_workers=2)<br><br>classes = (&#39;plane&#39;, &#39;car&#39;, &#39;bird&#39;, &#39;cat&#39;,<br>           &#39;deer&#39;, &#39;dog&#39;, &#39;frog&#39;, &#39;horse&#39;, &#39;ship&#39;, &#39;truck&#39;)<br><br># 2. Define CNN<br>class BirdCNN(nn.Module):<br>    def __init__(self):<br>        super(BirdCNN, self).__init__()<br>        self.conv1 = nn.Conv2d(3, 16, 3, padding=1)<br>        self.pool = nn.MaxPool2d(2, 2)<br>        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)<br>        self.fc1 = nn.Linear(32 * 8 * 8, 128)<br>        self.fc2 = nn.Linear(128, 10)  # 10 classes in CIFAR-10<br><br>    def forward(self, x):<br>        x = self.pool(F.relu(self.conv1(x)))<br>        x = self.pool(F.relu(self.conv2(x)))<br>        x = x.view(-1, 32 * 8 * 8)<br>        x = F.relu(self.fc1(x))<br>        x = self.fc2(x)<br>        return x<br><br># 3. Initialize model, loss, optimizer<br>net = BirdCNN()<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(net.parameters(), lr=0.001)<br><br># 4. Training loop<br>for epoch in range(2):  # train for 2 epochs (for speed)<br>    running_loss = 0.0<br>    for i, data in enumerate(trainloader, 0):<br>        inputs, labels = data<br>        optimizer.zero_grad()<br><br>        outputs = net(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        optimizer.step()<br><br>        running_loss += loss.item()<br>        if i % 200 == 199:    # print every 200 mini-batches<br>            print(f&#39;[Epoch {epoch+1}, Batch {i+1}] loss: {running_loss/200:.3f}&#39;)<br>            running_loss = 0.0<br><br>print(&quot;Finished Training&quot;)<br><br># 5. Test accuracy<br>correct = 0<br>total = 0<br>with torch.no_grad():<br>    for data in testloader:<br>        images, labels = data<br>        outputs = net(images)<br>        _, predicted = torch.max(outputs.data, 1)<br>        total += labels.size(0)<br>        correct += (predicted == labels).sum().item()<br><br>print(f&#39;Accuracy on CIFAR-10 test images: {100 * correct / total:.2f}%&#39;)</pre><h4>Python Code for AlexNet</h4><pre>import torch<br>import torch.nn as nn<br><br>class AlexNetMedical(nn.Module):<br>    def __init__(self, num_classes=2):<br>        super(AlexNetMedical, self).__init__()<br>        <br>        # Feature extraction part (convolutions)<br>        self.features = nn.Sequential(<br>            nn.Conv2d(1, 64, kernel_size=11, stride=4, padding=2),  # grayscale input<br>            nn.ReLU(inplace=True),<br>            nn.MaxPool2d(kernel_size=3, stride=2),<br>            <br>            nn.Conv2d(64, 192, kernel_size=5, padding=2),<br>            nn.ReLU(inplace=True),<br>            nn.MaxPool2d(kernel_size=3, stride=2),<br>            <br>            nn.Conv2d(192, 384, kernel_size=3, padding=1),<br>            nn.ReLU(inplace=True),<br>            nn.Conv2d(384, 256, kernel_size=3, padding=1),<br>            nn.ReLU(inplace=True),<br>            nn.Conv2d(256, 256, kernel_size=3, padding=1),<br>            nn.ReLU(inplace=True),<br>            nn.MaxPool2d(kernel_size=3, stride=2),<br>        )<br>        <br>        # Fully connected classifier part<br>        self.classifier = nn.Sequential(<br>            nn.Dropout(),<br>            nn.Linear(256 * 6 * 6, 4096),<br>            nn.ReLU(inplace=True),<br>            nn.Dropout(),<br>            nn.Linear(4096, 4096),<br>            nn.ReLU(inplace=True),<br>            nn.Linear(4096, num_classes),<br>        )<br><br>    def forward(self, x):<br>        x = self.features(x)            # convolutional feature maps<br>        x = torch.flatten(x, 1)         # flatten into vector<br>        x = self.classifier(x)          # final classification<br>        return x</pre><h4>Python Code for U-Net</h4><pre>import torch<br>import torch.nn as nn<br><br>class UNet(nn.Module):<br>    def __init__(self, in_channels=1, out_channels=1):<br>        super(UNet, self).__init__()<br>        <br>        def conv_block(in_ch, out_ch):<br>            return nn.Sequential(<br>                nn.Conv2d(in_ch, out_ch, kernel_size=3, padding=1),<br>                nn.ReLU(inplace=True),<br>                nn.Conv2d(out_ch, out_ch, kernel_size=3, padding=1),<br>                nn.ReLU(inplace=True),<br>            )<br>        <br>        # Encoder (downsampling path)<br>        self.enc1 = conv_block(in_channels, 64)<br>        self.enc2 = conv_block(64, 128)<br>        self.enc3 = conv_block(128, 256)<br>        self.enc4 = conv_block(256, 512)<br>        <br>        self.pool = nn.MaxPool2d(2)<br>        <br>        # Bottleneck<br>        self.bottleneck = conv_block(512, 1024)<br>        <br>        # Decoder (upsampling path with skip connections)<br>        self.up4 = nn.ConvTranspose2d(1024, 512, kernel_size=2, stride=2)<br>        self.dec4 = conv_block(1024, 512)<br>        <br>        self.up3 = nn.ConvTranspose2d(512, 256, kernel_size=2, stride=2)<br>        self.dec3 = conv_block(512, 256)<br>        <br>        self.up2 = nn.ConvTranspose2d(256, 128, kernel_size=2, stride=2)<br>        self.dec2 = conv_block(256, 128)<br>        <br>        self.up1 = nn.ConvTranspose2d(128, 64, kernel_size=2, stride=2)<br>        self.dec1 = conv_block(128, 64)<br>        <br>        # Final output layer<br>        self.final = nn.Conv2d(64, out_channels, kernel_size=1)<br><br>    def forward(self, x):<br>        # Encoder path<br>        e1 = self.enc1(x)<br>        e2 = self.enc2(self.pool(e1))<br>        e3 = self.enc3(self.pool(e2))<br>        e4 = self.enc4(self.pool(e3))<br>        <br>        # Bottleneck<br>        b = self.bottleneck(self.pool(e4))<br>        <br>        # Decoder path with skip connections<br>        d4 = self.dec4(torch.cat([self.up4(b), e4], dim=1))<br>        d3 = self.dec3(torch.cat([self.up3(d4), e3], dim=1))<br>        d2 = self.dec2(torch.cat([self.up2(d3), e2], dim=1))<br>        d1 = self.dec1(torch.cat([self.up1(d2), e1], dim=1))<br>        <br>        return torch.sigmoid(self.final(d1))  # segmentation mask</pre><h4>Compare CNN and ResNet in terms of the Vanishing Problem</h4><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>import torchvision<br>import torchvision.transforms as transforms<br>import torch.nn.functional as F<br>import matplotlib.pyplot as plt<br><br>device = torch.device(&#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;)<br><br># 1. Load CIFAR-10 dataset<br>transform = transforms.Compose([<br>    transforms.ToTensor(),<br>    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>])<br><br>trainset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=True,<br>                                        download=True, transform=transform)<br>trainloader = torch.utils.data.DataLoader(trainset, batch_size=64,<br>                                          shuffle=True, num_workers=2)<br><br>testset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=False,<br>                                       download=True, transform=transform)<br>testloader = torch.utils.data.DataLoader(testset, batch_size=64,<br>                                         shuffle=False, num_workers=2)<br><br>classes = (&#39;plane&#39;, &#39;car&#39;, &#39;bird&#39;, &#39;cat&#39;,<br>           &#39;deer&#39;, &#39;dog&#39;, &#39;frog&#39;, &#39;horse&#39;, &#39;ship&#39;, &#39;truck&#39;)<br><br># -------------------- Model 1: Simple CNN --------------------<br>class SimpleCNN(nn.Module):<br>    def __init__(self):<br>        super(SimpleCNN, self).__init__()<br>        self.conv1 = nn.Conv2d(3, 32, 3, padding=1)<br>        self.conv2 = nn.Conv2d(32, 64, 3, padding=1)<br>        self.conv3 = nn.Conv2d(64, 128, 3, padding=1)<br>        self.pool = nn.MaxPool2d(2, 2)<br>        self.fc = nn.Linear(128 * 3 * 3, 10)  <br><br>    def forward(self, x):<br>        x = self.pool(F.relu(self.conv1(x)))  <br>        x = self.pool(F.relu(self.conv2(x)))<br>        x = self.pool(F.relu(self.conv3(x)))<br>        x = x.view(x.size(0), -1)<br>        x = self.fc(x)<br>        return x<br><br><br># -------------------- Model 2: Mini ResNet with Residual Blocks --------------------<br>class ResidualBlock(nn.Module):<br>    def __init__(self, channels):<br>        super(ResidualBlock, self).__init__()<br>        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)<br>        self.bn1 = nn.BatchNorm2d(channels)<br>        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)<br>        self.bn2 = nn.BatchNorm2d(channels)<br>        self.relu = nn.ReLU()<br><br>    def forward(self, x):<br>        identity = x<br>        out = self.relu(self.bn1(self.conv1(x)))<br>        out = self.bn2(self.conv2(out))<br>        return self.relu(out + identity)<br><br>class MiniResNet(nn.Module):<br>    def __init__(self):<br>        super(MiniResNet, self).__init__()<br>        self.conv1 = nn.Conv2d(3, 32, 3, padding=1)<br>        self.bn1 = nn.BatchNorm2d(32)<br>        self.pool = nn.MaxPool2d(2, 2)<br><br>        self.resblock1 = ResidualBlock(32)<br>        self.conv2 = nn.Conv2d(32, 64, 3, padding=1)<br>        self.bn2 = nn.BatchNorm2d(64)<br>        self.resblock2 = ResidualBlock(64)<br>        self.conv3 = nn.Conv2d(64, 128, 3, padding=1)<br>        self.bn3 = nn.BatchNorm2d(128)<br>        self.resblock3 = ResidualBlock(128)<br><br>        self.fc = nn.Linear(128 * 3 * 3, 10)<br><br>    def forward(self, x):<br>        x = self.pool(F.relu(self.bn1(self.conv1(x))))<br>        x = self.resblock1(x)<br>        x = self.pool(F.relu(self.bn2(self.conv2(x))))<br>        x = self.resblock2(x)<br>        x = self.pool(F.relu(self.bn3(self.conv3(x))))<br>        x = self.resblock3(x)<br>        x = x.view(x.size(0), -1)<br>        return self.fc(x)<br><br># -------------------- Training Function --------------------<br>def train_model(model, loader, epochs=10):<br>    model = model.to(device)<br>    criterion = nn.CrossEntropyLoss()<br>    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)<br>    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.5)<br><br>    losses, accs = [], []<br><br>    for epoch in range(epochs):<br>        total_loss, correct, total = 0, 0, 0<br>        model.train()<br>        for images, labels in loader:<br>            images, labels = images.to(device), labels.to(device)<br>            # Ensure images have 3 channels<br>            if images.shape[1] == 1:<br>                images = images.repeat(1, 3, 1, 1)<br><br>            outputs = model(images)<br>            loss = criterion(outputs, labels)<br>            optimizer.zero_grad()<br>            loss.backward()<br>            optimizer.step()<br><br>            total_loss += loss.item()<br>            _, predicted = torch.max(outputs.data, 1)<br>            total += labels.size(0)<br>            correct += (predicted == labels).sum().item()<br><br>        acc = 100 * correct / total<br>        losses.append(total_loss / len(loader))<br>        accs.append(acc)<br>        scheduler.step()<br>        print(f&quot;Epoch [{epoch+1}/{epochs}] Loss: {total_loss:.3f}, Acc: {acc:.2f}%&quot;)<br>    return losses, accs<br><br># -------------------- Train both models --------------------<br>print(&quot;\n=== Training SimpleCNN ===&quot;)<br>cnn = SimpleCNN()<br>cnn_loss, cnn_acc = train_model(cnn, train_loader, epochs=10)<br><br>print(&quot;\n=== Training MiniResNet ===&quot;)<br>resnet = MiniResNet()<br>resnet_loss, resnet_acc = train_model(model=resnet, loader=train_loader, epochs=10)<br><br># -------------------- Plot comparison --------------------<br>plt.figure(figsize=(12,5))<br>plt.subplot(1,2,1)<br>plt.plot(cnn_loss, label=&#39;SimpleCNN&#39;)<br>plt.plot(resnet_loss, label=&#39;MiniResNet&#39;)<br>plt.title(&#39;Training Loss&#39;)<br>plt.xlabel(&#39;Epoch&#39;)<br>plt.ylabel(&#39;Loss&#39;)<br>plt.legend()<br><br>plt.subplot(1,2,2)<br>plt.plot(cnn_acc, label=&#39;SimpleCNN&#39;)<br>plt.plot(resnet_acc, label=&#39;MiniResNet&#39;)<br>plt.title(&#39;Training Accuracy&#39;)<br>plt.xlabel(&#39;Epoch&#39;)<br>plt.ylabel(&#39;Accuracy (%)&#39;)<br>plt.legend()<br><br>plt.tight_layout()<br>plt.show()</pre><p><strong>Python Code for DenseNet</strong></p><pre># ============================================================<br># DenseNet Mini Example with CIFAR-10<br># ============================================================<br># Covers: Bottleneck Layer → Dense Block → Transition Layer<br># Uses: real CIFAR-10 images<br># ============================================================<br><br>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import torch.optim as optim<br>import torchvision<br>import torchvision.transforms as transforms<br><br><br># ============================================================<br># Step 1: Define the Bottleneck Layer<br># ============================================================<br><br>class BottleneckLayer(nn.Module):<br>    def __init__(self, in_channels, growth_rate):<br>        super(BottleneckLayer, self).__init__()<br>        inner_channels = 4 * growth_rate<br><br>        self.bn1 = nn.BatchNorm2d(in_channels)<br>        self.relu1 = nn.ReLU(inplace=True)<br>        self.conv1 = nn.Conv2d(in_channels, inner_channels, kernel_size=1, bias=False)<br><br>        self.bn2 = nn.BatchNorm2d(inner_channels)<br>        self.relu2 = nn.ReLU(inplace=True)<br>        self.conv2 = nn.Conv2d(inner_channels, growth_rate, kernel_size=3, padding=1, bias=False)<br><br>    def forward(self, x):<br>        out = self.conv1(self.relu1(self.bn1(x)))<br>        out = self.conv2(self.relu2(self.bn2(out)))<br>        out = torch.cat([x, out], 1)  # Dense connection<br>        return out<br><br><br># ============================================================<br># Step 2: Define a Dense Block<br># ============================================================<br><br>class DenseBlock(nn.Module):<br>    def __init__(self, num_layers, in_channels, growth_rate):<br>        super(DenseBlock, self).__init__()<br>        self.layers = nn.ModuleList()<br>        for i in range(num_layers):<br>            layer = BottleneckLayer(in_channels + i * growth_rate, growth_rate)<br>            self.layers.append(layer)<br><br>    def forward(self, x):<br>        for layer in self.layers:<br>            x = layer(x)<br>        return x<br><br><br># ============================================================<br># Step 3: Define the Transition Layer<br># ============================================================<br><br>class TransitionLayer(nn.Module):<br>    def __init__(self, in_channels, compression=0.5):<br>        super(TransitionLayer, self).__init__()<br>        out_channels = int(in_channels * compression)<br>        self.bn = nn.BatchNorm2d(in_channels)<br>        self.relu = nn.ReLU(inplace=True)<br>        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=1, bias=False)<br>        self.pool = nn.AvgPool2d(2, stride=2)<br><br>    def forward(self, x):<br>        x = self.conv(self.relu(self.bn(x)))<br>        x = self.pool(x)<br>        return x<br><br><br># ============================================================<br># Step 4: Define a Mini DenseNet for CIFAR-10<br># ============================================================<br><br>class MiniDenseNet(nn.Module):<br>    def __init__(self, in_channels=3, growth_rate=12, num_classes=10):<br>        super(MiniDenseNet, self).__init__()<br>        self.initial_conv = nn.Conv2d(in_channels, 24, kernel_size=3, padding=1, bias=False)<br>        self.block1 = DenseBlock(4, 24, growth_rate)<br>        self.trans1 = TransitionLayer(24 + 4 * growth_rate)<br>        self.block2 = DenseBlock(4, int((24 + 4 * growth_rate) * 0.5), growth_rate)<br>        self.trans2 = TransitionLayer(int((24 + 4 * growth_rate) * 0.5) + 4 * growth_rate)<br>        self.classifier = nn.Linear(int(((24 + 4 * growth_rate) * 0.5 + 4 * growth_rate) * 0.5), num_classes)<br><br>    def forward(self, x):<br>        x = self.initial_conv(x)<br>        print(&quot; Input Shape:&quot;, x.shape)<br><br>        x = self.block1(x)<br>        print(&quot; After Dense Block 1:&quot;, x.shape)<br><br>        x = self.trans1(x)<br>        print(&quot;After Transition 1:&quot;, x.shape)<br><br>        x = self.block2(x)<br>        print(&quot; After Dense Block 2:&quot;, x.shape)<br><br>        x = self.trans2(x)<br>        print(&quot; After Transition 2:&quot;, x.shape)<br><br>        x = F.adaptive_avg_pool2d(x, (1, 1))<br>        x = torch.flatten(x, 1)<br>        x = self.classifier(x)<br>        return x<br><br><br># ============================================================<br># Step 5: Load CIFAR-10 Dataset<br># ============================================================<br><br>transform = transforms.Compose([<br>    transforms.ToTensor(),<br>    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))<br>])<br><br>trainset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=True, download=True, transform=transform)<br>trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)<br><br>testset = torchvision.datasets.CIFAR10(root=&#39;./data&#39;, train=False, download=True, transform=transform)<br>testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=False)<br><br><br># ============================================================<br># Step 6: Train on CIFAR-10<br># ============================================================<br><br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>model = MiniDenseNet().to(device)<br><br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(model.parameters(), lr=0.001)<br><br>print(&quot;\n Starting training...\n&quot;)<br><br>for epoch in range(2):  # small demo: 2 epochs<br>    running_loss = 0.0<br>    for i, (inputs, labels) in enumerate(trainloader):<br>        inputs, labels = inputs.to(device), labels.to(device)<br><br>        optimizer.zero_grad()<br>        outputs = model(inputs)<br>        loss = criterion(outputs, labels)<br>        loss.backward()<br>        optimizer.step()<br><br>        running_loss += loss.item()<br>        if i % 100 == 99:<br>            print(f&quot;[Epoch {epoch+1}, Batch {i+1}] Loss: {running_loss/100:.3f}&quot;)<br>            running_loss = 0.0<br><br>print(&quot;\n Training complete!&quot;)<br><br># ============================================================<br># Step 7: Evaluate on Test Data<br># ============================================================<br><br>correct, total = 0, 0<br>model.eval()<br>with torch.no_grad():<br>    for inputs, labels in testloader:<br>        inputs, labels = inputs.to(device), labels.to(device)<br>        outputs = model(inputs)<br>        _, predicted = torch.max(outputs, 1)<br>        total += labels.size(0)<br>        correct += (predicted == labels).sum().item()<br><br>print(f&quot;\n Test Accuracy: {100 * correct / total:.2f}%&quot;)</pre><p><strong>References:</strong></p><ul><li><a href="https://developersbreach.com/convolution-neural-network-deep-learning/">https://developersbreach.com/convolution-neural-network-deep-learning/</a></li><li><a href="https://www.sciencedirect.com/topics/computer-science/u-net">https://www.sciencedirect.com/topics/computer-science/u-net</a></li><li><a href="https://pdf.sciencedirectassets.com/271093/1-s2.0-S0955598625X00042/1-s2.0-S0955598625001384/main.pdf?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEIT%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJGMEQCIApqjl%2BHZZ6xvnotqZT3W7BadxH%2BOpiYHOTz4YOrPKV5AiAf9ohGoOevNSLZ7zgIM7QJAmYrj8nsxyQSzd4vzM6HDCq8BQjM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAUaDDA1OTAwMzU0Njg2NSIMYb6570aP9T9II2qfKpAFqlYYiaqXnmZjRuUgh9mfGM3mnxQWh4ZZqLVC5dPgwPcqhsxfqhXyTJYJFzq1NZNjNjgR4s%2FWJf69rpxVdwF2IaT3NWChhQJUrjPXMEa3a3Cl8%2FvA2ggITRbIQSQ2sJzcIbE7TTFIzSHIYX%2BaazQoRjoBSlqdzgIrV601sNBPWPK7VonrOS6v5Nca2raXZZWPwp%2F2W3iVIAQDyZeWSo4aGn4vgdRMIUZrEgGlIHWE2y59KFibf6PpFxsMrPALRk7cgAlRAUeEzUayKoyPp2DJv7S6IYQR2mM7YDBlSJ147UJ3yidMPuk0NEcFtLTXLsEYiWD8jXCiQ4MsAiCdvNEzadDzjnqODXcZjLD8GCZZLB4BS7%2FEypkrsUNYnvrGwp%2BBWM91UOpBTiPweaEfXbalc8HCIOHGuUKOPAncuGpog8fkbZbt6WS1gq15fVcplY6xeCFso7kZekQutYaecak0hmhGvXdHy82G3uEbib53T6BdHQVGpTUSp%2F2DUoVW%2Fn0okr4omFxEmUmf6CBr2iC7GjJDndI%2BWiwJ4C5bh17fLE7PoeiDqc5m2mgkGWDHZlX%2B5Yt5svXFJZHsxwZKZP3rGZ0LRgNc4vdsC%2FI5PDVuFxAQ0fBm%2FnyZYYBw8HqgsIAREuQRS7gzZWsMBP%2F1BKzKWV49z%2B81HzY8sx15MJ5qE1FcUt6VZYDfquSE1BVmIzA%2BHj51e%2FU%2BwDbV4FqatCWzN9DnXkI1uvC1siKLfKqYDEawue3PrOiYOWo419lxNAbnu9hIUm%2FcXoXj2BHG4VOQAlRDtoadxmpU0aAOGz8l6u4n7ArrGTP2PEqiHZzNDwEbJvbzyL6rZSZN5X3BKxTNx9hwtslZ3F4VTPJefNJwLhkwm%2FmUxQY6sgElFX3z51Ez6UHYj1xSVoWKMcHoPXod%2FQ1tAsrR6uELP21EMdFywJIoeRYagSa50gwBewvG6%2BIq8ViKJkPQcJZaBbQhBWL%2BnMAkJTUB9V%2FFI6zIiZqz6xVkiW91BTC4dNEZvkuj6wYCO1%2BlIzboiiaM2qkZUOkkTzfu4BvpMXAvw0M5TWv9V%2BLZWZqz4VCJvs5QCDa%2B7XUV5mbKeCDSuNGdb5ETK5q%2F29ae6L2ty4aw7zYf&amp;X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20250820T033035Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=300&amp;X-Amz-Credential=ASIAQ3PHCVTY4ZP3M5YD%2F20250820%2Fus-east-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=908adf23b7cce526d35d898cf8134b4b51832b5a514582fd05e7f514d6ccc522&amp;hash=49445601438a4fb77b07c370a4d2373fbd4185ba5226aa3ef935c9afc1710fe0&amp;host=68042c943591013ac2b2430a89b270f6af2c76d8dfd086a07176afe7c76c2c61&amp;pii=S0955598625001384&amp;tid=spdf-f2449508-0272-430c-b286-063375b3837c&amp;sid=703b97593bea964c220a79e6d7acc24fcf8dgxrqb&amp;type=client&amp;tsoh=d3d3LnNjaWVuY2VkaXJlY3QuY29t&amp;rh=d3d3LnNjaWVuY2VkaXJlY3QuY29t&amp;ua=0b075853560b045e5f&amp;rr=971ed0199a5acac9&amp;cc=de">https://pdf.sciencedirectassets.com/271093/1-s2.0-S0955598625X00042/1-s2.0-S0955598625001384/main.pdf?X-Amz-Security-</a></li><li><a href="https://www.sciencedirect.com/science/article/pii/S0955598625001384">https://www.sciencedirect.com/science/article/pii/S0955598625001384</a></li><li><a href="https://www.scaler.com/topics/deep-learning/efficientNet/">https://www.scaler.com/topics/deep-learning/efficientNet/</a></li><li><a href="https://www.scaler.com/topics/deep-learning/efficientNet/">https://www.scaler.com/topics/deep-learning/efficientNet/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66d37fe88a88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thrustworthy Machine Learning: Concept-Based Models: Testing with Concept Activation Vector (TCAV)]]></title>
            <link>https://medium.com/@yalcinselcuk0/thrustworthy-machine-learning-concept-based-models-testing-with-concept-activation-vector-tcav-da951fff4ece?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/da951fff4ece</guid>
            <category><![CDATA[saliency-model]]></category>
            <category><![CDATA[tcav]]></category>
            <category><![CDATA[concept-based-learning]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[trustworthy-ai]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Fri, 05 Sep 2025 01:42:56 GMT</pubDate>
            <atom:updated>2025-09-05T01:42:56.279Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*mePMNL5ZjCaaGKc-E9kpaw.gif" /></figure><p>Understanding the behaviour of <strong>modern machine learning </strong>(ML) models, especially deep neural networks, remains one of the central challenges in AI research. These models<strong> are powerful but often treated as black boxes, making it difficult to understand why a particular decision was made.</strong> Since ML is applied in sensitive areas from medical imaging to autonomous driving , interpretability is not just an academic concern; it is essential for building trust, ensuring accountability, and aligning models with human values.</p><p>To understand why <strong>interpretability</strong> <strong>plays</strong> such a central <strong>role</strong>, it <strong>helps</strong> to consider the <strong>concrete</strong> <strong>benefits</strong> it provides for both researchers and practitioners. <strong>Interpretability</strong> serves several important <strong>goals</strong>. <strong>First</strong>, it enables <strong>scientists</strong> and <strong>engineers</strong> to <strong>design</strong>, <strong>develop</strong>, and <strong>debug</strong> <strong>ML models </strong>more effectively. <strong>Second</strong>, it helps <strong>ensure</strong> that <strong>models</strong> <strong>reflect human values </strong>and are not driven by hidden biases. <strong>Finally</strong>, it <strong>improves</strong> <strong>transparency</strong>, enabling <strong>users</strong> to <strong>place</strong> <strong>greater</strong> <strong>trust</strong> in model predictions.</p><p>To see what interpretability looks like in practice, consider some simple examples. Classic models such as <strong>logistic</strong> <strong>regression</strong> are relatively <strong>easy</strong> <strong>to</strong> <strong>interpret</strong>, since their <strong>coefficient</strong> <strong>weights</strong> can be read as <strong>indicators</strong> of each <strong>feature’s</strong> <strong>importance</strong>. Similarly, <strong>saliency</strong> <strong>maps</strong> in computer vision <strong>highlight</strong> which <strong>pixels</strong> contribute <strong>most</strong> to a <strong>decision</strong>, based on gradients.</p><p>The problem, however, is that <strong>neural</strong> <strong>networks</strong> often <strong>operate</strong> on <strong>low</strong>-<strong>level</strong> <strong>features (pixels, embeddings, activations)</strong> that <strong>do</strong> <strong>not</strong> <strong>directly</strong> <strong>correspond</strong> <strong>to high-level human concepts such as “striped,” “furry,” or “young.”</strong> For the model, these concepts exist only as points in a high-dimensional activation space, which often feels incomprehensible to humans. This gap between machine features and human concepts motivates the need for new methods of explanation.</p><p>This gap can also be described mathematically. <strong>The machine space E_m</strong> is <strong>spanned</strong> by <strong>vectors</strong> <strong>corresponding</strong> to <strong>input features or neural activations</strong>, while the <strong>human</strong> <strong>space</strong> <strong>E_h</strong> is spanned by vectors corresponding to <strong>human</strong>-<strong>understandable</strong> <strong>concepts</strong>. <strong>Interpretability can then be seen as a function that maps machine activations into human concepts:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*aplJQ3cxfE02LlD-7xDhkQ.png" /></figure><p><strong>which maps machine activations into human concepts.</strong> In practice, this mapping doesn’t need to be perfect; it just needs to give us a usable bridge between machine reasoning and human understanding.</p><p><strong>Concept Activation Vectors (CAVs)</strong> offer a simple yet powerful way to instantiate this bridge. By training a linear classifier on concept examples versus counterexamples, we obtain a direction in the model’s representation space that corresponds to the concept; the normalised version of this direction is the CAV.</p><p>Building on this idea, <strong>Testing with CAVs (TCAV)</strong> <strong>quantifies</strong> how<strong> important</strong> <strong>a concept is for a class prediction.</strong> <strong>Instead</strong> <strong>of pixel-level explanations</strong> (as with saliency maps), TCAV provides class-level, concept-based insights, for example, how much the concept of “stripes” influences the model’s prediction of “zebra.”</p><p>With this foundation in place, we can now walk through the <strong>TCAV workflow</strong> step by step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkPN0lvG30OBpOlVEWCwLw.png" /><figcaption><strong>Workflow of TCAV</strong></figcaption></figure><p><strong>Step 1: Define a concept of interest</strong></p><p>The first step in our method is to define a concept of interest. We do this <strong>simply</strong> by <strong>choosing</strong> <strong>a set of examples that represent this concept</strong>, or by finding an independent data set with the concept labelled.</p><p><strong>Step 2: Learn the Concept Activation Vector (CAV)</strong></p><p>Feed both sets into the network and extract activations from a chosen layer. Train a simple linear classifier to separate concept examples from random ones. The separating vector becomes the <strong>CAV(unit vector)</strong>, which represents the concept in that layer’s activation space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*SpV3rXQ4kg4hlFFtMmGIPA.png" /><figcaption><strong>Concept Activation Vector</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*TgblpmC5AvgWhXtS2UiJWQ.png" /><figcaption><strong>Concept Activation Vector(Unit Vector)</strong></figcaption></figure><h4>Step 3: Measure Concept Sensitivity</h4><p>With the <strong>CAV</strong> defined, we now measure how <strong>sensitive</strong> a <strong>class</strong> <strong>prediction</strong> is <strong>to this concept</strong>. For each test input <strong><em>x</em></strong>, we <strong>compute</strong> the <strong>gradient</strong> of the t<strong>arget logit with respect to the representation f_l(x) and project it onto the CAV.</strong> This projection shows whether moving in the concept direction increases or decreases the class score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*A3Axt3dWOYsI8GDPzsbcFg.png" /></figure><h4>Step 4: Compute the TCAV Score</h4><p>To quantify the overall importance of the concept, we calculate the fraction of test inputs where the projection is positive. This <strong>TCAV score</strong> <strong>gives</strong> a <strong>single number </strong>showing <strong>how much</strong> the model relies on the concept for predicting the class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*ClCIzPfucrRG_en0Am4NrA.png" /><figcaption><strong>TCAV Score</strong></figcaption></figure><h3>Python Code for Testing with Concept Activation Vector</h3><p><strong>Step 1 — Define a Concept</strong></p><p>In this step, we choose the datasets that will represent our concept, our control, and our test class.</p><ul><li>The <strong>concept set ©</strong> is <strong>dalmatian</strong>, representing a “dotted” texture.</li><li>The <strong>control set</strong> is <strong>airplanes</strong>, chosen as a random, unrelated class.</li><li>The <strong>test set (Xₖ)</strong> is <strong>leopards</strong>, which is the class we will later measure the sensitivity of.</li></ul><p>This setup allows us to learn a <strong>Concept Activation Vector (CAV)</strong> that captures the “dotted” idea, and then test how important this concept is for predicting the leopard class.</p><pre># ============================================================<br># Load Dataset and Model<br>#  • Use Caltech101 Dalmatian as the dotted (spotty) concept (positive).<br>#  • Use Caltech101 Airplanes as the random control (negative).<br>#  • Use Caltech101 Leopards as the test set (target class: ImageNet &#39;leopard&#39; idx=288).<br>#  • Pretrained ResNet50 (ImageNet) provides penultimate activations (avg_pool).<br># ============================================================<br><br># ============================================================<br># TCAV pipeline (Caltech101: dalmatian vs airplanes → test on leopards)<br># ============================================================<br><br>import tensorflow as tf, tensorflow_datasets as tfds, numpy as np<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.pipeline import make_pipeline<br><br># ===========================<br># Step 1 — Define a Concept<br># Choose positive concept &amp; a random control; collect example sets.<br># Concept C = &#39;dalmatian&#39; (dotted), Control = &#39;airplanes&#39; (random),<br># Test set X_k = &#39;leopards&#39; (class k we will measure sensitivity on).<br># ===========================<br>tf.random.set_seed(0); np.random.seed(0)<br>IMG_SIZE,BATCH,N_POS,N_NEG,N_TEST = 224,16,80,80,80<br>ds, info = tfds.load(&quot;caltech101&quot;, with_info=True, as_supervised=True)<br>train, names = ds[&quot;train&quot;], info.features[&quot;label&quot;].names<br>idx_dalmatian, idx_airplanes, idx_leopards = names.index(&quot;dalmatian&quot;), names.index(&quot;airplanes&quot;), names.index(&quot;leopards&quot;)<br><br>def only_label(idx): return lambda x,y: tf.equal(y, idx)<br>concept_ds, control_ds, test_ds = train.filter(only_label(idx_dalmatian)).take(N_POS), train.filter(only_label(idx_airplanes)).take(N_NEG), train.filter(only_label(idx_leopards)).take(N_TEST)<br><br>def preprocess(x,y): <br>    x = tf.image.resize(x,(IMG_SIZE,IMG_SIZE))<br>    return tf.keras.applications.resnet50.preprocess_input(x), y<br>def make_loader(ds,batch=BATCH): return ds.map(preprocess).batch(batch).prefetch(tf.data.AUTOTUNE)<br>ldr_pos, ldr_neg, ldr_test = make_loader(concept_ds), make_loader(control_ds), make_loader(test_ds, batch=1)</pre><p><strong>Step 2 — Learn the Concept Activation Vector (CAV)</strong></p><p>In this step, we extract the internal representations (features) of images at a chosen layer of the network (for example, the <strong>average pooling layer of ResNet50</strong>). These features are denoted as R = f_l(x).</p><p>We then train a simple linear classifier to separate the <strong>concept images</strong> (label = 1) from the <strong>control images</strong> (label = 0).</p><p>The <strong>weight vector of this classifier</strong> points in the direction that best distinguishes concept from control. After normalising this vector to unit length, it becomes the <strong>Concept Activation Vector (CAV)</strong>, which represents the learned direction of the concept in feature space.</p><pre># ===========================<br># Step 2 — Learn the Concept Activation Vector (CAV)<br># Extract layer-l features R = f_l(x) and train a linear classifier:<br>#   concept (1) vs control (0).<br># The separating hyperplane’s weight vector → v_l^C (Concept Activation Vector),<br># normalized to unit length.<br># ===========================<br>base = tf.keras.applications.ResNet50(weights=&quot;imagenet&quot;, include_top=True)<br>feat_model = tf.keras.Model(base.input, base.get_layer(&quot;avg_pool&quot;).output)  # R ∈ R^2048<br>pred_layer = base.get_layer(&quot;predictions&quot;)  # logits h_k(·)<br><br>def extract_features(loader):<br>    feats=[]<br>    for x,_ in loader: feats.append(feat_model(x, training=False).numpy())<br>    return np.concatenate(feats,axis=0) if feats else np.zeros((0,2048),np.float32)<br><br>X_pos,X_neg = extract_features(ldr_pos), extract_features(ldr_neg)<br>y = np.concatenate([np.ones(len(X_pos),int), np.zeros(len(X_neg),int)])<br>X = np.concatenate([X_pos,X_neg], axis=0)<br><br>clf = make_pipeline(StandardScaler(with_mean=True,with_std=True),<br>                    LogisticRegression(max_iter=5000,solver=&quot;liblinear&quot;,class_weight=&quot;balanced&quot;))<br>clf.fit(X,y)<br>w = clf.named_steps[&quot;logisticregression&quot;].coef_.ravel().astype(np.float32)<br>cav = w / (np.linalg.norm(w)+1e-12)   # v_l^C</pre><p><strong>Step 3 — Measure Concept Sensitivity</strong></p><p>For each test image, compute the gradient of the target class logit with respect to the representation. Then take the dot product of the CAV with itself.</p><ul><li>Positive value → the concept increases the class score.</li><li>Negative value → the concept decreases the class score.</li></ul><pre># ===========================<br># Step 3 — Measure Concept Sensitivity<br># For each test image x:<br>#   1. Compute the gradient of the target logit wrt the representation R:<br>#        ∇_R h_k(f_l(x))<br>#   2. Project gradient onto concept direction v_l^C:<br>#        S_{C,k,l}(x) = ∇_R h_k(f_l(x)) · v_l^C<br># This measures how sensitive class k is to moving along concept C.<br># ===========================<br>TARGET_CLASS = 288  # ImageNet index for &#39;leopard&#39;<br>def directional_derivative_R(img, cav_vec, target_class):<br>    with tf.GradientTape() as tape:<br>        R = feat_model(img, training=False)   # R = f_l(x)<br>        tape.watch(R)<br>        logit = pred_layer(R)[:, target_class]  # h_k(f_l(x))<br>    grad_R = tape.gradient(logit,R).numpy().reshape(-1)   # ∇_R h_k(f_l(x))<br>    return float(np.dot(grad_R, cav_vec))                 # S_{C,k,l}(x)</pre><p><strong>Step 4 — Compute the TCAV Score</strong></p><p>For all test images, count how often the sensitivity S_{C,k,l}(x) is positive. The TCAV score is the fraction of such cases.</p><ul><li>High TCAV → The concept is important for the class.</li><li>Low TCAV → The concept has little or no influence.</li></ul><pre># ===========================<br># Step 4 — Compute the TCAV Score<br># TCAV_{C,k,l} = fraction of test inputs x ∈ X_k<br># where S_{C,k,l}(x) &gt; 0, i.e. moving along concept increases logit.<br># This gives the concept’s importance for class k.<br># ===========================<br>def tcav_score(loader, cav_vec, target_class):<br>    dds=[directional_derivative_R(img,cav_vec,target_class) for img,_ in loader]<br>    dds=np.array(dds,np.float32)<br>    return float((dds&gt;0).mean()), dds<br><br>tcav, dd = tcav_score(ldr_test, cav, TARGET_CLASS)<br>print(f&quot;TCAV (dalmatian→leopard idx={TARGET_CLASS}): {tcav:.3f}&quot;)<br>print(f&quot;DD stats: mean={dd.mean():.4f}, std={dd.std():.4f}, min={dd.min():.4f}, max={dd.max():.4f}&quot;)</pre><p>In conclusion, deep learning models don’t have to stay black boxes. By using Concept Activation Vectors and TCAV, we can interpret model decisions in terms of human concepts, making AI systems more transparent, trustworthy, and aligned with human values.</p><p>TCAV is designed with several key goals in mind:</p><ul><li><strong>Accessibility:</strong> It requires little to no ML expertise from the user.</li><li><strong>Customisation:</strong> It can adapt to any concept (e.g., gender) and is not limited to those considered during training.</li><li><strong>Plug-in readiness:</strong> It works without retraining or modifying the ML model.</li><li><strong>Global quantification:</strong> It provides explanations at the class level, offering a single quantitative measure rather than just input-specific insights.</li></ul><p>Taken together, these properties make TCAV not just a tool for understanding models, but a step toward making machine learning more accountable and usable for real-world, high-stakes applications.</p><h3>References:</h3><ul><li><a href="https://www.themoonlight.io/en/review/controlling-large-language-models-through-concept-activation-vectors">https://www.themoonlight.io/en/review/controlling-large-language-models-through-concept-activation-vectors</a></li><li><a href="https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf">https://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf</a></li><li><a href="https://towardsdatascience.com/tcav-interpretability-beyond-feature-attribution-79b4d3610b4d/">https://towardsdatascience.com/tcav-interpretability-beyond-feature-attribution-79b4d3610b4d/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da951fff4ece" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trustworthy Machine Learning: From Black-Box Models to Transparent and Responsible AI]]></title>
            <link>https://medium.com/@yalcinselcuk0/trustworthy-machine-learning-from-black-box-models-to-transparent-and-responsible-ai-453f8fa28be8?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/453f8fa28be8</guid>
            <category><![CDATA[interpretability]]></category>
            <category><![CDATA[trustworthy-ai]]></category>
            <category><![CDATA[xai]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Wed, 03 Sep 2025 15:22:29 GMT</pubDate>
            <atom:updated>2025-09-05T01:49:05.392Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Trustworthy AI</strong> refers to artificial intelligence systems that are <strong>explainable, fair, interpretable, robust, transparent, safe, and secure</strong>. These qualities are essential for building trust and confidence among stakeholders and end users, ensuring that AI solutions can be responsibly integrated into society.</p><p>Building on this idea, <strong>Trustworthy Machine Learning</strong> embeds these qualities directly into the development and deployment of models. Throughout the AI system’s life cycle, several key dimensions must be continuously addressed: <strong>human agency and oversight</strong> ensure that people remain in control; <strong>technical robustness and safety</strong> protect against errors and adversarial attacks; and <strong>privacy and data governance</strong> safeguard sensitive information. <strong>Transparency</strong> and <strong>accountability</strong> make model decisions understandable and ensure responsibility, while <strong>diversity, non-discrimination, and fairness</strong> prevent biased outcomes. Finally, <strong>societal and environmental well-being</strong> emphasises that AI should contribute positively beyond technical performance.</p><p><strong>Together</strong>, these <strong>parameters</strong> <strong>provide</strong> a framework for creating AI systems that are <strong>not</strong> <strong>only</strong> <strong>effective</strong> but <strong>also</strong> <strong>trusted</strong>, ethical, and aligned with human values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L1eTfLrbvJu8TXEMAOy4Xw.png" /><figcaption><strong>Seven Principles for Trustworthy Artificial Intelligence</strong></figcaption></figure><h4><strong>Seven Principles for Trustworthy Artificial Intelligence</strong></h4><p><strong>Human agency &amp; oversight: </strong>AI should <strong>never</strong> <strong>diminish</strong> <strong>human autonomy</strong> but instead empower people. <strong>Users</strong> <strong>must</strong> be able to <strong>understand</strong> <strong>how</strong> the <strong>system works</strong> and have <strong>the ability to interfere or challenge decisions when necessary.</strong></p><p><strong>Technical robustness &amp; safety: </strong>A trustworthy AI system <strong>must be secure</strong> <strong>against attacks</strong> and resilient to failures. It should have fallback mechanisms in case something goes wrong and deliver <strong>results</strong> that are <strong>accurate, reliable, and reproducible.</strong></p><p><strong>Privacy &amp; data governance: Protecting</strong> <strong>personal data</strong> is essential for trust. AI systems <strong>must</strong> <strong>ensure</strong> the quality and accuracy of the data they use and apply <strong>clear rules and protocols for data</strong> access and handling.</p><p><strong>Transparency: People</strong> have the <strong>right</strong> to know <strong>when</strong> they are <strong>interacting with an AI system</strong>. <strong>Decisions</strong> should be <strong>traceable</strong> and <strong>explainable</strong>, allowing humans to <strong>understand</strong> <strong>how outcomes are produced.</strong></p><p><strong>Diversity, non-discrimination &amp; fairness: </strong>AI should be <strong>designed</strong> to <strong>avoid</strong> <strong>unfair</strong> <strong>bias</strong> and <strong>discriminatory</strong> outcomes. It must be accessible to everyone, regardless of background, and developed with input from relevant stakeholders to ensure inclusiveness.</p><p><strong>Societal &amp; environmental well-being: </strong>AI must <strong>contribute</strong> <strong>positively to society </strong>while <strong>minimising</strong> <strong>harm</strong>. This includes <strong>supporting sustainability,</strong> <strong>promoting social benefits, </strong>and considering <strong>impacts on democracy</strong> and institutions.</p><p><strong>Accountability: </strong>AI systems need to be regularly evaluated through audits and transparent reporting. <strong>Negative outcomes must be addressed responsibly,</strong> and mechanisms for redress should be available to ensure fairness.</p><p>Building on these principles, it is important to distinguish between <strong>Artificial Intelligence (AI)</strong> in general and <strong>Explainable Artificial Intelligence (XAI)</strong> in particular. AI refers to the broader technology that powers advanced applications and enables data-driven decision-making, often through complex <strong>“black-box” models</strong> such as Deep Neural Networks (DNNs), which are highly accurate but difficult to interpret. The main <strong>limitation</strong> of such <strong>black-box </strong>approaches <strong>lies in their opacity</strong>: while they can deliver precise predictions, they <strong>provide</strong> little to <strong>no</strong> insight into the <strong>reasoning</strong> behind those outputs. This lack of transparency makes it challenging for humans to trust, validate, or question the system’s decisions.</p><p><strong>Explainable AI (XAI)</strong> directly addresses this limitation by developing methods and processes that make AI systems more <strong>transparent, interpretable, and trustworthy</strong>. Its aim is not only to show what decision an AI system makes, but also <strong>why</strong> and <strong>how</strong> it arrives at that decision. XAI employs a variety of techniques such as <strong>LIME, SHAP, or Grad-CAM</strong> to highlight feature importance, visualise model behaviour, and provide human-understandable explanations of outputs.</p><p>In short, while <strong>AI</strong> focuses on <strong>building intelligent systems</strong>, <strong>Explainable AI (XAI)</strong> <strong>provides</strong> <strong>methods</strong> and <strong>processes</strong> that make machine learning <strong>models</strong> <strong>transparent</strong> and <strong>understandable</strong> to <strong>humans</strong> by clarifying how results are produced and ensuring trust, fairness, and accountability in AI-driven decisions. For example, in the <strong>health</strong> domain, <strong>explainability</strong> allows <strong>doctors</strong> to <strong>understand</strong> <strong>why</strong> an <strong>AI system suggests a particular diagnosis or treatment</strong>, helping them verify results, build patient trust, and ensure safe medical decisions.</p><h3>Comparing White-box, Gray-box, and Black-box</h3><p>This diagram illustrates the spectrum from <strong>white-box models</strong>, which are highly interpretable but less accurate, to <strong>black-box models</strong>, which deliver strong performance but lack transparency, with <strong>gray-box models</strong> occupying the middle ground.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chEboGT4GC7lbG5F46Y1Zw.jpeg" /></figure><p><strong>White-box (Transparent Models):</strong> White-box models allow full inspection of their internal logic, making decisions easy to understand. They provide high explainability and interpretability but often at the cost of lower accuracy. Because of this trade-off, they are less commonly used in daily-life applications where high accuracy is required.</p><p><strong>Gray-box (Partially Transparent Models): Gray-box models</strong> offer partial transparency, enabling a moderate level of interpretation of the internal workings of the model while still maintaining significant accuracy. If carefully designed, they can be applied even in critical domains, since they balance performance and interpretability.</p><p><strong>Black-box (Opaque Models): Black-box models</strong>, such as deep learning systems, achieve high accuracy but lack transparency, making it difficult to understand their decision-making process. This low explainability makes them impractical for critical applications where trust and accountability are essential. To address this, Explainable AI (XAI) methods are employed to reveal decision mechanisms and enhance model transparency.</p><h4>The trade-off between performance and explainability in an AI model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*BpXdlJ9KC-9tPzZndcoeNA.png" /></figure><p>When <strong>explainability</strong> is <strong>unimportant</strong>, <strong>any</strong> <strong>high-performing</strong> <strong>model</strong> can be <strong>used</strong>; however, if <strong>explainability</strong> is <strong>crucial</strong>, one must decide whether to <strong>prioritise</strong> <strong>accuracy</strong> (<strong>opaque/black-box models</strong>) or interpretability (<strong>transparent/white-box models</strong>). The framework highlights how explanations can support better models, build trust, and integrate user concerns, while also outlining pathways to improve explanations through new methods, easier communication, and addressing emerging challenges.</p><h3><strong>Interpretability</strong></h3><p><strong>A natural extension of explainability is interpretability</strong>, which refers to understanding the <strong>underlying</strong> <strong>workings</strong> <strong>of an AI model</strong>. It involves <strong>making</strong> the <strong>intrinsic</strong> <strong>properties</strong> of a model, especially in Deep Learning, accessible, so that <strong>developers</strong> can delve into its decision-making process and <strong>see</strong> <strong>how</strong> <strong>inputs</strong> <strong>are transformed into outputs.</strong> While <strong>explainability</strong> is often <strong>aimed</strong> at <strong>helping</strong> end-<strong>users</strong> <strong>build trust by clarifying outcomes</strong>, <strong>interpretability</strong> <strong>focuses</strong> on the <strong>model’s</strong> <strong>internal</strong> <strong>structure</strong> and behaviour itself, providing an interface that <strong>opens</strong> <strong>a</strong> <strong>door</strong> <strong>into</strong> <strong>the</strong> <strong>“black box” </strong>for those with the necessary knowledge and skills. In this sense, interpretability forms the foundation upon which XAI builds to provide meaningful and trustworthy explanations of AI decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BbA8fL00BUgvDkNIEP3NQA.jpeg" /><figcaption><strong>Taxonomy of Interpretability Methods</strong></figcaption></figure><p><strong>By Design:</strong> These models can be <strong>understood directly by humans</strong>, without the need for extra explanation methods. Known as <strong>“white-box” or “glass-box” models</strong>, they are <strong>transparent and interpretable by design</strong>. This ensures that <strong>interpretability is built into the model’s structure</strong>, although the trade-off is that their <strong>simplicity may reduce accuracy compared</strong> to more complex <strong>“black-box” models</strong>. <strong>For instance</strong>, <strong>Decision Trees</strong> are inherently interpretable due to their <strong>rule-based, hierarchical structure</strong>, while <strong>Linear Regression</strong> with only a few features is also <strong>easy to interpret</strong>.</p><p><strong>Post-hoc: </strong>These methods are applied <strong>after training a complex, black-box model (e.g., deep neural networks, ensemble models)</strong>to provide explanations. Instead of directly accessing the model’s internal logic, they work by <strong>analysing inputs and outputs</strong> to generate interpretable insights. Often, this involves building a <strong>simpler surrogate model</strong> that approximates the black-box behaviour. For instance, <strong>LIME</strong> and <strong>SHAP</strong> explain individual predictions, while <strong>Grad-CAM</strong> produces visual explanations for Convolutional Neural Networks (CNNs).</p><p><strong>Model Agnostic:</strong> <strong>Model-agnostic interpretation methods</strong> do not depend on the internal structure of a machine learning model. Instead, they treat the model purely as a <strong>“black box”</strong> by observing only its <strong>inputs and outputs</strong>, without requiring access to its architecture, parameters, or training process. This flexibility makes them highly valuable, as they can be applied to <strong>any type of model</strong> — from linear classifiers to complex deep neural networks. Because they operate after the model has already been trained, model-agnostic approaches generally provide <strong>post-hoc interpretability</strong>, helping users understand predictions without altering the model itself. For instance, <strong>LIME</strong> and <strong>SHAP</strong> are widely used model-agnostic methods, capable of explaining predictions for both classification and regression tasks across diverse algorithms.</p><p><strong>Local explanation: </strong>Local methods are designed to clarify <strong>why a model made a specific prediction for a single instance</strong> rather than explaining the model as a whole. They focus on analysing the relationship between the input features of one data point and the corresponding output, which makes them especially useful in case-specific scenarios such as medical or financial decisions. By doing so, they provide <strong>fine-grained, individualised insights</strong> that help users understand the reasoning behind a particular outcome. For instance, if a medical AI predicts a high risk of cardiovascular disease for a given patient, a local explainer like <strong>LIME</strong> could highlight which factors, such as <strong>age, blood pressure, or cholesterol level</strong>, were most influential in that individual prediction.</p><p><strong>Global Explanation: </strong>Global methods aim to provide insights into the <strong>overall behaviour of a model across the entire dataset</strong>, rather than focusing on individual predictions. They highlight <strong>general patterns, feature relationships, and average influences</strong> that drive the model’s decisions, giving a <strong>holistic understanding</strong> of how the model operates. This makes them useful for identifying biases, dominant predictors, or trends that consistently affect outcomes. For instance, <strong>Partial Dependence Plots (PDPs)</strong> show how changing one or two features impacts the average prediction of a black-box model across its full range, while aggregated <strong>SHAP values</strong> reveal the overall contribution of each feature, positive or negative, to the target variable.</p><p><strong>Model-specific Explanation: Model-specific methods</strong> are designed for and <strong>limited to particular types of models</strong>, since they exploit the <strong>internal structure and unique properties</strong> of those architectures to generate explanations. This means they cannot be applied universally but are powerful when used in the right context. In fact, inherently interpretable models are themselves <strong>model-specific by definition</strong>, as their architecture directly provides transparency. For instance, <strong>Class Activation Maps (CAM)</strong> and its extensions, such as <strong>Grad-CAM</strong> are specific to <strong>Convolutional Neural Networks (CNNs)</strong>, leveraging convolutional layers to highlight the regions in an input image that most strongly contribute to a class prediction.</p><p><strong>In conclusion,</strong> <strong>Explainable AI (XAI)</strong> emerges as a crucial <strong>bridge between performance and transparency</strong> in machine learning. While white-box models offer high interpretability but limited accuracy, and black-box models achieve strong performance at the cost of opacity, gray-box approaches and XAI techniques provide a middle ground. By revealing decision-making mechanisms and enhancing interpretability, <strong>XAI not only improves model trustworthiness but also supports fairness, accountability, and safe adoption of AI in critical real-world applications</strong>.</p><p><strong>References:</strong></p><ul><li><a href="https://christophm.github.io/interpretable-ml-book/interpretability.html">https://christophm.github.io/interpretable-ml-book/interpretability.html</a></li><li><a href="https://www.ibm.com/think/topics/trustworthy-ai#:~:text=Trustworthy%20AI%20refers%20to%20artificial,among%20stakeholders%20and%20end%20users.">https://www.ibm.com/think/topics/trustworthy-ai#:~:text=Trustworthy%20AI%20refers%20to%20artificial,among%20stakeholders%20and%20end%20users.</a></li><li><a href="https://www.netapp.com/blog/explainable-ai/">https://www.netapp.com/blog/explainable-ai/</a></li><li><a href="https://www.sciencedirect.com/science/article/pii/S1566253523001148?via%3Dihub">https://www.sciencedirect.com/science/article/pii/S1566253523001148?via%3Dihub</a> no</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=453f8fa28be8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Entropy, KL Divergence, and Binary Cross-Entropy: An Information-Theoretic View of Loss]]></title>
            <link>https://medium.com/@yalcinselcuk0/entropy-kl-divergence-and-binary-cross-entropy-an-information-theoretic-view-of-loss-436d973ede71?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/436d973ede71</guid>
            <category><![CDATA[kl-divergence]]></category>
            <category><![CDATA[cross-entropy]]></category>
            <category><![CDATA[shannon-entropy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[binary-cross-entropy]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 23:45:38 GMT</pubDate>
            <atom:updated>2025-08-28T16:10:27.444Z</atom:updated>
            <content:encoded><![CDATA[<p>In the field of <strong>machine learning</strong>, <strong>loss functions</strong> are more than just mathematical tools; they are the language that models use to learn from data. A deep connection to <strong>information theory,</strong> particularly the concepts of <strong>entropy</strong>, <strong>cross-entropy</strong>, and <strong>KL divergence</strong>, is at the <strong>heart</strong> of many popular <strong>loss functions.</strong></p><p>This article will take you on <strong>a step-by-step journey</strong>, starting with the foundations of <strong>entropy</strong> <strong>as a measure of uncertainty</strong> and moving through <strong>KL divergence</strong> and <strong>cross-entropy</strong> before narrowing down to binary cross-entropy loss, which is used in classification tasks.</p><p>By the end, we’ll not only understand why <strong>binary cross-entropy</strong> is the <strong>standard choice</strong> for <strong>logistic regression</strong> and <strong>neural networks</strong>, but also how its <strong>roots</strong> in <strong>information</strong> <strong>theory</strong> give it a solid theoretical foundation.</p><p>Let’s start with the definition of a loss function and its relation with entropy.</p><p><strong>A loss function</strong> is a mathematical tool that measures the <strong>error</strong> between <strong>predictions</strong> and true values, guiding the model to <strong>improve its parameters</strong>. Its connection to information theory comes from the use of concepts like <strong>entropy (uncertainty)</strong>and <strong>KL divergence</strong>, which quantify how much information is lost when one probability distribution is used to approximate another.</p><h3>Shannon Entropy: Measuring Uncertainty</h3><p>Shannon Entropy measures the average amount of uncertainty (or information) in a probability distribution<strong><em> p</em></strong>. It tells us how unpredictable the outcome of a random variable is, and this can be expressed with the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*LhI2wnYepUlrzQEtTuY7hw.png" /><figcaption><strong>Shannon Entropy</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*h8hte4NbpC5VObX1ZRz-1A.png" /></figure><h3>Cross-Entropy and KL Divergence</h3><p>As explained above,<strong> Shannon Entropy H(p)</strong> measures <strong>the average uncertainty</strong> (or information) in <strong>the true distribution p</strong>. Building on this, <strong>Cross-Entropy</strong> <strong>H(p,q)</strong> measures the <strong>uncertainty</strong> we face when we use a <strong>model distribution</strong> <strong>q</strong> instead of <strong>the true distribution p</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*DqBICcwVMTEgtg5c2VBbAw.png" /><figcaption><strong>Cross Entropy</strong></figcaption></figure><p>The difference between cross-entropy and the entropy of the true distribution is captured by the <strong>Kullback–Leibler (KL) Divergence</strong>, defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*Emtvql17eYCJPZrt1b1fTg.png" /><figcaption><strong>KL Divergence</strong></figcaption></figure><p>Expanding the KL divergence gives us:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*N-f9NtXgCZI3bEs3PJWP5Q.png" /></figure><p>If we <strong>plug in the entropy and cross-entropy</strong> definitions, we obtain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*745kdfv7_bZlQESjOX-i3Q.png" /></figure><p>From this relation, we can <strong>write</strong> the <strong>cross-entropy</strong> as the <strong>sum of the entropy</strong> of the <strong>true distribution</strong> and the<strong> KL Divergence (</strong>the extra cost<strong>)</strong> between the true and model distributions. It measures how much information is <strong><em>lost</em></strong> when you use <strong>q</strong> to approximate <strong>p</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*TYXPfXYR-uTESeteIcSiwQ.png" /></figure><p><strong>As you might guess, the entropy term H(p)</strong> is determined solely by the data and <strong>does not depend on the model</strong>, meaning the <strong>model</strong> c<strong>annot change it.</strong> For this reason, when we <strong>minimise</strong> <strong>the cross-entropy H(p,q)</strong>, we are effectively <strong>minimising</strong> <strong>the Kullback-Leibler Divergence(the extra cost)</strong>. In practice, <strong>reducing cross-entropy</strong> directly <strong>reduces KL Divergence,</strong> which brings the model’s predicted distribution closer to the true distribution <strong>p</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*uPxgkRxIutrQlvpxt0Tg5w.png" /></figure><p>As shown in the graph, <strong>KL Divergence</strong> measures the<strong> gap between the true distribution p(x)</strong> <strong>(in red) and the model distribution q(x) (in blue). Minimising</strong> cross-entropy therefore <strong>reduces this gap,</strong> making the two distributions <strong>overlap more closely.</strong></p><h3>Binary Cross-Entropy (BCE)</h3><p><strong>Binary Cross-Entropy</strong> (log loss) is a <strong>loss function</strong> used in binary classification problems and is directly <strong>derived</strong> from the <strong>concept of entropy.</strong> While entropy measures the inherent uncertainty in the true distribution of labels, binary cross-entropy quantifies how much additional uncertainty is introduced when using the model’s predicted probabilities instead.</p><p>In the case of <strong>binary classification</strong>, the true distribution is represented by <strong>the observed label y</strong>, and <strong>the model predicts a probability y_hat </strong>for the positive class. The cross-entropy, therefore, takes the form</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*hmg_1dS9A-7S6QozBIbN-g.png" /></figure><p>Replacing the true distribution p with the observed label y, we obtain the <strong>Binary Cross-Entropy (BCE) loss function</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*taP9Ui6sHSx4jed5jY2lkg.png" /></figure><p>Thus, <strong>BCE</strong> is <strong>not</strong> just an <strong>arbitrary loss function</strong> but directly <strong>arises</strong> <strong>from</strong> the general definition of <strong>cross-entropy</strong>. <strong>Minimising BCE </strong>is equivalent to <strong>minimising the KL Divergence</strong> between the true label distribution and the predicted probabilities, thereby <strong>bringing</strong> the <strong>model’s predictions y_hat closer to the true distribution.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20Al-mIvN0gt6v1ojVgZ8w.png" /></figure><p>As shown in the figure above, BCE captures how the model behaves in binary classification. <strong>When</strong> the <strong>predicted probability (y_hat)</strong> <strong>aligns</strong> with <strong>the true label</strong>, the <strong>loss</strong> remains <strong>close to zero</strong>. But as the model becomes confidently wrong, the loss shoots up towards infinity, heavily penalising such mistakes. This directly reflects its connection to KL Divergence: <strong>the larger the gap between the true and predicted distributions, the higher the loss.</strong></p><h3>Conclusion</h3><p><strong>In conclusion, loss functions</strong> are deeply rooted in information theory: we started from entropy as a measure of uncertainty, extended it to cross-entropy and KL Divergence, and showed how these concepts ground modern approaches to measuring error. Binary Cross-Entropy naturally arises in classification, and through gradients and optimisation, minimising it aligns model predictions with the true distribution.</p><p>References:</p><ul><li><a href="https://medium.com/@yalcinselcuk0/information-processing-and-communication-a-beginners-guide-to-information-processing-7d2f7e60f646">https://medium.com/@yalcinselcuk0/information-processing-and-communication-a-beginners-guide-to-information-processing-7d2f7e60f646</a></li><li><a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=436d973ede71" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Information Processing and Communication: A Beginner’s Guide to Information Processing]]></title>
            <link>https://medium.com/@yalcinselcuk0/information-processing-and-communication-a-beginners-guide-to-information-processing-7d2f7e60f646?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d2f7e60f646</guid>
            <category><![CDATA[entropy]]></category>
            <category><![CDATA[shannon-entropy]]></category>
            <category><![CDATA[information-processing]]></category>
            <category><![CDATA[joint-entropy]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Tue, 05 Aug 2025 19:32:44 GMT</pubDate>
            <atom:updated>2025-08-26T00:08:31.994Z</atom:updated>
            <content:encoded><![CDATA[<p>In a world flooded with information from voice assistants recognising our speech to algorithms sorting our photos, how do machines make sense of the data they receive? Whether it’s the rhythm of spoken words, the shape of a handwritten letter, or the colour of a traffic light, every piece of data must first be measured, processed, and interpreted. This process, known as <strong>information processing</strong>, lies at the heart of modern artificial intelligence and machine learning systems. Let’s begin with the core base of Information Processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8PHhqqitqoL_D_Bwrpo-mQ.png" /></figure><h4>Information</h4><p><strong>Information</strong> measures <strong>how much we learn from an event</strong> and how <strong>surprised</strong> we are by its outcome. If something <strong>expected happens</strong>, we gain <strong>little to no new information.</strong> However, when something <strong>unlikely occurs</strong>, we receive a great deal of information.</p><p><strong>Information</strong> can <strong>appear</strong> in many formats: <strong>text</strong>, <strong>sound</strong>, <strong>image, signal</strong> or <strong>video</strong>. A song, a news article, or a photo on your phone all carry data, just in a different structure. But the <strong>essence</strong> is the same: <strong>information</strong> <strong>reduces</strong> <strong>uncertainty</strong>.</p><p><strong>Imagine</strong> you’re <strong>trying</strong> to guess the <strong>weather</strong> tomorrow. If someone tells you, “There will be weather,” that doesn’t help. But if they say, “It will <strong>snow</strong> in <strong>August</strong>,” that’s <strong>unexpected</strong> and <strong>contains</strong> a <strong>lot of information, </strong>especially in a place where it rarely snows. The <strong>more</strong> <strong>surprising</strong> the outcome, the <strong>more</strong> <strong>information</strong> it carries.</p><p>In 1948, <strong>Claude</strong> <strong>Shannon</strong> <strong>formalised</strong> this idea through the <strong>concept</strong> of <strong>information entropy</strong>, giving us tools to quantify uncertainty in data. His theory underpins everything from text compression and file sharing to how modern devices interpret sounds, images, and messages.</p><h4>Communication as Information Transfer</h4><p>In any communication system, information follows a series of steps: it begins as an idea in the sender’s mind, gets <strong>encoded</strong> into words or signals, is <strong>transmitted</strong> through a chosen channel, and finally gets <strong>decoded</strong> by the receiver. This process ensures that the message is understood and can be responded to, completing the communication loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I465MXVeRW0peJ5qyVzC0Q.png" /></figure><p><strong>Encoding</strong></p><p><strong>Encoding</strong> refers to <strong>the process of creating a message using symbols</strong> that others can understand. These symbols can be <strong>words</strong>, <strong>images</strong>, <strong>gestures</strong>, or <strong>sounds</strong>. The person sending the message (the <strong>sender</strong>) chooses these based on what they think the other person (the <strong>receiver</strong>) will understand.</p><p>The encoding of a message depends on what the sender wants to convey and how they want the receiver to feel or react.</p><p><strong>Encoding</strong> is, in other words, <strong>the process of converting data into a format that computers can understand</strong>, typically <strong>binary code</strong>. The simplest way to do this is to assign each character, symbol, or instruction a numeric value (usually in decimal), and then convert that value into <strong>binary</strong>, <strong>octal</strong>, or <strong>hexadecimal</strong> — the number systems used by computers to process and store information.</p><h4>Transmission</h4><p><strong>Transmission</strong> is the step where <strong>the encoded message is sent from the sender to the receiver through a specific channel. </strong>This channel can be spoken words, printed text, phone lines, radio waves, or digital networks.</p><p>The <strong>quality</strong> and <strong>clarity</strong> of the <strong>transmission</strong> depend on the medium and can be <strong>affected</strong> <strong>by noise, distance, or interference.</strong></p><h4>Decoding</h4><p><strong>Decoding</strong> is the process the <strong>receiver</strong> uses to <strong>interpret</strong> the <strong>message</strong>. The <strong>receiver</strong> <strong>takes</strong> the <strong>symbols</strong> (words, gestures, sounds) and <strong>tries</strong> to <strong>understand</strong> the <strong>meaning</strong> behind them, based on their own experiences, context, and language.</p><p>In brief, decoding is the <strong>reverse</strong> <strong>of</strong> <strong>encoding</strong>. It means <strong>converting</strong> <strong>binary</strong> or <strong>encoded</strong> <strong>data</strong> <strong>back</strong> <strong>into a readable or usable format.</strong> For example, when you receive a file or a video, your computer decodes the compressed or encoded format (like .zip, .mp4) so you can read or watch it.</p><h4>Information Processing</h4><p>As we see in the diagram, the process starts with raw information like text, sound so on. First, it gets <strong>encoded</strong>, then it’s <strong>transmitted</strong> (maybe through the internet, a phone, or even a letter), and finally, it gets <strong>decoded</strong> on the other side so the receiver can understand it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hOsu6-dRP0AwXJ-ZbFfi0A.png" /></figure><p><strong>For</strong> <strong>example</strong>, in the <strong>heart</strong> <strong>disease</strong> <strong>detection</strong> <strong>system</strong> shown in the diagram, the <strong>process</strong> <strong>begins</strong> <strong>with</strong> <strong>raw clinical data</strong> such as patient <strong>symptoms</strong> or <strong>medical</strong> <strong>test</strong> <strong>results</strong>. This data is first <strong>preprocessed,</strong> and the <strong>encoded </strong>noise is removed, and <strong>only</strong> the <strong>most</strong> <strong>important</strong> <strong>features</strong> are <strong>selected</strong>. Then, the <strong>processed</strong> <strong>data</strong> is <strong>passed</strong> <strong>through</strong> a <strong>machine learning pipeline</strong> (similar to transmission), where <strong>feature extraction and oversampling techniques help prepare it for analysis</strong>. <strong>Finally</strong>, the data is <strong>decoded</strong> by a <strong>classification</strong> <strong>model</strong>, which <strong>decides</strong> <strong>whether</strong> <strong>heart</strong> <strong>disease is present or not</strong>, <strong>just like a receiver interpreting the original message.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lDccGc5jT8qN4kChF-6QaQ.png" /></figure><h4>Entropy and Measuring Information</h4><p>Before any information can be processed, transmitted, or interpreted, as seen in systems like <strong>heart disease detection</strong>, we must first understand what information is and how much of it is present in a message. This is where <strong>entropy</strong> comes in. <strong>In information theory</strong>, <strong>entropy</strong> <strong>measures the amount of uncertainty </strong>or unpredictability <strong>in a probability distribution</strong>. It tells us how much “<strong>surprise</strong>” or “<strong>information</strong>” is associated with the possible outcomes of a random variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lhOzgwwsMNam2kwuvZZ3CQ.png" /><figcaption><strong>High and Low Entropy</strong></figcaption></figure><p>When the <strong>probabilities</strong> of <strong>all outcomes are spread out</strong> evenly, like in a uniform distribution, the <strong>entropy</strong> is <strong>high</strong>, because it’s <strong>hard</strong> <strong>to</strong> <strong>predict</strong> which outcome will occur. In contrast,<strong> if one outcome is much more likely than the others</strong>, the <strong>entropy is low</strong>, meaning<strong> the system</strong> is more <strong>predictable</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VwrGJGwc3OKK5DzmgoZAcw.png" /></figure><p>Entropy is measured using the <strong>Shannon entropy formula</strong>, which quantifies the <strong>average amount of information</strong> (or <strong>uncertainty</strong>) in <strong>a probability distribution</strong>. If we have<strong> a random variable X</strong> that can take different values, and <strong>we know the probability of each value</strong> (given by <strong>pX​(x)</strong>), then the self-information of seeing the outcome <strong>x</strong> is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*pG5HVqMQXEKGaWHdJrESuQ.png" /><figcaption><strong>Self Information</strong></figcaption></figure><p>As we mentioned before, entropy is the <strong>average amount of information (expected information content) </strong>in <strong>a probability distribution</strong>. Therefore, if we multiply each possible outcome by its information content and sum them, we obtain the <strong>entropy</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*qP0nlFnfuQaZ-7PO6If-aA.png" /><figcaption><strong>Shannon Entropy</strong></figcaption></figure><h3>Types of Entropy</h3><p>As we mentioned before, <strong>entropy</strong> measures <strong>the average amount of information in a probability distribution</strong>. However, depending on the context and the type of data we’re dealing with, <strong>entropy can take different forms</strong>. For example, when analysing <strong>the combined uncertainty of two random variables</strong>, we use <strong>joint entropy</strong>. When we want to measure the <strong>uncertainty of one variable given that we know another</strong>, we use <strong>conditional entropy</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*CO-H5bx2uXufrq3eZ4nARQ.png" /><figcaption><strong>Types of Entropy</strong></figcaption></figure><h4><strong>Joint Entropy H(X,Y)</strong></h4><p>The <strong>joint entropy</strong> measures the <strong>total uncertainty or randomness</strong> associated with both variables <strong>taken together</strong>. Just like we define entropy for a single random variable, we can extend this concept to a <strong>pair of random variables</strong> <strong>(X, Y)</strong>. The <strong>joint entropy</strong> is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*lpf1Y5vg8jA9cnSqjNosfQ.png" /></figure><p>If X and Y are discrete random variables, the joint entropy becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*PP4P8rpKA6TCvJrCsTOJmw.png" /></figure><p>If X and Y are continuous, we define the <strong>differential joint entropy</strong> as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*zDJWgqQkoCN1um2h8YSmRw.png" /></figure><p>Let’s consider two extreme scenarios:</p><p>If <strong>X = Y</strong> (i.e., perfectly correlated), then knowing one gives complete information about the other. In this case entropy of X is the same as the entropy of Y:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*Lwarq9IQRGud4P095Jwjzw.png" /></figure><p>If X and Y are <strong>independent</strong>, there is <strong>no shared information</strong>, and the joint entropy equals the <strong>sum of individual entropies(uncertainty)</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*HAdUpk67mLQWnEpfBdXB7w.png" /><figcaption><strong>Independent events</strong></figcaption></figure><p>Another important fact to know is that for any two random variables X and Y, the <strong>joint entropy H(X, Y) always satisfies the following inequality</strong>, as it captures the combined uncertainty of X and Y, ranging from <strong>full overlap</strong> (perfect dependence) to <strong>no overlap</strong> (complete independence):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*3_pMlMCYtdm7DWVuOCkh4A.png" /></figure><p>So, <strong>Joint Entropy quantifies </strong>the <strong>total information</strong> needed to <strong>describe</strong> both <strong>X and Y</strong></p><h4><strong>Conditional Entropy (H(X|Y)</strong></h4><p>In information theory, <strong>conditional entropy</strong> measures how much <strong>uncertainty remains about a variable Y</strong> when we already <strong>know the value of another variable X</strong>. <strong>It tells us how much more information we still need to fully describe Y, given that X is known.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*AkClDepzCDGLtySOyMotAg.png" /><figcaption><strong>Conditional Entropy</strong></figcaption></figure><h4>Mutual Information</h4><p>Mutual information intuitively measures how much <strong>information is shared between two random variables</strong>,<strong> X and Y</strong>. It tells us <strong>how much knowing one variable reduces the uncertainty about the other</strong>. We calculate mutual information as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*V_F9Y_ymK3DIL0sZ_MeZ5Q.png" /><figcaption><strong>Mutual Information</strong></figcaption></figure><p>If you take a closer look at the Venn diagram aga, you’ll notice that the <strong>mutual information</strong> of the shared part between X and Y can be expressed in <strong>several different but equivalent ways</strong>.</p><p>Each of these forms offers a slightly different perspective on how much information is shared between the two variables and how uncertainty is reduced when one variable is known.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*fTURewN5sjMyRPYCJIuwhw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2xXlkOCE6THaDo2NQex3hA.png" /></figure><p>These different expressions highlight that <strong>mutual information</strong> is not just about overlap; it also tells us how much uncertainty is removed, or how much redundancy exists between two sources of information.</p><p>Another important fact about mutual information is that <strong>if X and Y are independent</strong>, then <strong>the mutual information</strong> between them is <strong>zero</strong>. In this case, knowing the value of X gives you no insight into Y, and vice versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzbtyeW0eVVviia339mB6w.png" /><figcaption><strong>Mutual Information for Independent Events</strong></figcaption></figure><p><strong>Proving that mutual information is zero for independent events(x,y)</strong></p><p>If two variables (x,y) are independent, the joint probability would be calculated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*BWjIdLRBbGvbO4f5y3HZHQ.png" /></figure><p>Let&#39;s plug our joint probability into our mutual information equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vR3WO6U9_6U98lIM-nm4Cg.png" /></figure><p>As we see above, if there is no connection between events, our <strong>independent</strong> <strong>mutual</strong> <strong>information</strong> will be <strong>zero.</strong></p><p><strong>For dependent events</strong>, <strong>the joint probability p(x, y)</strong> does <strong>not equal</strong> the <strong>product of the marginals p(x) times p(y)</strong>. This means there is <strong>shared information between X and Y</strong>, and mutual information becomes <strong>greater</strong> <strong>than zero</strong>. In such cases, knowing one variable <strong>reduces the uncertainty</strong> of the other, which is exactly what mutual information captures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*fTURewN5sjMyRPYCJIuwhw.png" /><figcaption><strong>Mutual information’s relationship with joint entropy and conditional entropy</strong></figcaption></figure><p><strong>Prove mutual information for dependent events.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*5YliiMQD-n9yhr_C1Y_9QA.png" /></figure><p>First, as a reminder, we write the joint probability for dependent variables as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*OUeTDGvlbViPKfL-YD1jHA.png" /><figcaption><strong>Joint Probability for Dependent Variables</strong></figcaption></figure><p>Second, substitute this into our mutual information formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_lubRmc4sGb1pM3O72bAkA.png" /></figure><p>And then expand our logarithmic term of the equation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eb_qWgsmPJ8pCgxhrBPjAw.png" /></figure><p>Remember that we can write the total entropy of X and the conditional entropy as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5deEwVvflfnKdGGICIorA.png" /></figure><p>If we substitute entropy into our equation, we can end up with the following equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*bPWysYaVytuolp6eaTZKqw.png" /><figcaption><strong>Mutual</strong> <strong>Information</strong></figcaption></figure><p>The idea of measuring uncertainty is not just theoretical; it’s central to many real-world applications, especially in machine learning.</p><p>In machine learning, entropy is a fundamental concept used to measure uncertainty, make informed decisions, and guide model learning. It plays a key role in various areas, such as<strong> Decision tree algorithms</strong> or <strong>classification models</strong>.</p><p><strong>In decision tree learning</strong>, entropy is used to evaluate how mixed a dataset is at each node, allowing the algorithm to choose the attribute that best splits the data and reduces <strong>entropy</strong> (uncertainty).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/0*Ni2HCOnGD3Ai9Jr9.png" /><figcaption><strong>Desicion Tree</strong></figcaption></figure><p><strong>In classification</strong> tasks such as logistic regression and neural networks, entropy appears in the form of <strong>cross-entropy loss</strong>, which measures the difference between the <strong>predicted probabilities</strong> and <strong>the actual labels</strong>. By minimising this loss, the model becomes better at making accurate predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2B9lks5aN3OxVjghgkwxDA.png" /></figure><p>Overall, entropy helps guide models toward more certainty, better decisions, and improved performance.</p><p>References:</p><ul><li><a href="https://web.njit.edu/~lipuma/352comproc/comproc.htm">https://web.njit.edu/~lipuma/352comproc/comproc.htm</a></li><li>https://www.sciencedirect.com/topics/engineering/shannon-entropy#:~:text=Shannon%20entropy%20is%20defined%20as,of%20obtaining%20a%20value%20xi.</li><li><a href="https://www.geeksforgeeks.org/computer-networks/encoding-and-decoding-in-communication-process/">https://www.geeksforgeeks.org/computer-networks/encoding-and-decoding-in-communication-process/</a></li><li><a href="https://noone.io/blog/blockchain-security-random-numbers-entropy-cryptography/">https://noone.io/blog/blockchain-security-random-numbers-entropy-cryptography/</a></li><li><a href="https://www.mdpi.com/1099-4300/21/7/691">https://www.mdpi.com/1099-4300/21/7/691</a></li><li><a href="https://medium.com/data-science/decision-tree-ba64f977f7c3">https://medium.com/data-science/decision-tree-ba64f977f7c3</a></li><li><a href="https://www.youtube.com/watch?v=i78buAHKXpI">https://www.youtube.com/watch?v=i78buAHKXpI</a></li><li><a href="https://bernstein-network.de/wp-content/uploads/2021/02/18_Lecture-18-Application-to-neurons.pdf">https://bernstein-network.de/wp-content/uploads/2021/02/18_Lecture-18-Application-to-neurons.pdf</a></li><li><a href="https://pub.towardsai.net/quantifying-gain-fb5a9c18cdac">https://pub.towardsai.net/quantifying-gain-fb5a9c18cdac</a></li><li><a href="https://sccskp.lovestoblog.com/ics-1/data-communication/?i=1">https://sccskp.lovestoblog.com/ics-1/data-communication/?i=1</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d2f7e60f646" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Algebra Essentials for Machine Learning]]></title>
            <link>https://medium.com/@yalcinselcuk0/linear-algebra-essentials-for-machine-learning-28ea16d81dd3?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/28ea16d81dd3</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[l1-regularization]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 18:55:34 GMT</pubDate>
            <atom:updated>2025-07-31T16:52:45.111Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>Every time you adjust the brightness of a photo, rotate a 3D model, or train a machine learning model to recognise a cat in an image, you are, whether you realize it or not, using the language of <strong>linear algebra</strong>.</p><p>It powers the inner workings of modern AI systems, from <strong>calculating similarity scores</strong> in recommendation engines to <strong>computing gradients</strong> in <strong>neural networks</strong> and so on.</p><p>In the context of <strong>machine learning</strong>, linear algebra is more than just a mathematical theory; it’s a practical framework for representing data, modelling relationships,<strong> optimising parameters</strong>, and <strong>scaling algorithms</strong> efficiently. Whether you’re <strong>computing the distance</strong> between two data points, <strong>projecting high-dimensional data</strong> into <strong>lower dimensions</strong>, or <strong>minimising a loss function</strong>, you require concepts like <strong>vectors, norms, vector projection, dot products, and gradients</strong>.</p><p>In this post, we’ll tackle the most essential linear algebra concepts that frequently appear in machine learning workflows. We won’t just define them; we’ll break them down, build intuition, and demonstrate how they’re applied in practice. We’ll examine the following topics:</p><ul><li>Vector operations like <strong>dot</strong>, <strong>cross</strong>, and <strong>Hadamard products</strong></li><li><strong>Norms</strong> and how they measure <strong>distances</strong> in feature space</li><li>The <strong>role of projections</strong> in <strong>dimension reduction and optimisation</strong></li><li><strong>Gradient</strong> and how they help <strong>models learn</strong></li><li>Quadratic forms and matrix calculus behind optimization routines</li></ul><p>Let&#39;s begin with vector operations.</p><h3><strong>Vector Operations</strong></h3><h4><strong>Dot Product (Inner Product)</strong></h4><p><strong>The dot product</strong> is a way to <strong>combine</strong> two <strong>vectors</strong> or matrices and get a single number (<strong>scalar</strong>) as the result. You do this by multiplying the matching numbers in each vector or matrix, and then adding all those results together. In some resources, it is also referred to as the <strong>inner product </strong>or <strong>scalar product</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*qgsbILeMbwxlcspPRaIGqw.png" /></figure><p><strong>For example, the dot product</strong> is used in N<strong>atural Language Processing (NLP)</strong> through <strong>cosine similarity</strong> to measure the <strong>similarity between two text embeddings</strong> in terms of meaning. In <strong>neural networks</strong>, the dot product plays a crucial role during the <strong>forward pass</strong>, where it <strong>computes</strong> weighted <strong>sums between inputs and weights</strong>. In terms of <strong>Principal Component Analysis (PCA),</strong> &quot;dot product&quot; helps us to <strong>project high-dimensional data onto principal components</strong>, enabling effective <strong>dimension reduction.</strong></p><h4>Tensor (Outer) Product</h4><p>A tensor is a generalisation of scalars, vectors, and matrices that can have any number of dimensions. In other words, the <strong>tensor product</strong> is a mathematical operation that combines two or more vector spaces into a new, larger space that captures all possible <strong>multilinear interactions</strong> between them. We calculate the tensor product as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*6Gr1wy6K1TMufSdWDRECxQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*dp4nm_W9MBwPw524_0HAjA.png" /><figcaption><strong>Tensor Product</strong></figcaption></figure><p>For instance, in compositional semantics within Natural Language Processing (NLP), <strong>tensor</strong> <strong>products</strong> are used to <strong>represent the interaction between words in a sentence.</strong> Take, for example, the sentence <strong>“John loves Mary”</strong>: instead of representing each word in isolation, we compute the tensor product between the subject (“John”) and the object (“Mary”) to <strong>capture all pairwise interactions between their semantic features</strong>. This combined representation is then transformed via a relation-specific tensor (e.g., “loves”) that models the meaning of the verb. The goal is to <strong>encode</strong> the full relational meaning of the sentence as a structured, high-dimensional representation, enabling more accurate inference and reasoning in downstream language understanding tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*IGIfcgin6dfA7G4fs1Kamw.png" /></figure><h4><strong>Cross Product</strong></h4><p>The <strong>cross product </strong>is a vector operation applied to<strong> two 3-dimensional vectors</strong>, which results in a new vector that is <strong>perpendicular</strong> (orthogonal) to both of the original vectors.</p><p><strong>Given two 3D vectors:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*M_4c78Osslt-aTMeR4xYMw.png" /></figure><p>We can operate the cross product as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*8sK6kqM7D1dMbLiuio83OA.png" /><figcaption><strong>Cross Product</strong></figcaption></figure><p><strong>Hadamard product (Element-wise multiplication)</strong></p><p><strong>Hadamard product</strong>, also referred to as <strong>element-wise multiplication</strong>, involves <strong>multiplying</strong> corresponding elements of two <strong>matrices</strong>. Dissimilar like the <strong>dot product</strong> (<strong>scalar)</strong>, the result of the <strong>Hadamard product</strong> is a vector or matrix with t<strong>he same dimensions as the input. Product of two matrices by Hadamard Product:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*BsK69L1XUSGXDptq06uJDQ.png" /><figcaption><strong>Hadamard Product</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*1AgrH06NRtql__IhkbbbTw.png" /></figure><p><strong>For example</strong>, in <a href="https://medium.com/@yalcinselcuk0/thrustworthy-machine-learning-gradient-decomposition-based-xai-gradcam-8d9b61dc6ffe">the explainable AI technique Grad-CAM</a>, <strong>element-wise multiplication</strong> is used to <strong>highlight important regions </strong>of an image by <strong>combining feature maps with their corresponding importance weights. </strong>As shown below, <strong>the activation maps are multiplied element-wise</strong> to generate <strong>the final class-specific heatmap.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s1hNSKWZ1-h0woXM.jpg" /><figcaption><strong>GradCAM</strong></figcaption></figure><h3>Norm</h3><p>Norm is the function that <strong>measures</strong> the <strong>length</strong> of a <strong>vector</strong>. It always gives a similar result to how we measure distance from the origin.</p><h4><strong>L1 Norm (Manhattan norm)</strong></h4><p><strong>The L1 norm</strong>, also known as the <strong>Manhattan norm</strong> or <strong>Taxicab norm</strong>, measures the <strong>magnitude</strong> (length)of a vector by <strong>summing the absolute values of its components</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*TQ7Uds0gONYf2EsbPouCJw.png" /></figure><h4><strong>L2 Norm (Euclidean norm)</strong></h4><p><strong>The L2 norm</strong>, also known as the <strong>Euclidean norm</strong>, is calculated as the <strong>square root of the sum of the squared components of a vector</strong>. It represents the <strong>straight-line (or Euclidean) distance</strong> from the origin to the point in space defined by the vector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*bhzdCC_iFEGwRHSwq3YDfg.png" /></figure><h4><strong>L∞ Norm (Max norm)</strong></h4><p>The<strong> L∞ </strong>norm, also referred to as the <strong>max norm</strong>, of a vector is <strong>the largest absolute value </strong>of its components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*rvaCgF5MShK8u0KHWbJQtw.png" /></figure><p>For example, <strong>in adversarial training </strong>for cyber-physical systems, the L∞ norm is used to generate worst-case perturbations that are small but highly effective.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*4HiYNTUgDthfkrIdtbK9lQ.png" /></figure><p>As seen above, the training process <strong>minimises</strong> the expected loss under the <strong>strongest possible perturbation within an -bounded region</strong>. This leads to models that remain robust even when attacked, ensuring safer decision-making in real-world CPS environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IqlOXO1qGqtb6HfTg2ag4Q.png" /></figure><h3><strong>Distance</strong></h3><p><strong>Distance</strong> in mathematics refers to <strong>how far apart two points are</strong> from each other. It&#39;s always a <strong>non-negative</strong> number.</p><h4><strong>Manhattan Distance (L1 Norm)</strong></h4><p><strong>The Manhattan distance </strong>is a way to calculate the distance between two points in a <strong>grid-like path</strong>. The name <strong>originates</strong> from <strong>the</strong> <strong>streets of Manhattan</strong>, where you can’t move diagonally but must travel along a grid of horizontal and vertical streets. To go from one point to another, you add up the total number of blocks (or units) you move horizontally and vertically. This total is the Manhattan distance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*kbtwwtwgEUQ0-03IF1FKwg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BRT0VdNLVPec5GIF.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*AxWsajSaVKaXRkFlQbL1Hg.png" /></figure><h4><strong>Euclidean Distance</strong></h4><p><strong>Euclidean distance</strong> is the straight-line distance between two points in Euclidean space; it&#39;s just <strong>like using a ruler to measure the shortest path from one point to another.</strong> It’s calculated by measuring <strong>the length of the line</strong> segment connecting the two points, making it the most direct and intuitive way to define distance in geometry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*C5AMowy4ZEnY-yk7tx60fQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*GiWetxk56dByPV3JVnbZDA.png" /></figure><h4><strong>Cosine Distance and Cosine Similarity</strong></h4><p><strong>Cosine similarity</strong> measures how <strong>aligned</strong> two vectors are by calculating the <strong>cosine of the angle</strong> between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*DdQ4lQbQ3OY_GRzOvXRSxA.png" /></figure><p>The result of <strong>cosine similarity</strong> is in the <strong>range</strong> of <strong>-1 and 1</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmzZ3HybapBWoWLk6D6Ifw.png" /></figure><p><strong>Cosine distance measures how different two vectors are in terms of direction. </strong>It is based on the angle between the vectors and ignores their magnitude.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*S0ck1tppT770KiGdKcoW7A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*s4plZ4L3g5bWijvix2lRGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CaVcVj-AEIBXPqeJ.png" /></figure><p><strong>In machine learning</strong>, <strong>cosine similarity</strong> is widely <strong>used</strong> to determine <strong>how closely two vectors point in the same direction</strong>, regardless of their magnitude.</p><p>In the instance of <strong>information retrieval</strong>, it matches user queries with the most relevant documents. In <strong>document clustering</strong>, it groups texts based on <strong>similarity</strong> in content. In <strong>text classification</strong>, it helps assign documents to predefined categories. In <strong>recommender systems</strong>, it suggests new items by comparing them to the user’s past preferences. In <strong>image processing</strong>, it <strong>compares</strong> <strong>visual features</strong> to <strong>identify similar</strong> images, focusing on feature orientation rather than raw pixel values.</p><h4>Vector Projection</h4><p>Vector projection represents how much one vector “acts in the direction of” another. It is essentially the <strong>shadow</strong> <strong>of a vector onto a second vector</strong>, capturing its influence in that specific direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*0pCLryAnaAsfmS_G.png" /></figure><p>The <strong>scalar projection</strong> (length) of a⃗ onto b⃗ is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*VIqgWWSTirabpEnEMOVB-Q.png" /></figure><p><strong>Lets derive vector projection formula step by step</strong></p><p>Recall the dot production definition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*RI7FUwKPotupICIm9dMyaw.png" /><figcaption><strong>Dot product of two vectors</strong></figcaption></figure><p>So we can solve for cos⁡(θ):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*6UK4BUnCOE_Z2Z4LVLVfaA.png" /></figure><p>But we want a <strong>vector</strong> in the direction of <strong>b⃗,</strong> scaled by how much <strong>a⃗ </strong>aligns with <strong>b⃗.</strong> The <strong>unit vector</strong> in the direction of <strong>b⃗</strong> is, which gives us direction of projection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*GBhgDZEWEKs8YGW-AidsPA.png" /><figcaption><strong>Unit Vector</strong></figcaption></figure><p>If we sum all the <strong>components</strong> of the <strong>scalar projections</strong> and <strong>multiply</strong> the result by the<strong> unit vector</strong>, we arrive at the final formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*gop4_yVppg6cMvZtPt8B4w.png" /><figcaption><strong>Vector Projection of a⃗ onto b⃗</strong></figcaption></figure><p><strong>For instance</strong>, in dimension reduction, <strong>Principal Component Analysis (PCA)</strong> uses <strong>vector projection</strong> to <strong>transform high-dimensional</strong> data into a <strong>lower-dimensional</strong> space, aiming to <strong>preserve as much variance as possible.</strong> This is done by <strong>projecting</strong> <strong>each data</strong> point onto <strong>new axes called</strong>, <strong>principal components</strong> that <strong>capture</strong> directions of <strong>maximum variability</strong>. <strong>The goal is to minimize the difference between the original high-dimensional data and its lower-dimensional representation</strong>, often measured by <strong>squared Euclidean distance</strong> ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/0*BHMo5JwbJrFlBz7h.png" /><figcaption><strong>Principal Component Analysis</strong></figcaption></figure><p>As illustrated in the image, projecting 3D data onto a 2D plane makes the <strong>clusters (e.g., red, green, blue)</strong> more interpretable. In contrast, in the original 3D space, these clusters might overlap and be harder to separate.</p><p><strong>Gradients of Linear Functions</strong></p><p>A <strong>gradient</strong> in vector calculus is similar to the slope but applies when you have three or more variables. It becomes the vector of <strong>partial derivatives for all independent variables </strong>and is denoted f for the maximum gradient increase or ∇f for the maximum gradient decrease of the function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*1HRRFjQtIJO0jnGhGm9NTQ.png" /></figure><p>For instance, <strong>in machine learning optimization</strong>, gradients are <strong>used</strong> to iteratively <strong>update model parameters</strong> in order to <strong>minimize the loss function.</strong> In neural networks, methods like stochastic gradient descent rely on gradients to adjust weights during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*Fu835t2YeWwtNyFMHU49Lg.jpeg" /></figure><p>As shown in the image, the <strong>gradient</strong> indicates the direction of the steepest increase in cost, so taking incremental steps in the opposite direction helps move the weight toward the point of<strong> minimum cost</strong>(optimum). This process continues until the model reaches an <strong>optimal or near-optimal solution.</strong></p><p>References:</p><ul><li><a href="https://www.cuemath.com/algebra/product-of-vectors/">https://www.cuemath.com/algebra/product-of-vectors/</a></li><li><a href="https://www.analyticsvidhya.com/blog/2025/01/dot-product-vs-element-wise-multiplication/">https://www.analyticsvidhya.com/blog/2025/01/dot-product-vs-element-wise-multiplication/</a></li><li><a href="https://pubmed.ncbi.nlm.nih.gov/static-page/down_bethesda.html">https://pubmed.ncbi.nlm.nih.gov/static-page/down_bethesda.html</a></li><li><a href="https://www.singlestore.com/blog/distance-metrics-in-machine-learning-simplfied/#:~:text=Euclidean%20distance%2C%20often%20known%20as,distance%20in%20the%20physical%20world.">https://www.singlestore.com/blog/distance-metrics-in-machine-learning-simplfied/#:~:text=Euclidean%20distance%2C%20often%20known%20as,distance%20in%20the%20physical%20world.</a></li><li><a href="https://en.wikipedia.org/wiki/Vector_projection">https://en.wikipedia.org/wiki/Vector_projection</a></li><li><a href="https://www.singlestore.com/blog/distance-metrics-in-machine-learning-simplfied/#:~:text=Euclidean%20distance%2C%20often%20known%20as,distance%20in%20the%20physical%20world.">https://www.singlestore.com/blog/distance-metrics-in-machine-learning-simplfied/#:~:text=Euclidean%20distance%2C%20often%20known%20as,distance%20in%20the%20physical%20world.</a></li><li><a href="https://aitechtrend.com/how-cosine-similarity-can-improve-your-machine-learning-models/">https://aitechtrend.com/how-cosine-similarity-can-improve-your-machine-learning-models/</a></li><li><a href="https://doi.org/10.1016/j.jobe.2023.107289">https://doi.org/10.1016/j.jobe.2023.107289</a></li><li><a href="https://www.researchgate.net/publication/372985352_Stochastic_gradient_descent_optimisation_for_convolutional_neural_network_for_medical_image_segmentation">https://www.researchgate.net/publication/372985352_Stochastic_gradient_descent_optimisation_for_convolutional_neural_network_for_medical_image_segmentation</a></li><li><a href="https://www.futurelearn.com/info/courses/intelligent-systems/0/steps/245920">https://www.futurelearn.com/info/courses/intelligent-systems/0/steps/245920</a></li><li><a href="https://www.researchgate.net/publication/215991898_Vector_Symbolic_Architectures_A_New_Building_Material_for_Artificial_General_Intelligence">https://www.researchgate.net/publication/215991898_Vector_Symbolic_Architectures_A_New_Building_Material_for_Artificial_General_Intelligence</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28ea16d81dd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thrustworthy Machine Learning: Gradient/Decomposition-based XAI: GradCAM]]></title>
            <link>https://medium.com/@yalcinselcuk0/thrustworthy-machine-learning-gradient-decomposition-based-xai-gradcam-8d9b61dc6ffe?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d9b61dc6ffe</guid>
            <category><![CDATA[trustworthy-ai]]></category>
            <category><![CDATA[model-interpretability]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[grad-cam]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Fri, 18 Jul 2025 12:21:06 GMT</pubDate>
            <atom:updated>2025-07-18T12:21:06.083Z</atom:updated>
            <content:encoded><![CDATA[<p>Grad-CAM (Gradient-weighted Class Activation Mapping) is a technique designed to produce<strong> ‘visual explanations’ </strong>for decisions made by a wide range of Convolutional Neural Network (CNN)-based models, thereby making them more transparent and explainable.</p><p>Its primary <strong>goal</strong> is to help users understand why a deep neural network predicts what it predicts. This interpretability is crucial for building trust in intelligent systems and for their meaningful integration into daily life.</p><p>To support trustworthy machine learning, Grad-CAM provides a range of benefits that enhance the reliability and fairness of deep learning systems. First and foremost, it improves <strong>transparency</strong> and <strong>interpretability</strong> by visually highlighting the regions in an image that contribute most to a model’s decision. This allows users to understand the reasoning behind predictions better, even when they don’t have technical expertise.</p><p>By making the model’s internal logic visible, Grad-CAM helps build appropriate levels of trust between users and AI systems. It also plays a valuable role in diagnosing errors by showing why a model may have made an incorrect prediction, revealing that even seemingly flawed outputs can have underlying logic.</p><p>Ultimately, by helping identify and correct such issues, Grad-CAM contributes to the development of models that generalize better to real-world data and behave in more ethically responsible ways.</p><p>We call Grad-CAM a <em>gradient-based</em> method because it uses the <strong>gradients of the class score with respect to the feature maps</strong> in the final convolutional layer. These gradients serve as a signal to weight the importance of each feature map, enabling the generation of class-discriminative heatmaps.</p><p>Now, we can visualize and explain each step of GradCAM by using the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*OTBsRsOJfjEltaezRYxYlw.png" /></figure><h4>Section 1: Convolutional Layers (Feature Extraction)</h4><p>The input image passes through multiple <strong>convolutional layers</strong>. These layers output <strong>feature maps (Aᵏ)</strong>, where each feature map corresponds to a different filter or neuron.</p><h4>Section 2: Grad-CAM (Gradient-weighted Class Activation Mapping)</h4><p>Once the feature maps (<strong>Aᵏ</strong>) are generated by the convolutional layers, the following steps are applied to compute the class-discriminative saliency map using the Grad-CAM method</p><p><strong>Step 1: Forward Pass &amp; Class Selection</strong></p><p>Perform a <strong>forward pass</strong> to obtain the raw output scores (logits) and <strong>select the target class</strong> c, usually the top predicted class.</p><p><strong>Step 2: Backward Pass (Gradient Computation)</strong></p><p><strong>Backpropagate</strong> the class score <strong>yᶜ</strong> (before softmax) with respect to the feature maps Aᵏ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*t8GQwgCYtFfzNXZaqq0K5w.png" /></figure><p>This gives the <strong>sensitivity of each spatial location</strong> (i,j) in feature map Aᵏ<em> </em>for class <em>c</em>.</p><p><strong>Step 3: Calculate Importance Weights (αᶜₖ)</strong></p><p>Compute the average of the gradients over spatial dimensions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*6_QA-UF0MWcBA66w-DN-Ww.png" /></figure><p>Z: Total number of spatial locations in the feature map <em>(i.e., Z=i×j)</em></p><p><strong>Step 4: Combine Feature Maps and Apply ReLU, and Generate the Saliency Map</strong></p><p>Multiply each feature map Aᵏ with its importance weight αᶜₖ, sum over k, and apply ReLU to focus only on features that positively influence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*ma0c_qZDGojZRSh8Rgqeuw.png" /></figure><p>This produces the final <strong>class-discriminative saliency map</strong>, highlighting image regions that <strong>positively contribute to</strong> class c.</p><h4>Step 5: Visualise the results</h4><p><strong>Resize</strong> the resulting saliency map to match the original image size. <strong>Overlay</strong> it onto the original image to highlight the regions contributing to the selected class prediction.</p><h3>Python Code for GradCAM</h3><pre>import torch<br>import torch.nn.functional as F<br>from torchvision import models, transforms<br>from PIL import Image<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import cv2<br>#!!If you apply the code using Google Colab, write the code below<br>from google.colab import drive<br>drive.mount(&#39;/content/drive&#39;)<br><br># ----------- Preprocessing and Model Setup -----------<br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br># Load pre-trained model (e.g., ResNet50)<br>model = models.resnet50(pretrained=True).to(device)<br>model.eval()<br><br># Choose the target layer (last conv layer in ResNet50)<br>target_layer = model.layer4[-1]<br><br># Image preprocessing<br>preprocess = transforms.Compose([<br>    transforms.Resize((224, 224)),<br>    transforms.ToTensor(),<br>    transforms.Normalize(mean=[0.485, 0.456, 0.406],<br>                         std=[0.229, 0.224, 0.225])<br>])<br><br>def load_image(path):<br>    img = Image.open(path).convert(&#39;RGB&#39;)<br>    return preprocess(img).unsqueeze(0).to(device)<br>def load_original_image(path):<br>    img = Image.open(path).convert(&#39;RGB&#39;)<br>    img = img.resize((224, 224))<br>    return np.array(img)<br><br># ----------- Grad-CAM Core Logic -----------<br><br>class GradCAM:<br>    def __init__(self, model, target_layer):<br>        self.model = model<br>        self.target_layer = target_layer<br><br>        self.gradients = None<br>        self.activations = None<br><br>        # Hook the forward and backward pass<br>        target_layer.register_forward_hook(self.save_activation)<br>        target_layer.register_backward_hook(self.save_gradient)<br><br>    def save_activation(self, module, input, output):<br>        self.activations = output.detach()<br><br>    def save_gradient(self, module, grad_input, grad_output):<br>        self.gradients = grad_output[0].detach()<br><br>    def __call__(self, input_tensor, class_idx=None):<br>        # Forward pass<br>        output = self.model(input_tensor)<br><br>        if class_idx is None:<br>            class_idx = output.argmax(dim=1).item()<br><br>        # ----------- Step 1: Compute Gradients of Class Score -----------<br>        score = output[0, class_idx]<br>        self.model.zero_grad()<br>        score.backward()<br><br>        # ----------- Step 2: Calculate Importance Weights (αᶜₖ) -----------<br>        grads = self.gradients  # [B, C, H, W]<br>        pooled_grads = torch.mean(grads, dim=[2, 3])  # Global average pooling<br><br>        # ----------- Step 3: Combine Feature Maps and Apply ReLU -----------<br>        activations = self.activations  # [B, C, H, W]<br>        for i in range(activations.shape[1]):<br>            activations[:, i, :, :] *= pooled_grads[:, i].unsqueeze(-1).unsqueeze(-1)<br><br>        heatmap = torch.sum(activations, dim=1).squeeze()<br>        heatmap = F.relu(heatmap)  # Apply ReLU to focus on positive contributions<br><br>        # Normalize and convert to NumPy<br>        heatmap = heatmap.cpu().numpy()<br>        heatmap = (heatmap - heatmap.min()) / (heatmap.max() - heatmap.min() + 1e-8)<br><br>        return heatmap, class_idx<br><br>def overlay_heatmap(heatmap, image_path, alpha=0.6):<br>    img = cv2.imread(image_path)<br>    img = cv2.resize(img, (224, 224))<br>    heatmap = cv2.resize(heatmap, (224, 224))<br>    heatmap_colored = cv2.applyColorMap(np.uint8(255 * heatmap), cv2.COLORMAP_JET)<br>    overlay = cv2.addWeighted(img, 1 - alpha, heatmap_colored, alpha, 0)<br>    return overlay<br><br><br># ----------- Run on Example Image -----------<br><br># Set your image path<br>image_path = &quot;your image&quot; # Write your patch of image here <br><br><br># Generate Grad-CAM heatmap and overlay<br>input_tensor = load_image(image_path)<br>gradcam = GradCAM(model, target_layer)<br>heatmap, class_id = gradcam(input_tensor)<br>overlay = overlay_heatmap(heatmap, image_path)<br><br># Load original image for comparison<br>original_image = load_original_image(image_path)<br><br># ----------- Plot: Side-by-Side Comparison -----------<br>plt.figure(figsize=(12, 5))<br><br># Original<br>plt.subplot(1, 3, 1)<br>plt.imshow(original_image)<br>plt.title(&quot;Original Image&quot;)<br>plt.axis(&quot;off&quot;)<br><br># Grad-CAM Overlay<br>plt.subplot(1, 3, 2)<br>plt.imshow(cv2.cvtColor(overlay, cv2.COLOR_BGR2RGB))<br>plt.title(f&quot;Grad-CAM (Class {class_id})&quot;)<br>plt.axis(&quot;off&quot;)<br><br><br>plt.tight_layout()<br>plt.show()<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*wNuUxoxUjPOFLI_qIpnCIQ.png" /></figure><p>In short, Grad-CAM is a method used to visualize which parts of an image a deep learning model focuses on when making a classification decision. It does this by generating a class-discriminative saliency map that highlights the image regions most relevant to the model’s prediction. The input is a regular image (e.g., a bird photo), and the output is a heatmap overlaid on the image, showing the areas that contributed most to the predicted class.</p><p><strong>References</strong>:</p><p><a href="https://arxiv.org/pdf/1610.02391">https://arxiv.org/pdf/1610.02391</a></p><p><a href="https://www.researchgate.net/profile/Aditya-Chattopadhyay/publication/320727679/figure/fig9/AS:631199986892800@1527501212032/An-overview-of-all-the-three-methods-CAM-Grad-CAM-GradCAM-with-their-respective.png">https://www.researchgate.net/profile/Aditya-Chattopadhyay/publication/320727679/figure/fig9/AS:631199986892800@1527501212032/An-overview-of-all-the-three-methods-CAM-Grad-CAM-GradCAM-with-their-respective.png</a></p><p><a href="https://arxiv.org/pdf/1611.07450">https://arxiv.org/pdf/1611.07450</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d9b61dc6ffe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thrustworthy Machine Learning: Gradient/Decomposition-based XAI: Saliency Maps]]></title>
            <link>https://medium.com/@yalcinselcuk0/thrustworthy-machine-learning-gradient-decomposition-based-xai-saliency-maps-ebfe5fe4d430?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebfe5fe4d430</guid>
            <category><![CDATA[saliency-map]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[graident-based-ai]]></category>
            <category><![CDATA[medical-imaging]]></category>
            <category><![CDATA[trustworthy-ai]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 21:34:10 GMT</pubDate>
            <atom:updated>2025-07-18T10:56:49.020Z</atom:updated>
            <content:encoded><![CDATA[<p>A <strong>saliency map</strong> is a visualization method used to identify the most influential regions of an input image that contribute to a deep Convolutional Neural Network’s (CNN or ConvNet) prediction for a specific class. It provides <strong>image-specific insights</strong>, highlighting which parts of a given image had the greatest impact on the model’s decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mk4bixHLr5oUqwZlcfioJg.png" /></figure><p>Generally, saliency maps are generated by computing the <strong>gradient of the output class score with respect to the pixels of the input image</strong>. <em>These gradients indicate how small changes in each pixel would affect the model’s output, allowing us to pinpoint which areas the network “focused on” when making its prediction.</em></p><p>For instance, when a Convolutional Neural Network (ConvNet) predicts that an X-ray shows signs of pneumonia, a <strong>saliency map</strong> for the “pneumonia” class highlights the specific regions in the image that most influenced the model’s decision.</p><p>Not only do we see the label “pneumonia detected,” but a radiologist also sees where the AI focused, for example, a particular lobe of the lung or an area showing abnormal patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*vRvytp5Pj0kDsQeG7wigLA.jpeg" /><figcaption><strong>Expert-Annotated CXRs and Saliency Maps Showing COVID-19 Severity Levels</strong></figcaption></figure><p>As we see in this example above, the AI-generated saliency maps highlight the regions of interest in chest X-rays for three patients with mild (a), moderate (b), and severe COVID-19 pneumonia. The predicted class probabilities for these images are 0.96, 0.67, and 0.89, respectively, given the AI. The highlighted areas in the saliency maps closely match the affected regions identified by an expert radiologist. This clearly shows that the model not only makes accurate predictions but also provides explanations that align well with expert clinical judgment.</p><p>We are able to summarise what we discussed so far using the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*3FnF3XSdQLoNb3fnwRCgeg.png" /><figcaption><em>Evaluating Saliency Maps in Chest X-rays: Comparing AI Attention with Radiologist Annotations</em></figcaption></figure><p><strong>Step 1: Saliency Maps Generation</strong>: <em>The model’s</em> <strong><em>(These predictions) </em></strong><em>predictions</em> are visualized using saliency methods that produce heatmaps, showing which image regions influenced the decision.</p><p><strong>Step 2: Segmentation by Thresholding</strong>: These heatmaps are thresholded to create segmented regions, indicating the model’s “focus” areas.</p><p><strong>Step 3: Expert Annotations for Evaluation</strong>: Radiologists independently annotate the same CXRs. These expert segmentations are used as ground truth to assess how well the model’s focus aligns with clinical understanding.</p><p><strong>Step 4: Performance Comparison</strong>: The AI-generated regions are compared to expert annotations to evaluate the <strong><em>saliency method&#39;s localisation performance</em>.</strong> Human radiologists’ segmentations are also compared to one another to establish a <em>human benchmark</em>.</p><p>In short, this approach helps evaluate not only the accuracy of the AI model but also focuses on the correct medical indicators, a crucial step toward developing trustworthy and clinically reliable AI in healthcare.</p><h3>A step-by-step workflow of the saliency maps</h3><p>We want to determine <strong>which pixels</strong> of an image influence the classification decision of a neural network for a <strong>specific class</strong>.</p><p>Given:</p><ul><li>Image: Io (input)</li><li>Class: c</li><li>Class score function: Sc(I), which outputs how strongly the network thinks the image belongs to class c</li></ul><p><strong>Step 1: Start with a Simple Linear Model: </strong>We will firstly consider a simple linear classifier to understand how pixel importance can be measured. The image I(input) is flattened into a vector, and the class score is calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*i_vptQMTmarorvCZgcAKPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eAp_izaOVgx8HD0fzm2DYA.png" /></figure><p><strong>Step 2: Deep Networks Are Non-Linear: </strong>In deep neural networks, the class score function Sc(I)is <strong>non-linear</strong>, so the simple weight-based interpretation does not directly apply.</p><p>To analyze pixel importance, we have to find a way to <strong>approximate</strong> this non-linear function locally around the input image Io​.</p><p><strong>Step 3: Use First-Order Taylor Expansion: </strong>We apply a first-order Taylor expansion around image Io​, which gives us a local linear approximation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/308/1*kxMMLI_2PW6euNpBUq9ekQ.png" /></figure><p>This allows us to treat the model <strong>as if it were linear near Io​</strong>, bringing back the idea of weight-based pixel importance.</p><p><strong>Step 4: Take the Derivative of the Class Score with Respect to the Input Image: </strong>We compute the <strong>derivative (gradient)</strong> of the class score with respect to the input image I, evaluated at the specific image Io:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*3w9Fkgl9wBYbA66X8ZplVQ.png" /></figure><p>This gradient tells us how much each pixel influences the class score. <strong>The larger the value, the more sensitive the model is to that pixel.</strong></p><p><strong>Step 5: Generate a Saliency Map: </strong>By computing this gradient for a given image Io​, we get a <strong>saliency map, </strong>a pixel-wise visualization of what the model focuses on when making its prediction for class c.</p><h3>Python Code for Saliency Maps</h3><pre>import numpy as np<br>import tensorflow as tf<br>import matplotlib.pyplot as plt<br>from tensorflow.keras.applications.densenet import DenseNet121, preprocess_input<br>from tensorflow.keras.preprocessing import image<br><br># Download a different chest X-ray image from the COVID Chest X-ray dataset<br>img_url = &#39;https://github.com/ieee8023/covid-chestxray-dataset/raw/master/images/000003-2.jpg&#39;<br>img_path = tf.keras.utils.get_file(&#39;demo_cxr_new.jpg&#39;, img_url)<br>img = image.load_img(img_path, target_size=(224, 224))<br>img_array = image.img_to_array(img)<br>x_input = np.expand_dims(img_array, axis=0)<br>x_input = preprocess_input(x_input.copy())<br><br># Load pretrained DenseNet121 (placeholder for CheXNet)<br>model = DenseNet121(weights=&#39;imagenet&#39;)<br>preds = model.predict(x_input)<br>class_idx = np.argmax(preds[0])<br><br># Compute gradients for saliency map<br>with tf.GradientTape() as tape:<br>    x_tensor = tf.convert_to_tensor(x_input)<br>    tape.watch(x_tensor)<br>    output_tensor = model(x_tensor)<br>    pred_score = output_tensor[0][class_idx]<br>grads = tape.gradient(pred_score, x_tensor)[0].numpy()<br><br># Normalize saliency map<br>saliency = np.max(np.abs(grads), axis=-1)<br>saliency = (saliency - saliency.min()) / (saliency.max() - saliency.min())<br><br># Plot original image, saliency map, and overlay using jet colormap<br>plt.figure(figsize=(12, 6))<br><br>plt.subplot(1, 3, 1)<br>plt.imshow(img, cmap=&#39;gray&#39;)<br>plt.title(&quot;Original CXR&quot;)<br>plt.axis(&#39;off&#39;)<br><br>plt.subplot(1, 3, 2)<br>plt.imshow(saliency, cmap=&#39;jet&#39;)<br>plt.title(&quot;Saliency Map&quot;)<br>plt.axis(&#39;off&#39;)<br><br>plt.subplot(1, 3, 3)<br>plt.imshow(img, cmap=&#39;gray&#39;)<br>plt.imshow(saliency, cmap=&#39;jet&#39;, alpha=0.5)<br>plt.title(&quot;Overlay&quot;)<br>plt.axis(&#39;off&#39;)<br><br>plt.tight_layout()<br>plt.show()<br><br>print(&quot;Predicted class index:&quot;, class_idx)<br>print(&quot;Prediction confidence:&quot;, preds[0][class_idx])<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GafPz3Kzuk18joQZvsF3kw.png" /></figure><p>The saliency map highlights several areas in the lungs that influenced the model’s prediction, with a stronger focus on the lower right region. This suggests that the AI is picking up on clinically relevant features, even if not perfectly localized. The results are visually informative and demonstrate that the model is not making random decisions. Overall, this saliency map provides a useful and interpretable clue into how the AI assessed the chest X-ray.</p><p>References:</p><p><a href="https://the-decoder.com/clinical-ai-widespread-method-performs-poorly-in-benchmark/">https://the-decoder.com/clinical-ai-widespread-method-performs-poorly-in-benchmark/</a></p><p><a href="https://arxiv.org/pdf/1312.6034">https://arxiv.org/pdf/1312.6034</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebfe5fe4d430" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thrustworthy Machine Learning Score Estimation: Nested Cross-Validation]]></title>
            <link>https://medium.com/@yalcinselcuk0/thrustworthy-machine-learning-score-estimation-nested-cross-validation-5f03dc03e665?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f03dc03e665</guid>
            <category><![CDATA[model-selection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[trustworthy-ai]]></category>
            <category><![CDATA[score-estimation]]></category>
            <category><![CDATA[nested-cross-validation]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Sun, 06 Jul 2025 15:12:55 GMT</pubDate>
            <atom:updated>2025-08-21T00:15:21.294Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Nested cross-validation</strong> is a trustworthy method for <strong>model evaluation </strong>and <strong>score estimation.</strong></p><p>In typical model selection, we tune hyperparameters and evaluate performance using the same dataset, most commonly via k-fold cross-validation. However, this can lead to <strong>biased and overly optimistic score estimates</strong> as a result of the model being indirectly exposed to the test data during tuning.</p><p>Nested cross-validation helps us solve this issue, &quot;<strong>biased and overly optimistic score estimates&quot;,</strong> by using two separate loops of cross-validation:</p><ul><li>An <strong>inner loop</strong> for hyperparameter tuning</li><li>An <strong>outer loop</strong> for unbiased score estimation</li></ul><p>By keeping the evaluation process separate from the tuning process, N<strong>ested Cross Validation provides a more accurate and trustworthy estimate</strong> of how a model will perform on new, unseen data. This is the reason why it’s often the go-to method when attentive model comparison is needed.</p><p>After understanding the general idea of nested cross-validation, let&#39;s go through the process to distinguish how it works step by step.</p><h3>A step-by-step workflow of the nested cross-validation method</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8hpdI_XdRgG9TpE4bChAg.png" /></figure><h4><strong>Step 1: Split the dataset</strong></h4><p><strong>Divide</strong> your full <strong>dataset</strong> into K folds<strong> </strong>(e.g., K = 3). This is the <strong>outer loop</strong>, which is used to <strong>estimate</strong> the model’s final performance.</p><h4><strong>Step 2: Outer loop for generalisation estimation</strong></h4><p>For each of the K folds:</p><p><strong>Use fold k as the test set: </strong>Set aside one fold (fold <em>k</em>) as the test set.</p><p><strong>Use the rest as training and validation: </strong>Combine the remaining K-1 folds to form a training/validation set (called train-validation).</p><p><strong>Split train validation into L inner folds:</strong> Split this train validation set into L new folds (e.g., L = 3 ).</p><p><strong>Inner loop for hyperparameter tuning</strong></p><p>For each inner fold:</p><p><strong>Use one fold as validation: </strong>Choose one of the L folds as the validation set.</p><p><strong>Train on the remaining inner folds: </strong>Train your model using <strong>different hyperparameter</strong> values.</p><p><strong>Evaluate on validation set: </strong>Examine(test!!!) how well each model performs on the validation fold. Record the performance for each hyperparameter setting.</p><p><strong>Select the best hyperparameter:</strong> From the inner loop results, <strong>average the validation scores</strong> across all L folds. Choose the <strong>best-performing hyperparameter</strong>.</p><p><strong>Train &amp; test on outer loop:</strong> Train a new model using the <strong>entire train validation set</strong> and the <strong>best hyperparameters</strong>. Then, <strong>evaluate this model on the outer test set (fold k)</strong>. Save the performance score.</p><h4><strong>Step 3: Final generalisation score</strong></h4><p>Repeat the process for all K outer folds. Finally, <strong>average all the outer test scores</strong> to get an <strong>unbiased generalisation estimate</strong>.</p><p><strong>Pseudo-code of Nested Cross Validation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*MHXtG2coE3vefIhSpDId_A.png" /></figure><h3>Python Code for Hold out Set Evaluation Method</h3><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.datasets import load_iris<br>from sklearn.model_selection import KFold<br>from sklearn.neighbors import KNeighborsClassifier<br>from sklearn.metrics import accuracy_score<br>import matplotlib.pyplot as plt<br><br># Load dataset<br>X, y = load_iris(return_X_y=True)<br><br># Step 1: Split the dataset — Outer CV loop (for estimating generalization)<br>K = 3  # Number of outer folds<br>outer_cv = KFold(n_splits=K, shuffle=True, random_state=42)<br>outer_scores = []<br>outer_fold_indices = []<br>all_inner_scores = []<br><br># Step 2: Outer loop for generalization estimation<br>for fold_idx, (outer_trainval_idx, outer_test_idx) in enumerate(outer_cv.split(X), 1):<br>    X_trainval, X_test = X[outer_trainval_idx], X[outer_test_idx]<br>    y_trainval, y_test = y[outer_trainval_idx], y[outer_test_idx]<br><br>    # Inner loop for hyperparameter tuning<br>    L = 3  # Number of inner folds<br>    inner_cv = KFold(n_splits=L, shuffle=True, random_state=42)<br>    hyperparams = list(range(1, 11))  # Trying k from 1 to 10<br>    best_avg_score = -np.inf<br>    best_k = None<br>    inner_scores_for_k = []<br><br>    # Try each k value<br>    for k in hyperparams:<br>        inner_scores = []<br><br>        # Inner CV loop <br>        for inner_train_idx, inner_val_idx in inner_cv.split(X_trainval):<br>            X_inner_train, X_inner_val = X_trainval[inner_train_idx], X_trainval[inner_val_idx]<br>            y_inner_train, y_inner_val = y_trainval[inner_train_idx], y_trainval[inner_val_idx]<br><br>            model = KNeighborsClassifier(n_neighbors=k)<br>            model.fit(X_inner_train, y_inner_train)<br>            val_pred = model.predict(X_inner_val)<br>            acc = accuracy_score(y_inner_val, val_pred)<br>            inner_scores.append(acc)<br><br>        avg_val_score = np.mean(inner_scores)<br>        inner_scores_for_k.append(avg_val_score)<br><br>        if avg_val_score &gt; best_avg_score:<br>            best_avg_score = avg_val_score<br>            best_k = k<br><br>    # Save inner results for plotting<br>    all_inner_scores.append(inner_scores_for_k)<br><br>    # Train with best hyperparameter on full trainval and evaluate on outer test set<br>    final_model = KNeighborsClassifier(n_neighbors=best_k)<br>    final_model.fit(X_trainval, y_trainval)<br>    test_pred = final_model.predict(X_test)<br>    test_score = accuracy_score(y_test, test_pred)<br>    outer_scores.append(test_score)<br>    outer_fold_indices.append(fold_idx)<br><br># Step 3: Final generalization score<br>final_score = np.mean(outer_scores)<br>print(f&quot;Nested CV Final Generalization Accuracy: {final_score:.4f}&quot;)<br><br># Inner loop scores per outer fold<br>plt.figure(figsize=(12, 5))<br>for i, scores in enumerate(all_inner_scores):<br>    plt.plot(range(1, 11), scores, marker=&#39;o&#39;, label=f&#39;Outer Fold {i+1}&#39;)<br>plt.title(&#39;Inner Loop Validation Accuracy for Each k (per Outer Fold)&#39;)<br>plt.xlabel(&#39;k (n_neighbors)&#39;)<br>plt.ylabel(&#39;Validation Accuracy&#39;)<br>plt.xticks(range(1, 11))<br>plt.ylim(0.8, 1.0)<br>plt.grid(True)<br>plt.legend()<br>plt.tight_layout()<br>plt.show()<br><br><br># Outer loop test scores<br>plt.figure(figsize=(7, 5))<br>plt.bar(outer_fold_indices, outer_scores, color=&#39;skyblue&#39;, edgecolor=&#39;black&#39;)<br>plt.axhline(y=final_score, color=&#39;red&#39;, linestyle=&#39;--&#39;, label=f&#39;Avg Accuracy = {final_score:.3f}&#39;)<br>plt.title(&#39;Outer Loop Test Accuracy per Fold&#39;)<br>plt.xlabel(&#39;Outer Fold&#39;)<br>plt.ylabel(&#39;Test Accuracy&#39;)<br>plt.ylim(0.8, 1.0)<br>plt.legend()<br>plt.grid(axis=&#39;y&#39;)<br>plt.tight_layout()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fS8hCIY_qlXIHL_sQmqlcQ.png" /></figure><p>We tested different values of k (the number of neighbours) within each outer fold using<strong> </strong>3-fold cross-validation <strong>(the inner loop)</strong>. For each outer fold, we trained the model on one part of the data and validated it on another part. We repeated this process for different values of k ranging from 1 to 10. The plot gives us the validation accuracy for <strong>each k</strong>; each coloured line corresponds to a different outer fold. The <strong>objective </strong>was to <strong>identify</strong> the <strong>optimal k</strong> value that yields the highest average validation accuracy for each outer fold.<br>As can be seen in the plot, the accuracy is consistently high (mostly above 94%) across all k values, which suggests that the KNN model performs robustly over a wide range of k values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*dyFdW802JrJkd8IOaVvIvg.png" /></figure><p>After finding the best <strong>k</strong> in the inner loop, we used that <strong>k</strong> to train a model on the full <strong>train-validation set</strong> and evaluated it on the <strong>outer test fold</strong> (which was never seen during tuning). This process was repeated for <strong>3 outer folds</strong> (since K=3). The <strong>plot</strong> gives us the <strong>test accuracy</strong> for each outer fold using the selected <strong>best k</strong>. Accuracies are very high (94%–98%), and the <strong>average test accuracy</strong> across folds is <strong>96%</strong>, shown as a red dashed line. This final score is a reliable estimate of the model’s true generalization performance.</p><p>In summary, by using nested cross-validation, we were <strong>able to select the most appropriate hyperparameter for each outer fold</strong> and estimate the <strong>model’s performance </strong>on completely unseen data. This process helps reduce bias and provides an almost unbiased estimation of the model’s generalization accuracy.</p><p>References:</p><p><a href="https://liuziyang1106.github.io/work/2021/05/14/cross-validation/">https://liuziyang1106.github.io/work/2021/05/14/cross-validation/</a></p><p>Evaluating Machine Learning Models and Algorithms <a href="https://arxiv.org/abs/2506.21067v1">arXiv:2506.21067v1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f03dc03e665" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thrustworthy Machine Learning Score Estimation: Cross-Validation]]></title>
            <link>https://medium.com/@yalcinselcuk0/thrustworthy-machine-learning-score-estimation-cross-validation-13fcec6659d6?source=rss-21bd9cde6b10------2</link>
            <guid isPermaLink="false">https://medium.com/p/13fcec6659d6</guid>
            <category><![CDATA[trustworthy-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[crossvalidation]]></category>
            <category><![CDATA[xai]]></category>
            <dc:creator><![CDATA[Yalcinselcuk]]></dc:creator>
            <pubDate>Fri, 04 Jul 2025 22:20:36 GMT</pubDate>
            <atom:updated>2025-07-04T22:20:36.097Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Cross-validation</strong>, like holdout set evaluation or the bootstrapping method, is a widely used <strong>score estimation technique in</strong> machine learning. Its main goal is to estimate how well a model will perform on unseen data, which is essential for understanding generalization. When performing <strong>model selection</strong>, we often generate multiple candidate models by varying hyperparameters. To choose the best one, we must evaluate and compare their performance reliably.</p><p>However, using the same test set repeatedly during this process can lead to <strong>biased and overly optimistic results</strong>, because the model might indirectly “learn” from the test data, a problem known as <strong>information leakage</strong>. To avoid this, <strong>cross-validation</strong> splits the data into multiple subsets, ensuring that each sample gets a chance to be in the test set at least once. This way, every model is evaluated fairly, and we get a more robust estimate of its real-world performance.</p><p>Often, cross-validation is combined with an <strong>independent test set</strong> that’s kept completely separate until the final evaluation, making sure that the final performance estimate remains unbiased.</p><p><strong>Now let’s try to understand this method step-by-step.</strong></p><h3>A step-by-step workflow of the cross-validation method</h3><p><strong>Step 1:</strong> <strong>Initial Data Split into Training and Independent Test Set</strong></p><p>We begin by splitting your entire available dataset into two primary parts: a training set and an independent test set. (This step is the same as Hold out set evaluation)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*NSCWHMx7hpnie_v4mLlkOQ.png" /></figure><p><strong>Step 2: Hyperparameter Tuning with K-Fold Cross-Validation</strong></p><p>The second step involves testing different hyperparameter settings using <strong>k-fold cross-validation</strong> on the training set only. Each configuration is evaluated by averaging performance across k folds, helping to choose the best hyperparameters while avoiding bias by keeping preprocessing steps inside the loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvoPaQa76Vbt9VM36F6hRw.png" /></figure><p><strong>Step 3: Model Selection and Refitting on Combined Training Set</strong></p><p>After selecting the best hyperparameters from cross-validation, we retrain the model on the <strong>entire training set</strong> using those optimal settings. This allows the model to learn from all available training data, which can lead to better generalization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*z1p94CuhMb018bIjMHlp1Q.png" /></figure><p>Step 4: <strong>Final Evaluation on the Independent Test Set</strong></p><p>We now use the <strong>independent test set</strong> that was set aside in Step 1 to evaluate the final model. This gives us an <strong>unbiased estimate</strong> of how well the model is likely to perform on truly unseen data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*mmC56fPJn-i7n_M59GsMDQ.png" /></figure><p>Step 5: <strong>Final Model Training for Deployment</strong></p><p>Finally, after evaluating the model on the test set, we can optionally retrain it on the <strong>entire dataset</strong> (training + test combined). This version of the model uses all available data and is typically the one used for <strong>real-world deployment</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*_fvHnsqaBxAQQoRt2HqfmQ.png" /></figure><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from sklearn.datasets import load_iris<br>from sklearn.model_selection import train_test_split<br>from sklearn.neighbors import KNeighborsClassifier<br>from sklearn.metrics import accuracy_score<br>from sklearn.model_selection import GridSearchCV<br><br># Step 1: Split into Training + Test Set (Test set is held out for final evaluation)<br>X, y = load_iris(return_X_y=True)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br># Step 2: Hyperparameter tuning using 5-fold cross-validation on training set<br>param_range = list(range(1, 21))<br>param_grid = {&#39;n_neighbors&#39;: param_range}<br><br>grid = GridSearchCV(KNeighborsClassifier(), param_grid, cv=5, scoring=&#39;accuracy&#39;)<br>grid.fit(X_train, y_train)<br><br># Step 3: Select best model and refit it on the full training set<br>best_k = grid.best_params_[&#39;n_neighbors&#39;]<br>best_model = grid.best_estimator_<br><br># Step 4: Evaluate the selected model on the independent test set<br>y_test_pred = best_model.predict(X_test)<br>test_accuracy = accuracy_score(y_test, y_test_pred)<br><br># Step 5: Optionally refit model on all data (train + test) for deployment<br>final_model = KNeighborsClassifier(n_neighbors=best_k)<br>final_model.fit(X, y)<br><br># Visualization of Validation Accuracy vs. K<br>mean_scores = grid.cv_results_[&#39;mean_test_score&#39;]<br><br>plt.figure(figsize=(10, 6))<br>plt.plot(param_range, mean_scores, marker=&#39;o&#39;, color=&#39;blue&#39;, label=&#39;Validation Accuracy&#39;)<br>plt.axvline(x=best_k, color=&#39;green&#39;, linestyle=&#39;--&#39;, label=f&#39;Best k = {best_k}&#39;)<br>plt.title(&#39;Validation Accuracy vs. n_neighbors (K)&#39;)<br>plt.xlabel(&#39;n_neighbors&#39;)<br>plt.ylabel(&#39;Accuracy&#39;)<br>plt.xticks(param_range)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br>plt.show()<br><br># Summary<br>print(f&quot;Best K: {best_k}&quot;)<br>print(f&quot;Validation Accuracy (CV): {grid.best_score_:.4f}&quot;)<br>print(f&quot;Test Accuracy: {test_accuracy:.4f}&quot;)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*JSg4VJnOesVpajZ3xMGNJA.png" /></figure><p>As we can see from the graph, the validation accuracy of the K-Nearest Neighbors (KNN) model changes with different values of n_neighbors. The highest validation accuracy is achieved at <strong>k = 3</strong>, reaching approximately <strong>0.958</strong> (or <strong>95.8%</strong>). This indicates that using 3 neighbors provides the best balance between model complexity and predictive performance for this dataset.</p><p>Reference: Evaluating Machine Learning Models and Algorithms <a href="https://arxiv.org/abs/2506.21067v1">arXiv:2506.21067v1</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13fcec6659d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>