<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ornela Megne on Medium]]></title>
        <description><![CDATA[Stories by Ornela Megne on Medium]]></description>
        <link>https://medium.com/@pmegne?source=rss-608320f9ab7c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wcAijNGIGyenPOvDoDUOqQ.jpeg</url>
            <title>Stories by Ornela Megne on Medium</title>
            <link>https://medium.com/@pmegne?source=rss-608320f9ab7c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:07:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pmegne/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Image Orientation in 3D NIfTI Medical Imaging.]]></title>
            <link>https://ai.plainenglish.io/image-orientation-in-3d-nifti-medical-imaging-06267dd854f8?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/06267dd854f8</guid>
            <category><![CDATA[medical-imaging]]></category>
            <category><![CDATA[image-orientation]]></category>
            <category><![CDATA[nifti]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Wed, 28 Aug 2024 11:25:47 GMT</pubDate>
            <atom:updated>2024-09-02T09:15:41.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*WPDNo7AtGo-IxxmVx6kbiQ.png" /><figcaption><a href="https://medicine107.wordpress.com/2014/12/24/anatomical-planes/">Human Anatomical Planes</a>.</figcaption></figure><p>Working with 3D NIfTI medical images (e.g., CT, MRI, PET) sometimes requires attention to image orientation during preprocessing, as scanned images can have varying fields of view. Depending on the desired orientation, it may be necessary to harmonize all images to a consistent orientation or convert images from one orientation to another.</p><p>In this post, we will discuss some slice orientations and how to transition between them.</p><h3><strong>I. Slice Orientation Names</strong></h3><p>It’s important to recall the three main planes of motion:</p><ul><li><strong>Transversal Plane (Axial)</strong>: divides the body into bottom (inferior) and top (superior).</li><li><strong>Coronal Plane (Frontal)</strong>: divides the body into front (anterior) and back (posterior).</li><li><strong>Sagittal Plane</strong>: divides the body into left and right sides.</li></ul><p>Next, we will discuss the reference coordinate system used for naming orientations in relation to the patient/subject as mentioned in [1].</p><h4><strong>1 . RAS</strong></h4><p>RAS refers to the coordinate system where the axes are oriented as Left to Right (X-axis), Posterior to Anterior (Y-axis), and Inferior to Superior (Z-axis) (L-R, P-A, I-S). In this system, the Right, Anterior, and Superior directions are positive values on their respective axes. It can also referred to as the LPI coordinate system, indicating that the axes start from the Left, Posterior, and Inferior sides of the subject.</p><ul><li>Transversal plane view: If you want to consider a transversal view, or feed the image into a model in the transversal plane, the third axis (I-S) will represent the depth, and the slice <strong>image[:,:,0]</strong> corresponds to the first transversal slice view.</li><li>Coronal plane view: If you are interested in the coronal plane view, the second axis (P-A) will denote the depth, and the slice <strong>image[:,0,:]</strong> represents the first coronal slice view.</li></ul><h4><strong>2. LIA</strong></h4><p>LIA indicates that the axes are oriented as Right to Left, Superior to Inferior, and Posterior to Anterior (R-L, S-I, P-A).</p><ul><li>Coronal plane view: the slice <strong>image[:,:,0]</strong> represents the first coronal slice view.</li><li>Transversal plane view: the slice <strong>image[:,0,:]</strong> represents the first transversal slice view.</li><li>Sagittal plane view: the slice <strong>image[0,:,:]</strong> represents the first sagittal slice view.</li></ul><h3><strong>II. Script</strong></h3><p>To illustrate this, we will use the obtained converted MRI image in nifti format done in this <a href="https://medium.com/@pmegne/dicom-to-nifti-conversion-dc436c0b5fa6">posts</a>. We will plot the image in the transversal, coronal, and sagittal planes using Python. Additionally, we will check the original orientation of the image and show how to switch to another orientation.</p><p>First we need to check the orientation of the image:</p><pre>def get_orientation(image):<br><br>    &quot;&quot;&quot; Determine the orientation of a NIfTI image.<br>        The affine array is used for this purpose since it gives the position of the image array data in a reference coordinate system.<br>    &quot;&quot;&quot;<br><br>    affine = image.affine<br>    orientation = nib.orientations.io_orientation(image.affine)<br>    codes = nib.orientations.aff2axcodes(affine)<br>    return codes, orientation<br><br>codes, orientation = get_orientation(MRI_nii)<br>print(image_orientation, &#39;\n&#39;)<br>print(orientation)</pre><p>The result is: <strong>(‘L’, ‘A’, ‘S’) [[ 0. -1.] [ 1. 1.] [ 2. 1.]]</strong>, where:</p><ul><li><strong>0</strong> represents the <strong>x-axis</strong> of the image and <strong>-1</strong> indicates that the x-axis is oriented from Right to Left (R-L).</li><li><strong>1</strong> represents the <strong>y-axis</strong> of the image and 1 indicates that the y-axis is oriented from Posterior to Anterior(P-A).</li><li><strong>2</strong> represents the <strong>z-axis</strong> of the image and <strong>1</strong> indicates that the z-axis is oriented from Inferior to Superior (I-S).</li></ul><p>To plot this orientation, we can call the plot_image_in_planes() function below:</p><pre>def plot_image_in_planes(image_data):<br><br>    &quot;&quot;&quot; Plot the image data in sagittal, coronal, and transversal planes.<br>        We use the function rotate for a better view of the image.<br><br>    Parameters:<br>    image_data (numpy.ndarray): The 3D image data to be plotted.<br>    &quot;&quot;&quot;<br>    angle=90<br>    sagittal_slice = image_data[image_data.shape[0] // 2, :, :]<br>    coronal_slice = image_data[:, image_data.shape[1] // 2, :]<br>    transversal_slice = image_data[:, :, image_data.shape[2] // 2]<br><br>    plt.figure(figsize=(10, 30))<br><br>    plt.subplot(1, 3, 1)<br>    plt.imshow(rotate(sagittal_slice, angle=angle),cmap=&#39;gray&#39;)<br>    plt.title(&#39;Sagittal&#39;)<br>    plt.axis(&#39;off&#39;)<br><br>    plt.subplot(1, 3, 2)<br>    plt.imshow(rotate(coronal_slice, angle=angle), cmap=&#39;gray&#39;)<br>    plt.title(&#39;Coronal&#39;)<br>    plt.axis(&#39;off&#39;)<br><br>    plt.subplot(1, 3, 3)<br>    plt.imshow(rotate(transversal_slice, angle=angle), cmap=&#39;gray&#39;)<br>    plt.title(&#39;Transversal&#39;)<br>    plt.axis(&#39;off&#39;)</pre><p>Figure 1 shows view of some slices in the three planes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jRaVSdoflr8fseBg84gQ2Q.gif" /><figcaption>Figure 1: MRI slices view. Image made by the authors.</figcaption></figure><p>Finally, to change the orientation, we simply need to specify the target orientation as shown below:</p><pre>def reorient_image(image, target_orientation):<br><br>    &quot;&quot;&quot; Reorient image data to the target orientation.<br>    &quot;&quot;&quot;<br><br>    cur_orientation = nib.orientations.io_orientation(image.affine)<br>    target_orientation = nib.orientations.axcodes2ornt(target_orientation)<br>    print(f&quot;target_orientation: {target_orientation}&quot;)<br>    transform = nib.orientations.ornt_transform(cur_orientation, target_orientation)<br>    reoriented_data = nib.orientations.apply_orientation(image.get_fdata(), transform)<br>    return reoriented_data<br><br>target_orientation = (&#39;R&#39;, &#39;A&#39;, &#39;S&#39;)  #(&#39;L&#39;, &#39;I&#39;, &#39;A&#39;)<br>reoriented_data = reorient_image(MRI_nii, target_orientation)</pre><p>The new (target) orientation is now: <strong>[[0. 1.] [1. 1.] [2. 1.]].</strong></p><p>The full code is available on my <a href="https://github.com/MEGNEOrnela/My_medium_Posts/tree/main/Image%20Orientation">github</a>.</p><h3>Summary</h3><p>This post provides a straightforward guide to checking the orientation of a given 3D NIfTI image, adjusting it as needed, and plotting it in different planes.</p><p>Thanks for reading!!!</p><h3>References</h3><p>[1]: <a href="https://nipy.org/nibabel/coordinate_systems.html#coordinate-systems-and-affines">Coordinate systems and affines</a></p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06267dd854f8" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/image-orientation-in-3d-nifti-medical-imaging-06267dd854f8">Image Orientation in 3D NIfTI Medical Imaging.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gradient Underflow and Gradient Scaling]]></title>
            <link>https://ai.plainenglish.io/gradient-underflow-and-gradient-scaling-60f72976496a?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/60f72976496a</guid>
            <category><![CDATA[gradient-clipping]]></category>
            <category><![CDATA[gradient-underflow]]></category>
            <category><![CDATA[mixed-precision-training]]></category>
            <category><![CDATA[gradient-scaling]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Tue, 20 Aug 2024 22:08:06 GMT</pubDate>
            <atom:updated>2024-08-27T19:17:41.427Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve worked with deep neural networks using mixed precision training to speed up the training process. In this post, I will briefly explain what mixed precision training entails and how to maintain numerical stability during training.</p><h3>I. Definition</h3><h4><strong>1. Mixed precision training</strong> [1]</h4><p>Mixed precision training can be defined as a technique that combines different numerical formats (32-bit/float32/FP32 and 16-bit/float16/FP16) to optimize deep learning model training without sacrificing accuracy. It significantly reduces memory usage and accelerates complex operations. In PyTorch, this is automatically handled by the <strong>autocast()</strong> method, which selects the appropriate floating-point precision (either FP16 or FP32) for each operation in the neural network to keep the calculations accurate and fast.</p><p>However, using reduced precision (FP16) can lead to gradient underflow, affecting model performance.</p><h4><strong>2. Gradient underflow</strong></h4><p>It occurs when the weight gradients during backpropagation become too small to be represented in FP16, as any value smaller than 2⁻²⁴ becomes zero in FP16 [1].</p><p>To address this issue, <strong>Gradient Scaling</strong> can be used to dynamically adjust the scaling factor of the gradients during training.</p><h3>II. How does Gradient Scaling Works?</h3><p>Consider a simple neural network architecture with: one input x, one hidden layer with one neuron, and one output y (as shown in Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*4ip0z2JrFN2QntPutt5-3A.png" /><figcaption>Figure 1: Simple Neural Network Architecture.</figcaption></figure><h4>Forward pass:</h4><ul><li>a = 𝑓(z₁) = 𝑓(𝑤₁ . x), where 𝑤₁ is the weight between the input and the hidden neuron, and 𝑓 is the activation function (e.g., ReLU, etc.).</li><li>y = g(z₂) = g(𝑤₂ . a), where 𝑤₂ is the weight between the hidden neuron and the output, and g is the activation function of the output layer.</li></ul><p>Let’s denote the loss function by L(y, yₜᵣᵤₑ), where yₜᵣᵤₑ is the ground truth.</p><h4>Backward Pass using Chain Rule:</h4><ul><li>Gradient w.r.t 𝑤₂:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*tiyd4q64rbSsucS_JpAk5g.png" /><figcaption>Equation 1</figcaption></figure><ul><li>Gradient w.r.t 𝑤₁</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*cB6RGn9m--W1OA3V3oqPRA.png" /><figcaption>Equation 2</figcaption></figure><h4>Scale gradients by a factor λ:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/1*udu0Xj8CXLTjqo1poCAp9A.png" /><figcaption>Equation 3</figcaption></figure><p>The scaling factor ensures that the gradients are less likely to underflow.</p><h4>Unscale and update the weights:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*oqQXfu-lxM5KlSXOKnIftA.png" /><figcaption>Equation 4</figcaption></figure><p>Where α is the learning rate (same update rule for 𝑤₂). Unscaling before updating ensures that the weight updates are based on the original gradient values.</p><h3>III. Pseudo code based on PyTorch</h3><p>The general training loop in PyTorch looks like the code below:</p><pre># Initialize optimizer<br>optimizer = ...<br><br>for epoch in range(...):<br>    for input, target in dataloader:<br>        optimizer.zero_grad()<br><br>        # Forward pass <br>          output = model(input)<br>          loss = loss_function(output, target)<br>            <br>        # Backward pass<br>          loss.backward()<br><br>        # Update<br>          optimizer.step()</pre><p>With <strong>Automatic mixed precision training and the Gradient Scaling</strong>, the code becomes [2]:</p><pre># Initialize scaler, optimizer<br>scaler = torch.cuda.amp.GradScaler()<br>optimizer = ...<br><br>for epoch in range(...):<br>    for input, target in dataloader:<br>        optimizer.zero_grad()<br><br>        # Forward pass with autocasting <br>        with torch.cuda.amp.autocast():<br>            output = model(input)<br>            loss = loss_function(output, target)<br>            <br>        # Backward pass: scaled loss to create scaled gradients (Equation 1,2,3)<br>        scaler.scale(loss).backward()<br>        <br>        # Update the weights (Equation 4)<br>        scaler.step(optimizer) <br>        <br>        # Update the scaler for next iteration<br>        scaler.update()</pre><p>The <strong>scaler.step(optimizer)</strong>call implicitly does the following :</p><ul><li>Unscales the gradients.</li><li>Checks for Inf/NaN values to ensure that the gradients are within a reasonable range (if they aren’t, the optimizer step might be skipped to prevent bad updates).</li><li>Updates the model parameters using the unscaled gradients (Equation 4)</li></ul><p>Furthermore, if there is a need to perform custom operations like <strong>gradient clipping¹</strong> on the gradients before updating the weights, manual unscaling can be done using <strong>unscale_()</strong> method as follow:</p><pre># Initialize scaler, optimizer<br>scaler = torch.cuda.amp.GradScaler()<br>optimizer = ...<br><br>for epoch in range(...):<br>    for input, target in dataloader:<br>        optimizer.zero_grad()<br><br>        # Forward pass with autocasting<br>        with torch.cuda.amp.autocast():<br>            output = model(input)<br>            loss = loss_function(output, target)<br>            <br>        # Backward pass: scaled loss to create scaled gradients<br>        scaler.scale(loss).backward()<br>        <br>        # Unscale gradients explicitly<br>        scaler.unscale_(optimizer)<br><br>        # (Optional) Gradient clipping <br>        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)<br><br>        # Update the weights with unscaled gradients<br>        scaler.step(optimizer)<br><br>        # Update the scaler<br>        scaler.update()</pre><p>The <strong>unscale_()</strong> method provides explicit control over when gradients are unscaled.</p><p><strong>¹</strong><em>Gradient clipping is a technique used to prevent exploding gradient by clipping the gradients to a maximum threshold value.</em></p><h3>Summary</h3><p>This post briefly discusses the mixed precision training and how to manage gradients to maintain stable learning.</p><p>I hope this has been helpful!</p><h3>References</h3><p>[1]: <a href="https://arxiv.org/abs/1710.03740">Mixed Precision Training</a></p><p>[2]: <a href="https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html">Automatic Mixed Precision</a></p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60f72976496a" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/gradient-underflow-and-gradient-scaling-60f72976496a">Gradient Underflow and Gradient Scaling</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dice Score, False Positive Volume, and False Negative Volume]]></title>
            <link>https://medium.com/@pmegne/dice-score-false-positive-volume-and-false-negative-volume-16e67462f89c?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/16e67462f89c</guid>
            <category><![CDATA[false-negative]]></category>
            <category><![CDATA[dice-score]]></category>
            <category><![CDATA[false-positive]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Thu, 25 Jan 2024 12:56:27 GMT</pubDate>
            <atom:updated>2024-01-25T12:56:27.182Z</atom:updated>
            <content:encoded><![CDATA[<p>As indicated in a <a href="https://medium.com/ai-in-plain-english/prostate-mri-segmentation-fda4e35704de">previous post</a>, dice scores, which measure the overlap between the ground truth and the predicted mask, can serve as a metric for assessing a segmentation model. Nevertheless, relying solely on the dice score does not distinguish between the volumes of false positive and false negative connected components.</p><p>In this post, we are going to define and illustrate what the terms false positive and false negative volumes refer to.</p><h4>1. False positive volume</h4><p>It corresponds to the volume of incorrectly identified connected components in the predicted segmentation mask that do not overlap with the actual structures in the ground truth. For example, in the context of tumor lesion segmentation, it signifies the volumes erroneously marked by the algorithm as tumors, when they are, in fact, normal structures like the brain or kidneys.</p><h4>2. False negative volume</h4><p>It represents the volume of connected components in the ground truth segmented mask that do not overlap with the predicted mask. In order words, it can be defined as the volume occupied by the tumor lesions that the segmentation algorithm failed to identify.</p><p>Consider the ground truth and the predicted segmentation mask of a given image as show in figure 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*omOJ2HGad2nJMLT6" /><figcaption>Figure 1: Ground truth vs predicted segmentation mask</figcaption></figure><ul><li><strong>TP</strong> is the true positive volume which refers to the overlaping volume between the predicted and ground truth mask.</li><li><strong>TN</strong> is the true negative volume representing the background</li><li><strong>FP</strong> is the false positive volume</li><li><strong>FN</strong> is the false negative volume</li></ul><p>For the remainder of this post, we will focus on 2D slice images. Therefore, the term ‘area’ will be used instead of ‘volume,’ as ‘volume’ is specific to 3D space.</p><h4>3. Illustration</h4><p>We refer to the results obtained in a <a href="https://medium.com/ai-in-plain-english/prostate-mri-segmentation-fda4e35704de">previous post</a> using a dataset sourced from the <a href="http://medicaldecathlon.com/">Medical Segmentation Decathlon</a> (MSD), a biomedical image analysis challenge. As outlined in that post, a U-Net architecture was employed to predict the segmented mask divided into three classes: “background”, “PZ” (peripheral zone), and “TZ” (transitional zone). The best-saved model based on the highest validation dice score is used to generate masks for the test set.</p><p>For a given (prostate) Magnetic Resonance/MR image, we will select one 2D slice and compute the areas corresponding to false positive and false negative.</p><p>To achieve this, we use the Python library <strong>cc3d <em>[1]</em></strong>, an implementation of connected components in 3D and 2D that employs a neighborhood relationship called connectivity. Essentially, connectivity is a numerical value (varying between 2 and 26) that determines which neighboring voxels/pixels should be considered when deciding if they are part of the same connected component <strong><em>[1]</em></strong>. The choice of connectivity affects how connected components are identified.</p><p>Let’s look at figure 2, which shows a 2D slice of a given prostate MR image as well as the ground truth and the predicted mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*JR595Hrd3odzaoy5gLsmkg.png" /><figcaption>Figure 2: 2D slice of a prostate MR image (left), segmented mask (middle), and predicted mask (right)</figcaption></figure><p>By following the steps and subsequently calling the defined functions below <strong><em>[2]</em></strong>, we are going to compute the dice score, false positive and false negative areas.</p><p><strong>a. Setup</strong></p><p>Install the ccd3 library:</p><pre>!pip install connected-components-3d</pre><p>Import some useful libraries:</p><pre>import cc3d<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from mpl_toolkits.mplot3d import Axes3D</pre><p><strong>b. Define the functions</strong></p><pre>def false_positive_volume(gt_array, pred_array):<br><br>    &#39;&#39;&#39;<br>    This function computes the number of pixels/voxels of false positive connected components in prediction mask<br>    - inputs: a multiclass (3) segmentation array (gt_array) and predicted mask array (pred_array)<br>    - output: the corresponding area/volume<br>    - cc3d.connected_components: analyzes the input array and assigns a unique label to each connected component.<br>    it outputs an array where each pixel&#39;s (voxel&#39;s) value correspond to the label of the connected component which it belongs to<br>    <br>    &#39;&#39;&#39;<br><br>    pred_conn_comp = cc3d.connected_components(pred_array, connectivity=8)<br>    false_pos = 0<br>    for idx in range(1, pred_conn_comp.max() + 1):<br>        comp_mask = np.isin(pred_conn_comp, idx)<br>        if (comp_mask * gt_array).sum() == 0:<br>            false_pos = false_pos + comp_mask.sum()<br>    return false_pos<br><br>def false_negative_volume(gt_array, pred_array):<br><br>    &#39;&#39;&#39;<br>    This function computes the number of pixels/voxels of false negative connected components in prediction mask<br>    &#39;&#39;&#39;<br><br>    gt_conn_comp = cc3d.connected_components(gt_array, connectivity=8)<br>    false_neg = 0<br>    for idx in range(1, gt_conn_comp.max() + 1):<br>        comp_mask = np.isin(gt_conn_comp, idx)<br>        if (comp_mask * pred_array).sum() == 0:<br>            false_neg = false_neg + comp_mask.sum()<br>    return false_neg<br><br>def dice_score(pred, mask, num_classes=3):<br><br>    &#39;&#39;&#39;<br>    This function computes the Dice score for each class and averages them to obtain the overall score for a given predicted mask<br>    &#39;&#39;&#39;<br><br>    smooth = 1e-8<br>    dice_scores = []<br><br>    for class_id in range(num_classes):<br>      mask_class = (mask == class_id).astype(int)<br>      pred_class = (pred == class_id).astype(int)<br>      intersection = 2.0 * (pred_class * mask_class).sum()<br>      union = pred_class.sum() + mask_class.sum() +smooth<br>      <br>      dice = intersection / union<br>      dice_scores.append(dice)<br><br>    return np.mean(dice_scores)</pre><p><strong>c. Call the functions</strong></p><pre>false_pos_vol = false_positive_volume(mask, pred_mask)<br>false_neg_vol = false_negative_volume(mask, pred_mask)<br>dice_score = dice_coef_metric(pred_mask,mask)<br><br>print(f&quot;Dice score: {dice_score}&quot;)<br>print(f&quot;False Positive area: {false_pos_vol} pixels&quot;)<br>print(f&quot;False Negative area: {false_neg_vol} pixels&quot;)<br><br>&gt;&gt;&gt; results:<br>Dice score: 0.7321941259313943<br>False Positive area: 16 pixels<br>False Negative area: 0 pixels</pre><p>The false positive area is equal to 16, while the false negative area is 0. On one side, the dice score value of 0.73 suggests a good overlap between the predicted and actual segmented regions. However, a false positive area of 16 indicates regions where the algorithm falsely detects structures not present in the ground truth.</p><h4>Summary</h4><p>This blog defines and illustrates the computation of false positive and false negative volumes/areas, which are also used to evaluate segmentation models.</p><h4>References</h4><p>[1]: <a href="https://github.com/seung-lab/connected-components-3d">connected-components-3d</a></p><p>[2]: <a href="https://github.com/lab-midas/autoPET">autoPET</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16e67462f89c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DICOM to NIfTI Conversion]]></title>
            <link>https://medium.com/@pmegne/dicom-to-nifti-conversion-dc436c0b5fa6?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc436c0b5fa6</guid>
            <category><![CDATA[dicom]]></category>
            <category><![CDATA[dicom2nifti]]></category>
            <category><![CDATA[image-conversion]]></category>
            <category><![CDATA[nifti]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Sun, 14 Jan 2024 17:42:11 GMT</pubDate>
            <atom:updated>2024-01-14T17:42:11.959Z</atom:updated>
            <content:encoded><![CDATA[<p>Many frameworks for medical image analysis may require a specific format such as NIfTI (Neuroimaging Informatics Technology Initiative). However, the raw data from scanners is often in DICOM (Digital Imaging and Communications in Medicine) format, so it needs to be first converted to the appropriate format to facilitate its utilization.</p><p>In this post, we are going to write a simple script that uses the Python library <strong><em>dicom2nifti</em></strong> to demonstrate how to convert DICOM format to NIfTI format.</p><h4>1. Dataset</h4><p>The data used is an MRI (Magnetic Resonance Imaging) DICOM data set of the head of a normal male human aged 52. It can be downloaded <a href="https://zenodo.org/records/16956#.YFMM5PtKiV5">here</a>.</p><p>The DICOM images correspond to a series of 2D slices, each representing a cross-sectional view of the head of that person. These slices are then organized in a specific order to reconstruct the entire 3D volume of the head. The NIfTI format is a common way to store and represent such 3D medical imaging data, where each slice is an integral part of the overall volume.</p><h4>2. Conversion script</h4><p>The function “dcm2nii” below will convert DICOM files from a directory to NIfTI format and save the resulting file in a specified output path.</p><pre>def dcm2nii(MRI_dcm_path, nii_out_path):<br><br>    &#39;&#39;&#39;<br>    - MRI_dcm_path: Path to the directory containing MRI DICOM files<br>    - nii_out_path: Path to the directory where the resulting NIfTI file will be saved.<br>    - tempfile.TemporaryDirectory(): create a temporary directory (tmp) that will be automatically<br>      deleted when the block of code inside it finishes execution<br>    - dicom2nifti.convert_directory: convert the DICOM file to NIfTI format.<br>    &#39;&#39;&#39;<br><br>    with tempfile.TemporaryDirectory() as tmp:<br>        tmp = Path(str(tmp))<br><br>        # convert dicom directory to nifti<br>        dicom2nifti.convert_directory(MRI_dcm_path, str(tmp),<br>                                      compression=True, reorient=True)<br><br>        #looks for the first NIfTI file (*nii.gz) in temp<br>        nii = next(tmp.glob(&#39;*nii.gz&#39;))<br><br>        # copy nifti file to the specified output path and named it &#39;MRI.nii.gz&#39;<br>        shutil.copy(nii, nii_out_path+&#39;MRI.nii.gz&#39;)</pre><h4>3. Read and plot slices from the converted file</h4><pre>import nibabel as nib<br><br>path = &quot;./MRI.nii.gz&quot;<br>MRI_nii = nib.load(path)<br><br>#Get the pixel data<br>mri = MRI_nii.get_fdata()<br>print(mri.shape)</pre><p>The output is <strong>(256, 256, 27)</strong>, where each number refers to the height, width and the number slices, respectively.</p><pre>#Visualize some slices (starting from the 5th one)<br>def plot(no_):<br>  plt.figure(figsize=(15,10))<br><br>  for i in range(1,no_+1):<br>    plt.subplot(1,no_,i)<br>    plt.imshow(mri[:,:,5+i])<br>    plt.axis(&quot;off&quot;)<br><br>  plt.show<br><br>plot(4)</pre><p>The output is shown in figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2TGIFKY8EJL_mm9YzfOFPQ.png" /><figcaption>Figure 1: Visualization of some MRI slices.</figcaption></figure><p>The whole script can be find in my <a href="https://github.com/MEGNEOrnela/My_medium_Posts/blob/main/convert_dicom2nifti.ipynb">github</a>.</p><h4>Summary</h4><p>This post primarily shows how to convert DICOM images from a specified directory to a NIfTI format.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc436c0b5fa6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prostate MRI Segmentation]]></title>
            <link>https://ai.plainenglish.io/prostate-mri-segmentation-fda4e35704de?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fda4e35704de</guid>
            <category><![CDATA[dice-loss]]></category>
            <category><![CDATA[image-segmentation]]></category>
            <category><![CDATA[multiclass-classification]]></category>
            <category><![CDATA[mri-scan]]></category>
            <category><![CDATA[unetwork]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Tue, 09 Jan 2024 11:09:00 GMT</pubDate>
            <atom:updated>2024-01-23T11:12:40.003Z</atom:updated>
            <content:encoded><![CDATA[<p>Image segmentation is a crucial aspect of medical image analysis. In the context of prostate Magnetic Resonance Imaging (MRI), it relies on MRI to identify or delineate suspicious regions in MR images that may indicate the presence of cancer. Manual segmentation is however laborious, time-consuming, and limited to experts. Automating this process proves beneficial by extracting valuable features, assisting clinicians in making more accurate diagnoses.</p><p>In this post we are going to implement prostate MRI segmentation using U-net architecture.</p><h4>1. Brief introduction to Magnetic Resonance Imaging (MRI)</h4><p>In simple terms, MRI is a medical imaging technique that uses strong magnets and radio waves to create detailed images of many internal structures in a human body, including the organs, bones, muscles and blood vessels .</p><p><strong>Operating principle of MRI</strong><br>An MRI machine, often called MRI scanner, resembles a cylindrical tube with powerful magnets. These magnets generate a strong magnetic field around the body. Additionally, the MRI scanner emits non-harmful radio waves that interact with the atoms inside the body. Since the human body consists mostly of water, and water is composed of minuscule particles known as atoms, these radio waves, when activated, cause the atoms to change their orientation. When the radio waves are switched off, the atoms gradually return to their original positions. As they do so, they release energy, and the MRI machine captures this energy to create images. It’s important to note that MRI does not involve radiation, making it a safe imaging technique for most individuals [1].</p><h4>2. MRI images segmentation using U-net architecture</h4><p>U-net architecture [2] which owes its name to its U shape is a Deep Learning architecture used for semantic segmentation tasks in image analysis. It is made up of two paths namely contracting path or encoder and the expansive path or decoder.</p><p><strong>a. Contracting path or encoder</strong></p><p>In this path, convolution layers are used to extract features from the input image. It consists of two convolutions layers followed by a max-pooling layer; each convoluted layer is followed by an activation function (rectified linear unit/ReLU). The size of the input image gradually decreases (spatial resolution of the feature maps reduces) while the depth increases.</p><p><strong>b. Expansive path or decoder</strong></p><p>In the decoder, transpose convolution layers are used to upsample the feature maps to the original size of the input image, while incorporating or preserving the spatial information from the encoder. It consists of an upsampling layer (ConvTranspose2d), a concatenation with the corresponding encoder feature map, and two convolutional layers, separated by the activation function. At the final layer a 1x1 convolution is used to map each feature vector to the desired number of classes. The size of the image gradually increases while the depth decreases.</p><h4>3. Methodology</h4><p>To implement prostate MRI images segmentation , a series of steps has been performed.</p><p><strong>1. Data preparation</strong></p><p>The dataset used contains prostate MR images along with the corresponding masks obtained from Medical Segmentation Decathlon (MSD), a biomedical image analysis challenge [3] . All images are provided in <em>.nii</em> format with 2 channels per image. The dataset consists of 33 training samples (images + masks) and 16 testing samples (images only), wherein each 3D MRI of a given patient refers to many 2D slices. The labels or masks are divided into three classes: 0: “background”, 1: “PZ” (peripheral zone), 2: “TZ” (transitional zone).<br>A medical image framework called <a href="https://monai.io/"><strong>Monai</strong></a> has been employed for image pre-processing. It includes diverse transformations like scaling, cropping, flipping, rotation, changing the image intensity or contrast, and more. Further details can be found in the <a href="https://docs.monai.io/en/0.3.0/transforms.html"><strong>documentation</strong></a>. Each image has been reshaped to dimensions (2,11, 256, 256), where 2 is the number of channels, 11 represents the number of slices, and (256, 256) denotes the height and width, respectively.</p><p>Figure 1 is a gif that displays all the slices of a given prostate MR image scan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Re-h11ly_ol-kS7EyKZyKA.gif" /><figcaption>Figure 1: Gif created from a training sample of a prostate MRI in nii.gz format.</figcaption></figure><p><strong>2. Model building</strong></p><p>A Unet-architecture has been built according to the description provided in part 2. Additionally, to assess the model’s performance or its attempt to fit the training sample, the dice coefficient has been employed as both a metric and a loss function, combined with negative log likelihood/binary cross-entropy.</p><ul><li><strong>Dice coefficient:</strong> it measures the overlap between predictions and the ground truth (segmentation masks). It is calculated as two times the intersection between the ground truth and the predicted mask, divided by the sum of the ground truth and the predicted mask (as shown in equation 1). The dice coefficient ranges from 0 to 1, where 0 indicates no intersection between the ground truth and the prediction, and values close to 1 indicate accurate predictions by the model. As the intersection increases, the dice coefficient also increases.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*JR_U_mlzs1h-blwv" /><figcaption>Equation 1</figcaption></figure><ul><li><strong>Dice loss</strong>: it can be derived as 1 minus the dice coefficient, which implies that as the dice coefficient increases, the dice loss decreases. When the dice coefficient reaches its maximum value, the loss becomes 0. This relationship is designed so that a smaller dice loss corresponds to better performance, aligning with the goal of minimizing the loss function.</li><li><strong>Binary Cross Entropy/ BCE loss:</strong> it measures the pixel-wise similarity between the predicted and true masks.</li><li><strong>By combining these two losses</strong>, the BCE loss will help with pixel-wise binary classification, while the Dice loss will focus on achieving a good overlap between the predicted and ground truth masks.</li></ul><p><strong>Note:</strong> Since we are dealing with a multi-class scenario, the output of the last convolutional layer in the Unet-architecture has been passed through a softmax function to obtain the logits as show in the figure 2 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JXpT5Ed11oxZZABUiXPKng.png" /><figcaption>Figure 2: Straightfoward steps of the segmentation workflow.</figcaption></figure><h4>4. Results</h4><p>The model has been trained using the hyper-parameters below:</p><ul><li>Number of epoch = 30</li><li>Batch size = 4</li><li>Learning rate = 1e-5</li><li>Optimizer= Adam</li></ul><p>During training, a batch of 2D segmented slices is passed through the model at the time. This approach is used because 2D convolution layers are employed to train on a 3D image. For instance, one image of size ( 2,11, 256, 256) is referred to as a batch of 11 2D images, each with a shape of (2, 256, 256), where 2 represents the number of channels.</p><p>The training loss and validation loss are shown in figure 3 followed by visualizations of some predictions in figures 4 and 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*U1vrw5czbHwFf8CD6fbDmw.png" /><figcaption>Figure 3: Train and validation losses</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*hZxYmjNEZpxDb6MqXaQwOg.png" /><figcaption>Figure 4: Predictions for some validation samples.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jZfvS1i7vGbydB-ejsoxsg.png" /><figcaption>Figure 5: Predictions for some test samples.</figcaption></figure><p>We can observe from figures 4 and 5 that the model attempted to predict the three classes (peripheral zone/PZ, transitional zone/TZ, and the background). During training, the best model based on the highest validation dice coefficient was saved and later used to generate masks for the test samples. However, several techniques an approaches need to be explored to enhance the similarities between the ground truth masks and the predictions.</p><p>The code is available on my <a href="https://github.com/MEGNEOrnela/My_medium_Posts/tree/main/Image_segmentation">github</a>.</p><h4>Summary</h4><p>This work briefly presents and implements the key steps behind MRI segmentation, specifically focusing on segmenting a prostate MRI. Image segmentation is considered extremely important for effectively analyzing a patient’s image. Using Deep Learning to automate this process can be useful and beneficial in assisting clinicians in making decisions.</p><h4>References</h4><p>[1]: <a href="https://www.researchgate.net/profile/Dr-Shireen/publication/279471369_Magnetic_resonance_imaging_MRI_-_A_review/links/5cd61f53299bf14d9589b943/Magnetic-resonance-imaging-MRI-A-review.pdf">Magnetic Resonance Imaging (MRI) — A Review</a></p><p>[2]: <a href="https://arxiv.org/pdf/1505.04597.pdf%EF%BC%89">U-Net: Convolutional Networks for Biomedical Image Segmentation</a></p><p>[3]: <a href="http://medicaldecathlon.com/">Medical Segmentation Decathlon</a></p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fda4e35704de" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/prostate-mri-segmentation-fda4e35704de">Prostate MRI Segmentation</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vocal Biomarker to Predict Fatigue in People with COVID-19]]></title>
            <link>https://ai.plainenglish.io/vocal-biomarker-to-predict-fatigue-in-people-with-covid-19-16db2774c001?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/16db2774c001</guid>
            <category><![CDATA[mel-spectrogram]]></category>
            <category><![CDATA[voice-feature-extraction]]></category>
            <category><![CDATA[librosa]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mfcc]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Mon, 30 Oct 2023 08:55:03 GMT</pubDate>
            <atom:updated>2023-11-01T02:45:48.910Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xsrPV2jaiKnzcycC" /><figcaption>Photo by <a href="https://unsplash.com/@alinnnaaaa?utm_source=medium&amp;utm_medium=referral">Alina Grubnyak</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>In this post, we’ll delve into the world of vocal biomarkers and how they can be harnessed to predict fatigue in people with COVID-19. Our primary focus is not on finding the perfect machine learning model but on understanding the crucial steps involved in moving from raw audio data to feature extraction and, finally, building a predictive model.</p><p>This blog post is inspired by the amazing work accomplished in [1], and the dataset we will be using for illustration is sourced from the same paper. The data is obtained from the <a href="https://bmjopen.bmj.com/content/10/11/e041834?ijkey=8d5daea966956c07073d0abd62ca973760e19d30&amp;keytype2=tf_ipsecsha"><em>Predi-COVID study</em></a> conducted in Luxembourg.</p><h4>1. What is a vocal biomarker?</h4><p>In simple terms, a biomarker is like a signpost for doctors to understand a person’s health. It’s a medical indicator that reveals what’s happening inside a patient’s body. Vocal biomarkers are medical indicators derived from voice characteristics.</p><h4>2. Voice/audio data analysis</h4><p>An audio data is a digital version of sounds we hear, it’s in a format that computers can understand and work with. A sound has three main characteristics named time (in second), amplitude (sound intensity in decibel/db), and frequency (number of vibrations per second in hertz).</p><p>Before moving forward, some keywords will be defined for easy follow-up.</p><h4><strong>a. Waveform</strong></h4><p>A waveform is a visual representation of an audio signal that illustrates how the signal’s amplitude changes over time, with time displayed on the x-axis and amplitude on the y-axis (Figure 1). However, it doesn’t provide information about the frequency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*baUFPi1nTsVzrBmFMmhbbQ.png" /><figcaption>Figure 1: Waveform of a voice from a random participant. Imaged by the author.</figcaption></figure><h4><strong>b. Spectrum</strong></h4><p>A spectrum is an alternative representation of sound, focusing on frequency (Figure 2). It provides information about the pitch of the sound, but it does not convey when the sound occurred (misses time component). To transition from the time domain to the frequency domain, a mathematical function known as the Fourier transform (FT) is used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*jPjxAMtZXipG-VChmFxIaA.png" /><figcaption>Figure 2: Spectrum of a voice from a random participant, calculated from a windowed segment of the audio with a size of 2048. Imaged by the author.</figcaption></figure><h4><strong>c. Spectrogam</strong></h4><p>A spectrogram is a detailed representation that covers the three characteristics of a sound. It depicts time on the x-axis, frequency on the y-axis, and represents amplitude through color as shown in Figure 3. The color is brighter for louder sounds, and silence is indicated by a black color. In order to generate a spectrum of a sound signal when varying over time, a sequence of Fourier transforms is applied on several windowed segments or portions of the sound signal: it is called <strong>short-time Fourier transform</strong> (<strong>STFT</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*nJPp9TibwmOZdW5r2Kv15Q.png" /><figcaption>Figure 3: Spectrogram of a voice from a random participant. Imaged by the author.</figcaption></figure><p>In contrast, the human auditory system is more sensitive to detecting differences in lower frequencies as opposed to higher frequencies. As a result, these frequencies are transformed into the human hearing range, known as the <a href="https://en.wikipedia.org/wiki/Mel_scale"><strong>Mel scale</strong></a>, leading to the <strong>Mel spectrogram</strong>.</p><blockquote>All these transformations (FT, STFT) can be done automatically with python using many libraries including <a href="https://librosa.org/doc/latest/index.html"><strong>librosa</strong></a>.</blockquote><p>The primary steps for voice analysis can be summarized as follows:</p><ul><li>Voice data collection</li><li>Data preprocessing</li><li>Feature extraction</li><li>Selection and training of a machine learning model based on the extracted features to make predictions.</li></ul><p>As previously mentioned in the introduction, the data utilized here is accessible in [1] and the preprocessing has been conducted by the authors.</p><h4>Feature Extraction</h4><p>Audio features are characteristics of signals derived from visualizations of preprocessed audio data. These features can fall into three domains (more details in [2]), but we will focus on time-frequency domain features.</p><p>As the name suggests, time-frequency domain features combines both time and frequency components, with the commonly used representation being <strong>Mel-frequency cepstral coefficients</strong> (<strong>MFCCs</strong>). MFCCs are derived from the Mel spectrogram through a series of mathematical transformations, including the application of the Discrete Cosine Transform (DCT). The resulting MFCCs provide compact representations of the audio signal’s spectral characteristic (see [3] for more details).</p><h4>Training Machine Learning models</h4><p>The extracted features can be used to detect fatigue in a person with COVID-19. To achieve this, two machine learning models, namely the Support Vector Machine (SVM) and K-Nearest Neighbors (KNN), have been trained and tested on a subset of 110 voice recordings, encompassing two distinct types of recordings from each individual. Further details regarding data collection and preprocessing can be found in [1].</p><p>A big picture of the entire process from raw data to the classification is shown in Figure 4 and the notebook for the entire implementation is available in my <a href="https://github.com/MEGNEOrnela/My_medium_Posts/blob/main/Voice_features_extraction.ipynb"><strong>github</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*LsOoDsm9yYey1c-5kQMJSg.png" /><figcaption>Figure 4: Big picture of vocal biomarkers predicting fatigue. Imaged by the author.</figcaption></figure><h3>Summary</h3><p>We’ve provided a brief overview of the steps involved in analyzing voice data and training a machine learning model for detecting fatigue in individuals with COVID-19. This serves as an introductory glimpse into the concept of vocal biomarkers.</p><p>Thanks for reading!</p><h4>References</h4><p>[1]: <a href="https://bmjopen.bmj.com/content/12/11/e062463#ref-20">Vocal biomarker predicts fatigue in people with COVID-19: results from the prospective Predi-COVID cohort study</a></p><p>[2]: <a href="https://www.altexsoft.com/blog/audio-analysis/">Audio Analysis With Machine Learning: Building AI-Fueled Sound Detection App</a></p><p>[3]: <a href="https://ieeexplore.ieee.org/abstract/document/9955539">Mel Frequency Cepstral Coefficient and its Applications: A Review</a></p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li><li><em>Check out our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a>, <a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a>, <a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16db2774c001" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/vocal-biomarker-to-predict-fatigue-in-people-with-covid-19-16db2774c001">Vocal Biomarker to Predict Fatigue in People with COVID-19</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image Captioning]]></title>
            <link>https://medium.com/@pmegne/image-captioning-5162e22ef2ac?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5162e22ef2ac</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[image-captioning]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Tue, 06 Jun 2023 08:37:16 GMT</pubDate>
            <atom:updated>2023-06-06T17:32:16.805Z</atom:updated>
            <content:encoded><![CDATA[<p>In the realm of multimodal learning, image captioning stands as a process that involves the generation of meaningful textual descriptions or captions corresponding to given images. Unlike the task of assigning class labels to images (image classification), image captioning combines two distinct modalities, namely images and texts used in computer vision and natural language processing (NLP) respectively. In this blog post, we embark on an exploration of the captivating domain of image captioning, specifically focusing on the utilization of Contrastive Language-Image Pre-training (CLIP)[<a href="https://arxiv.org/abs/2111.09734?ref=louisbouchard.ai">1</a>].</p><p>This blog post can be considered as a synopsis of the article [<a href="https://arxiv.org/abs/2111.09734?ref=louisbouchard.ai">1</a>] which incorporates some terminologies and concepts outlined in the paper.</p><p>As mentioned in the paper [1], the key idea of image captioning is to obtain a wide understanding of both visual and textual data. The whole architecture can be divided into three main modules namely CLIP’s encoding, mapping network, and the pre-trained language model as shown in figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qB8Ukm1v2pO9bJMqCC438Q.png" /><figcaption>Figure 1: Bref overview of the image captioning architecture. Image by the author.</figcaption></figure><p>Let’s break down each module for a better understanding.</p><h4>1. CLIP encoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*XRTJ3RsAYqzVlbSrmr-Krg.png" /><figcaption>Figure 2: CLIP’s architecture. Image by the author.</figcaption></figure><p>Developed by OpenAI in 2021 [2], CLIP jointly trains an image encoder and a text encoder to predict the correct image — text pair within a batch. During training, the image and text encoder produce respectively image embeddings (<em>I1</em> and <em>I2</em>) and text embeddings (<em>T1</em> and <em>T2</em>) in a way to maximize the cosine similarities between correct image-text pairs (&lt;<em>I1-T1</em>&gt;, and &lt;<em>I2-T2</em>&gt;) while minimizing the cosine similarities for dissimilar pairs (non-diagonal elements), in a contrastive fashion. The image encoder is a computer vision backbone such as ResNet, or Vision Transformer (ViT)[4] and the text encoder is a transformer [5].</p><p>The CLIP architecture has been trained on a dataset of 400 million image-text pairs collected from the internet. CLIP has a special ability called “zero-shot image classification”, whereby it can classify images even without specific training on those exact images. This means that CLIP possesses a generalized understanding of different objects or concepts and can apply this knowledge to classify unseen images. To illustrate this, the authors of [2] compare a ResNet-101 model trained specifically on ImageNet dataset to CLIP when both are evaluated on other datasets derived from ImageNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A23rbW3x3HtEiDj3rastdw.png" /><figcaption>Figure 3: The performance of best zero-shot CLIP model is compared with the performance of ResNet-101. Image from the paper [2].</figcaption></figure><p>The authors observed that the performance of ResNet-101, trained specifically on the ImageNet dataset, significantly lags behind CLIP when evaluated on comparable datasets for similar tasks.</p><h4>2. Language model</h4><p>To generate the image caption itself, a textual decoder is utilized. In the research study [1], the authors employed a state-of-the-art language model known as Generative Pre-trained Transformer 2 (GPT-2)[<a href="https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf">3</a>]. GPT-2 is a transformer-based model that has been pre-trained on a large corpus of text data and is capable of generating coherent and contextually relevant textual outputs. By leveraging the power of GPT-2, the researchers were able to generate meaningful captions that describe the content of the given image.</p><h4>3. Mapping network</h4><p>The mapping network acts as a bridge between the CLIP encoding and the GPT-2 language model, helping map the visual-textual representation of CLIP to the GPT-2 representation space. The researchers explored two different strategies for this purpose.</p><ul><li>Firstly, they employed a simple Multi-Layer Perceptron (MLP) approach, where the language model is fine-tuned using the CLIP’s embedding as a guide to orient the text generation process.</li><li>Alternatively, to mitigate the increase in trainable parameters resulting from fine-tuning the language model, the researchers opted to freeze the language model and utilize the transformer architecture as the mapping network. In this configuration, only the mapping network is trained.</li></ul><p>The experiments revealed that the second approach, utilizing the transformer-based mapping network while freezing the language model, achieved superior results compared to the MLP-based approach in certain scenarios.</p><h4>Simple mathematical description of the task</h4><p><strong>Problem statement:</strong> Given a dataset of paired images and captions/texts <em>{x^i, c^i}</em>, where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*aHSuvtShEmUnaxYIiw4OGg.png" /></figure><p>is a set of tokens of a fixed length <em>l</em>, the objective is to learn the generation of a meaningful caption for an unseen input image.<br>The training objective is defined as maximizing the likelihood of generating the target caption <em>c^i</em> given the input image <em>x^i</em> and a model parameterized by <em>θ</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*gdUki0jR58j0fpydRS8yRw.png" /></figure><p>Since GPT-2 is an autoregressive language model, it generates text one token at a time based on the previously generated tokens. Therefore, the training objective can be reformulated as maximizing the conditional probability of generating the current token <em>c_i</em> given the previously generated tokens and the input image <em>x_i</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*ACMWdsPHbgHGp48-SCZQtA.png" /></figure><p>For more details consult the paper [1].</p><h4>Some results</h4><p>Below are several inference cases that were tested within a <a href="https://replicate.com/rmokady/clip_prefix_caption">web-based</a> environment. The code of the paper is available <a href="https://github.com/rmokady/CLIP_prefix_caption">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*GYwx_WT7Xl25mGfEBhMAag.jpeg" /><figcaption><strong>Output: silhouette of a girl on the beach at sunset</strong>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*C_KAgc_5z57QQAiGxkjQUQ.jpeg" /><figcaption><strong>Output: ice cream in a glass bowl.</strong></figcaption></figure><h3>Conclusion</h3><p>We have explained the concept of image captioning using the Contrastive Learning Image Pre-training (CLIP) architecture. By harnessing the capabilities of CLIP and GPT-2, multimodal learning can extract informative insights from images. The field of image captioning holds immense potential for real-life applications. For instance, real-time provision of textual descriptions during video calls and multimedia presentations for individuals with hearing impairments would enable them appreciate and understand the content.</p><p>Thank you for taking the time to read this article! Hope that it has been both inspiring and insightful.</p><h4>References</h4><p>[1]: <a href="https://arxiv.org/abs/2111.09734?ref=louisbouchard.ai">ClipCap: CLIP Prefix for Image Captioning</a></p><p>[2]: <a href="https://arxiv.org/abs/2103.00020">Learning Transferable Visual Models From Natural Language Supervision</a></p><p>[3]: <a href="https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf">Improving Language Understanding by Generative Pre-Training</a></p><p>[4]: <a href="https://arxiv.org/abs/2010.11929">An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</a></p><p>[5]: <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a></p><h4><a href="https://mlearning.substack.com/about">BECOME a WRITER at MLearning.ai</a> // <a href="https://open.substack.com/pub/mlearning/p/new-career-xr-vr-ar-reality-pro-headset-jobs-3d?r=z7zu8&amp;utm_campaign=post&amp;utm_medium=web">new career</a> // <a href="https://open.substack.com/pub/evartology/p/super-cheap-ai-llm-one-person-startups-examples?r=9hp4d&amp;utm_campaign=post&amp;utm_medium=web">Super Cheap AI.</a></h4><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5162e22ef2ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Double Descent Phenomenon]]></title>
            <link>https://medium.com/@pmegne/double-descent-phenomenon-f3020172c99f?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3020172c99f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[double-descent]]></category>
            <category><![CDATA[bias-variance-tradeoff]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 11:32:25 GMT</pubDate>
            <atom:updated>2023-12-21T22:10:46.215Z</atom:updated>
            <content:encoded><![CDATA[<p>Overall , Machine learning’s main objective is to find a tradeoff between the model’s ability to fit the training data and its ability to generalize to new data. This is the so-called bias-variance tradeoff. In this blog we will talk a bit about the bias-variance tradeoff and drop on double descent phenomenon.</p><h4><strong>Bias-variance tradeoff derivation (for regression)</strong></h4><p>Let:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*ymVkxuOUtki38quXENHJww.gif" /><figcaption>Equation 1</figcaption></figure><p>Where <em>S</em> is the training data and <em>h*</em> the absolute model or the best model obtained if we had access to the distribution of the population.</p><ul><li><em>h_s,</em> the model obtained after training on <em>S</em>.</li><li><em>(x,y)</em> a test example such that :</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*wDeeHvu79AVRgF4A0qE0aQ.gif" /><figcaption>Equation 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/124/1*fR772IVQdHeUHPDHr0itYw.gif" /><figcaption>Equation 3</figcaption></figure><ul><li><em>h_avg</em>, the best model in the hypothesis space.</li></ul><p>The expected error/loss is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*msimT9jdo1YVNYuTs-oE3Q.gif" /></figure><p>We will add the term <em>h_avg</em> and remove it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C0occ3UAmcU4DsxQ7BKY-A.gif" /></figure><p>Where,</p><ul><li><em>𝞂^2</em> is the irreducible or unavoidable error due to the measurement.</li><li>The <strong>bias</strong> is the difference between the best model in the hypothesis class (<em>h_avg</em>) and the best possible model (<em>h*</em>). It represents the error due to the lack of expressivity of the model.</li><li>The <strong>variance</strong> is the error due to the randomness of the data. It’s the difference between the best model we find (<em>h_s</em>) and the best model in the hypothesis class (<em>h_avg</em>).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*tPNPD3fFe3MOQkd1MHJ4ZQ.png" /><figcaption>Figure 1: Illustration of the bias and variance definition.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SeIWWa6XrncNAYeXC1dF9A.png" /><figcaption>Figure 2: <a href="https://cs229.stanford.edu/main_notes.pdf">Bias-variance tradeoff</a></figcaption></figure><p>The figure 2 shows that with few parameters, the model tends to have high bias (<strong>underfitting</strong>) and low variance, while a complex model with many parameters tends to have low bias and high variance (<strong>overfitting</strong>). Therefore, the goal is to find a model with an optimal balance between bias and variance that can generalize well to new data.</p><p><strong>Underfitting</strong> occurs when the model is too simple to capture the complexity of the dataset, resulting in poor performance on both training and test set. To address this, one can increase the capacity of the model by selecting a more powerful class of function or look at some hyperparameters (number of iteration/epochs, learning rate, optimizer, etc).</p><p><strong>Overfitting</strong> on the other hand occurs when the model is too complex, resulting in poor performance on test data. To solve this we can:</p><ul><li>Decrease the capacity of the model by selecting a smaller class of functions or apply some regularization approaches (L1 and L2 regularization).</li><li>Increase the size of training data.</li><li>Apply optimization techniques like early stopping (it implies stopping the training at the point where the model starts to overfit).</li><li>Use the cross validation technique to provide a more accurate estimate of the generalization error.</li></ul><p>We saw that the test error first decreases (low complexity) and then increases as the model becomes more complex. However, in several works [1], it’s observed that the test error can have a second descent as described by the <strong>double descent phenomenon.</strong></p><h4><strong>Double descent phenomenon</strong></h4><p>There are two different manifestations of the double descent phenomenon in machine learning.</p><blockquote><strong>Model-wise double descent</strong></blockquote><p>Model-wise double descent is observed by varying the complexity of the model, such as the number of parameters or the norm of the model (the norm of a model here refers to a mathematical function that measures the size or the magnitude of the model’s parameters, e.g L1 norm and L2 norm). Conventionally, as the model complexity increases, the test error first decreases and then increases to a peak around when the model size is large enough to fit all the training data very well, and then decreases again in the so-called over-parameterized regime, where the number of parameters is larger than the number of data points [1].</p><blockquote><strong>Sample-wise double descent</strong></blockquote><p>Here the phenomenon is observed by varying the size of the training dataset. Some works [2] observed that the generalization error doesn’t strictly decrease as we increase the sample size. Instead, it decreases first, and increases to peak around when the number of samples is similar to the number of parameters, and then decreases again.</p><blockquote><strong>Experimental test</strong></blockquote><p>To illustrate this phenomenon, we generated two toy datasets for regression task.</p><p>The first dataset consists of 500 points and 1000 features. We built a simple linear regression model to evaluate its performance with respect to the complexity of the model. For this reason, we plot the test error against the number of parameters (number of features) of the model. Figure 3 shows that the test error increases and peaks when the number of sample (n) is approximately equal to the number of parameters (d), n≈d=500 and steadily decreases when number of parameters become much larger than number of samples, d&gt;&gt;n.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*z6CNWJCkqQNjRoQvda7seg.png" /><figcaption>Figure 3: Model-wise double descent phenomenon for linear regression model</figcaption></figure><p>On the other hand the second dataset consists of 1000 points and 500 features. As before, we built a simple linear regression model and plotted the test error against the number of samples to assess its performance regarding the sample size. Similarly, the graph displayed on figure 4 illustrates that the test error rises and reaches its maximum when the sample size (n) is roughly equal to the number of features (d); and finally drops where d&gt;&gt;n.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*I_gj6zooCszQEMJUOOrYPA.png" /><figcaption>Figure 4: Sample-wise double descent phenomenon for linear regression model.</figcaption></figure><h4>Conclusion</h4><p>This work gives a brief overview of the double descent phenomenon, a novel concept discovered in 2019 [3]. This phenomenon was observed through some algorithms such as linear regression and neural networks [4] and remains an active area of research in the field of Machine Learning/Deep Learning.</p><p>Hope this was helpful and enhanced your curiosity 😉.</p><h4><strong>References</strong></h4><p>[1]: <a href="https://cs229.stanford.edu/main_notes.pdf">https://cs229.stanford.edu/main_notes.pdf</a></p><p>[2]: <a href="https://arxiv.org/abs/1912.07242">More Data Can Hurt for Linear Regression: Sample-wise Double Descent</a></p><p>[3]: <a href="https://www.pnas.org/doi/full/10.1073/pnas.1903070116">https://www.pnas.org/doi/full/10.1073/pnas.1903070116</a></p><p>[4]: <a href="https://arxiv.org/pdf/2003.01897.pdf">Optimal Regularization Can Mitigate Double Descent</a></p><h4><a href="https://mlearning.substack.com/about"><strong>BECOME a WRITER at MLearning.ai</strong></a></h4><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3020172c99f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mask-RCNN for Object Detection with Detectron2]]></title>
            <link>https://medium.com/@pmegne/mask-rcnn-for-object-detection-with-dectectron2-46f62f6b7826?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/46f62f6b7826</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Sun, 22 May 2022 22:34:29 GMT</pubDate>
            <atom:updated>2022-06-06T21:03:44.193Z</atom:updated>
            <content:encoded><![CDATA[<p>In our previous article, we gave an intuition behind object detection. You can read it <a href="https://medium.com/@pmegne/introduction-to-object-detection-8be574b82d5c"><strong>here</strong></a> to refresh.</p><p>When considering the situation with more than one object in an image, current state-of-art models have proven to be very strong and powerful in identifying and locating multiple objects in an image. Some of them are R-CNN, Fast R-CNN, Faster R-CNN and Mask R-CNN. In this article, we will provide a simple understanding of <strong>Mask R-CNN</strong> an how it can be used to detect objects using the Detectron2 framework in PyTorch.</p><p>To understand Mask R-CNN, we will review the R-CNN variants first, starting from R-CNN architecture.</p><h4><strong>1. R-CNN</strong></h4><p>R-CNN stands for Region based Convolutional Neural Network and was developed by <em>Ross Girshick et al </em>[1]. It uses selective search[2] to propose a bunch of boxes (~2k) from an image; the boxes are called region proposals. R-CNN architecture consists of three main modules (Fig1). The first one generates region proposals, the second one extracts features with fixed length from each candidate region (region of interest/ ROI) using a large CNN. The third module is a classifier (linear SVM), to classify objects into the proposed class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*HfwVmMGhLE_QvsAviUGrow.png" /><figcaption>Figure1: <a href="https://arxiv.org/pdf/1311.2524.pdf">R-CNN architecture</a></figcaption></figure><p>However, during the feature extraction step, the CNN will pass independently on each ROI making the algorithm very slow.</p><h4><strong>2. Fast R-CNN</strong></h4><p>Fast R-CNN fixes the drawback of R-CNN by extracting the features at once, rather than running a CNN 2k times per image . In Fast R-CNN architecture, the CNN processes the whole input image and generate feature map. After this, a Region of Interest Pooling layer is applied to extract for each object proposal, a small fixed-length feature vector from the feature map. Finally, each feature vector is fed into a sequence of fully connected layers that will output the class labels and bounding boxes (Fig2). Recall that the proposal is done here using a heuristic selective search algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*mzmzi4Gxg2kipZWyx-m17A.png" /><figcaption>Figure2: <a href="https://arxiv.org/pdf/1504.08083.pdf">Fast R-CNN architecture</a></figcaption></figure><h4><strong>3. Faster R-CNN</strong></h4><p>This architecture is composed of two stages:</p><ul><li><strong>Region proposal network</strong> (RPN): after passing the input image in the CNN as before, the RPN is a deep network used to generate region proposals;</li><li><strong>Fast R-CNN detector</strong>: it will use the proposed region and output the bounding boxes and class labels.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*VGKCCfLS-JeUfb3O1V0P9g.png" /><figcaption>Figure3: <a href="https://arxiv.org/pdf/1506.01497.pdf">Faster R-CNN architecture</a></figcaption></figure><p>Compared to faster R-CNN architecture, the fast R-CNN ignores the time spent on region proposal, hence the term Faster R-CNN.</p><p>Note that, the bounding box doesn’t give information about pixels belonging to the foreground/ background of an object. In order to segment properly the object, a mask can be generated.</p><h4><strong>4. Mask R-CNN</strong></h4><p>Created by <em>Kaiming He et al </em>[3], Mask R-CNN is an extension of Faster R-CNN. It adopts the same two stages procedure , but during the second stage, in addition to outputting the class labels and bounding boxes, it also outputs an object mask. This additional mask output require extraction of much finer spatial layout of an object.</p><p>Moreover, the authors realized that although the quantization in the RoI pooling does not impact the label classification, it introduce a misalignment between the extracted features and the RoI. To avoid this, they include pixel-to-pixel alignment by using a <strong>RoI Alignment</strong> layer (Fig 4) to properly align the extracted features with the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*ZWxC5c52qdah8cWfXMa4Hw.png" /><figcaption>Figure4: <a href="https://arxiv.org/pdf/1703.06870.pdf">Mask R-CNN architecture</a></figcaption></figure><h4><strong>5. Implementation of Mask R-CNN using Detectron2</strong></h4><p>Detectron2 is a framework built by Facebook AI Research and implemented in Pytroch. It includes implementation for some object detection models namely Fast R-CNN, Faster R-CNN, Mask R-CNN, etc.</p><p>Here we use <a href="https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md"><strong>Mask R-CNN (R-101)</strong></a> with ResNet as the backbone architecture. This backbone architecture was pre-trained on <a href="https://arxiv.org/abs/1405.0312"><strong>COCO dataset</strong></a>.</p><h4><strong>6. Results</strong></h4><p>We hereby present some results and comments of our experimentation tested on a variety of images (persons, animals).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_hOSDvnKLTyT0JzyfpQ2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3Mlb9J9B5KSW8slKjC5ZQ.png" /></figure><p>Here are some observations obtained:</p><ul><li>In figure 5, the model is able to detect the people in the image as well as the objects around. In the same way, in figure 6, it detects the person and the dog that are in the image.</li><li>On the other hand in figure 7 there is a dog, a cat and two rabbits in the image, but the model detects all the objects as being either a cat or a dog. This can be due to the fact that rabbits have similar morphology like cats and dogs so it seems to be confused when predicting the rabbits.</li><li>In figure 8, there is a crowd of people and the model succeeds only in detecting some of them. The reason can be that, further individual in the crowd seems to be blurred therefore, the model doesn’t identify them clearly. On the other hand, Individual at the front-line presents well noticed morphological characteristics and therefore, are easily identified.</li></ul><p>Find <a href="https://github.com/MEGNEOrnela/My_medium_Posts/tree/main/Mask-RCNN%20for%20Object%20Detection%20with%20Dectectron2"><strong>here</strong></a><strong> </strong>the link associated to the code.</p><h3>Conclusion</h3><p>At the end of this post, we used an R-CNN mask to detect objects in an image with the Detectron2 framework. The model perform amazing predictions but seems to be confused in some cases.</p><p>Thank you for your time, hoping this post was helpful.</p><h4>References</h4><p>[1]: Girshick, R., Donahue, J., Darrell, T. and Malik, J., Rich feature hierarchies for accurate object detection and semantic segmentation. arXiv 2013. <em>arXiv preprint arXiv:1311.2524</em>.</p><p>[2]: Uijlings, J.R., Van De Sande, K.E., Gevers, T. and Smeulders, A.W., 2013. Selective search for object recognition. <em>International journal of computer vision</em>, <em>104</em>(2), pp.154–171.</p><p>[3]: He, K., Gkioxari, G., Dollár, P. and Girshick, R., 2017. Mask r-cnn. In <em>Proceedings of the IEEE international conference on computer vision</em> (pp. 2961–2969).</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46f62f6b7826" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to object detection]]></title>
            <link>https://medium.com/@pmegne/introduction-to-object-detection-8be574b82d5c?source=rss-608320f9ab7c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8be574b82d5c</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[mls]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ornela Megne]]></dc:creator>
            <pubDate>Mon, 09 May 2022 16:19:21 GMT</pubDate>
            <atom:updated>2022-05-23T06:18:38.874Z</atom:updated>
            <content:encoded><![CDATA[<p>In machine learning, computers are able to “identify” objects thanks to computer vision. But have you ever asked yourself how computers identify and locate objects in a given scene?. In this post we aim at answering this question. Here we go…!😎</p><h4>What is computer vision?</h4><p>Computer vision (CV) is a field of AI that tries to get computers to extract information from images and videos. In CV area, there are many different tasks such as: image classification, semantic segmentation, object detection, instance segmentation, etc. Here we will introduce the object detection task for a single object in a given image.</p><h4>Object Detection</h4><p>In general, object detection tries to answer the question :</p><p><strong>“What”</strong> objects are in an image and “<strong>where”</strong> are they?</p><ul><li>“What” means classification which is about labeling each object in a given image;</li><li>“Where” means bounding boxes, refers to localizing the position of the object in a given image.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iNtDA8-SIs0Gf29xeRNfFA.png" /><figcaption>Figure1: Made in canvas</figcaption></figure><p>So for a given input (images, videos), an object detector aims to predict both the bounding boxes and labels for each object.</p><p>For example consider that we want to detect a single object in a given image as shown in the figure x.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfcvPKICCEtcNYLgtMLt9Q.png" /><figcaption>Figure2 : <a href="https://project.inria.fr/bigvisdata/files/2020/01/cs231n_2019_lecture12_detection.pdf">Detecting a single object</a></figcaption></figure><ul><li>In order to extract features, the input image will pass through a convolutional neural network (CNN). This feature extractor can be a pertained model such as <a href="https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html">Alex Net</a>, <a href="https://arxiv.org/abs/1409.1556">VGG Net</a>, etc.</li><li>The output of the CNN process feeds into a fully connected layer to get our final feature vectors, from here the task is divided into two; The first part is a classifier (e.g: Linear layer with softmax, SVM, Random forest) to classify those vectors as one of the known classes, and the second one is a regressor, to predict the coordinates of the bounding box associated to the classified object. In this second case, the output of the model is the X, Y coordinates, the width ( W) and the height (H) of the located object in an image.</li><li>Now we can see that object detection requires a multi-task loss to realize both classification and localization.</li></ul><p><strong>a. Classification loss</strong></p><p>When dealing with classification task we use the Cross entropy loss as defined in the equation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8zZELZfpKHjR3dxqPcfOSg.png" /></figure><p>Generally it is also beneficial to look at the confusion matrix to understand the behavior of our model and which classes the model is better at predicting, this information can also be used to compute metrics namely, accuracy, precision, recall.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFCW1qr0PzFohIPD1i_zdg.png" /></figure><p><strong>b. Regression loss</strong></p><p>To measure how far the predicted bounding boxes are from their ground truth, we can use L1-norm or L2-norm (known as MSE loss) as our loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*8ieszC217o-MFfA6FHMxAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*_isgJOGrIRvW3tahZVOWzw.png" /></figure><p>Where y_hat is the predicted coordinate of the bounding boxes and y the ground truth coordinate.</p><p>Another widely used regression loss functions in many object detection models is the Intersection over Union (IoU) [1], also known as Jaccard index, IoU compare the similarity between two arbitrary boxes. By definition, it calculates the overlap of the two bounding boxes and divides it by their union.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WYgpXFKu-ZUY8DM68J_bAg.jpeg" /><figcaption>Figure4: <a href="https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/overview/evaluation">IoU</a></figcaption></figure><p>The final loss of the detector is the <strong>weighted sum</strong> of the classification loss and the regression loss. The idea behind performing the weighted sum is to control the magnitude of each loss (one loss could have a higher magnitude than the other) so the loss with higher magnitude will not dominate the overall loss.</p><p><strong>Images can have more than one object! In this case how do we identify and locate multiple objects in a single image?</strong></p><p>Several approaches have been used so far; some of which are <a href="https://ieeexplore.ieee.org/abstract/document/655647"><strong>sliding window</strong></a>, <a href="https://arxiv.org/abs/1311.2524"><strong>R-CNN</strong></a> (using selective search),<a href="https://arxiv.org/abs/1504.08083"> <strong>Fast R-CNN</strong></a>, <a href="https://arxiv.org/abs/1506.01497"><strong>Faster R-CNN</strong></a>.</p><h3>Conclusion</h3><p>Coming to the end of this post we presented a brief description of object detection model’s architecture and how it works for single objects in a given input images. In our next post we will outline briefly how Mask-RCNN works and showcase its application using Detectron2.</p><p>Thanks for reading, and hope it was insightful 😉.</p><p><strong>References:</strong></p><p>[1]:Yu, J., Jiang, Y., Wang, Z., Cao, Z. and Huang, T., 2016, October. Unitbox: An advanced object detection network. In <em>Proceedings of the 24th ACM international conference on Multimedia</em> (pp. 516–520).</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8be574b82d5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>