<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kriz Moses on Medium]]></title>
        <description><![CDATA[Stories by Kriz Moses on Medium]]></description>
        <link>https://medium.com/@kriz17?source=rss-bbb5020375a9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SFzBgE5bte7zUfTbueoGNQ.jpeg</url>
            <title>Stories by Kriz Moses on Medium</title>
            <link>https://medium.com/@kriz17?source=rss-bbb5020375a9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:18:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kriz17/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Medical Image Segmentation: 2018 Data Science Bowl]]></title>
            <link>https://pub.towardsai.net/medical-image-segmentation-2018-data-science-bowl-3093bef449a5?source=rss-bbb5020375a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3093bef449a5</guid>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[image-se]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Kriz Moses]]></dc:creator>
            <pubDate>Mon, 25 Oct 2021 15:00:22 GMT</pubDate>
            <atom:updated>2021-11-02T05:54:16.538Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/machine-learning/deep-learning">Deep Learning</a></h4><h4>A case study on Nucleus Segmentation across imaging experiments using Deep CNNs (UNet, UNet++, HRNet)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wkuZvId4aq-u39GO" /><figcaption>Photo by <a href="https://unsplash.com/@nci?utm_source=medium&amp;utm_medium=referral">National Cancer Institute</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Table of Contents</h3><ol><li><strong>Abstract</strong></li><li><strong>Introduction<br></strong>- Problem Statement and Dataset used<br>- ML Formulation of the Business Problem<br>- Real-World Constraints<br>- Metric used</li><li><strong>Literature Review<br></strong>- Challenges of medical image segmentation<br>- Sliding Window Method<br>- UNet</li><li><strong>Exploratory Data Analysis</strong></li><li><strong>Deep Learning Architectures Used<br></strong>- UNet<br>- UNet++<br>- HRNet</li><li><strong>Results and Discussion</strong></li><li><strong>Post-Training Quantization</strong></li><li><strong>Web Application</strong></li><li><strong>Summary and Future Work</strong></li><li><strong>Acknowledgments</strong></li><li><strong>References</strong></li></ol><h3>Abstract</h3><p>To find the cure for any disease, researchers analyze how the cells of a sample react to various treatments and understand the underlying biological processes. Identifying the cells’ nuclei is the starting point for most analyses since it helps identify each individual cell in a sample. Automating this process can save a lot of time, allowing for more efficient drug testing and unlocking the cures faster. In this case study, I propose the use of Deep CNNs for the automation of nucleus detection in images across varied conditions. I design a total of three networks derived from U-Net, U-Net++, and HRNet and compare their performances using Mean IoU as the metric. It’s seen that U-Net based model performs the best with a Mean IoU score of 0.861.</p><h3>Introduction</h3><p><a href="https://en.wikipedia.org/wiki/Image_segmentation">Image Segmentation</a> is segmenting particular regions of an image for better understanding and analysis. For example, in the field of the medical domain, in a brain scan, doctors might want particular regions to be highlighted (which can’t be observed by the human eye that easily) for better diagnosis. Another application would be in a self-driving car when you need to segment different kinds of objects in the image. Below is shown an example of segmentation of blood vessels in retinal images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*PwGKskl-0JThpGULZsGigA.png" /><figcaption>Retinal image and its segmentation mask (<a href="https://drive.grand-challenge.org/">https://drive.grand-challenge.org/</a>)</figcaption></figure><p>In general, there are two types of segmentation: Semantic and Instance. In Semantic Segmentation, different classes of objects are segmented separately (eg. separating people from background). Whereas, in Instance Segmentation, different instances of the same class are segmented (eg. separating different people from each other in the same picture)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsXei_-NBulKfQhfkxTdKw.jpeg" /><figcaption><a href="https://learnopencv.com/human-pose-estimation-using-keypoint-rcnn-in-pytorch/">https://learnopencv.com/human-pose-estimation-using-keypoint-rcnn-in-pytorch/</a></figcaption></figure><h4>Real-World Problem Statement and Dataset used</h4><p>In this case study, I strive to solve a semantic-based medical segmentation problem. The dataset used is from the Kaggle competition <a href="https://www.kaggle.com/c/data-science-bowl-2018/">2018 Data Science Bowl</a>. This has been used as a benchmark dataset for U-Net++ and DoubleUNet. The dataset consists of images of nuclei in different background conditions. The task is to segment the nuclei from the background. Below are shown a couple of examples from the dataset, of the image and its segmented nuclei mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*muKYQCsIrivclDXSQWgVGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTbecI2z9dJ7ZGClpZfq-Q.png" /><figcaption><a href="https://www.kaggle.com/c/data-science-bowl-2018/">https://www.kaggle.com/c/data-science-bowl-2018/</a></figcaption></figure><p><strong>Real-World Importance: </strong>The task that I will try to solve is to automate nucleus detection. This can help in speeding up research and getting the cure for almost every disease, from lung cancer and heart disease to rare disorders. This is because identifying nuclei is the starting point for most analyses because most of the human body’s 30 trillion cells contain a nucleus full of DNA, the genetic code that programs each cell. Identifying nuclei allows researchers to identify each individual cell in a sample, and by measuring how cells react to various treatments, the researcher can understand the underlying biological processes at work [2].</p><h4>ML Formulation of the Problem</h4><p>In general, image segmentation can be also posed as a multiclass classification problem in which every pixel in the image has to be assigned a class. Notice that here the input and the output are both images. Here, my input data point would be the image of the cells, and the output data point would be its segmented nuclei mask. The output image is labeled by an expert in the field. For our problem, the output image is a binary image with a background as black and nuclei as white.</p><p>So now given an input image (say of shape HxWx3), the task is to produce an output binary image (shape HxWx1) of the segmented nuclei i.e. to assign a binary number of 0 or 1 to every pixel of the output binary image.</p><h4>Real-World Constraints</h4><p>In the real-world, segmentation of biomedical images is considered to be really important for diagnosing purposes. And in a number of cases, a small error can lead to a false diagnosis which might put the health of a patient to huge risk. However, in our case, since the segmented nuclei would be used more for a general cure, hence small errors should not be as big an issue.</p><p>As far as latency is concerned, the segmented images of the nuclei do not need to be produced in milliseconds. A few seconds, even minutes should not hurt in most situations.</p><h4>Metric Used</h4><p>The metric used is Mean IoU i.e. the Mean Intersection Over Union.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*01cbzFH4IVWhmiPp3mXp2Q.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Jaccard_index">https://en.wikipedia.org/wiki/Jaccard_index</a></figcaption></figure><p>Its a very commonly used metric along with dice coefficient. The mean IOU for two objects can be defined as the area of intersection between them divided by the area of union. In image segmentation, the two objects are the true region and predicted regions of the part to be segmented.</p><p>To explain this better, say our object to be segmented is a football. Here the first object is the region of football in the real image. And the second object will be the region of the football in the predicted image.</p><p>So what does “Mean” signify in “Mean IoU”? “Mean” in “Mean IoU” stands for the average of all the classes to be segmented. So say in a task we need to segment all the cats from the background. So this task can be explained as a binary classification task, the two classes being a cat and the background. So mean IoU would be calculated by IoU(cat) + IoU(background) / 2</p><p>Similarly for multiclass classification, <br>Mean IoU = (IoU(class1) + IoU(class2) ….. + IoU(classN)) / N</p><p>Taking the mean of all classes helps to reduce the effect of data imbalance. A detailed explanation of Mean IOU can be found <a href="https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2">here</a>.</p><h3>Literature Review</h3><h4>Challenges of medical image segmentation</h4><p>The task of Image segmentation in itself was a challenge for deep learning methods since the output of the network has to be an image as well. Typical CNNs take in an image as input and output a number. Here, we had to output an image. On top of that, in medical image segmentation, the problem became even tougher since the number of training images was very less. Hence the task is pretty challenging given the scarcity of data.</p><p>The traditional methods for solving this task include thresholding and clustering-based approaches. Over the last few years, a lot of development has been done in the field of deep learning for medical image segmentation.</p><h4>Sliding Window Method</h4><p>In this method, each pixel in the output image is predicted individually. The input fed for that every pixel is the cropped image of the surrounding (say 64x64) grid around that pixel. Thus, for every image (HxWx3), we have a total of HxW separate outputs and a 64x64x3 patch of the input image as input for each output. This method was very trivial in its time but then had a lot of disadvantages. 1) This method was very time-consuming at runtime and there was a lot of redundancy due to overlapping patches. 2) When using larger patches to preserve the global structure, a lot of max-pooling was to be used which failed in preserving the local structure, and for smaller patches, although localization was better the global context was lost.</p><h4>U-Net</h4><p>One of the pioneer <a href="https://arxiv.org/pdf/1505.04597v1.pdf">papers</a> in this field introduced a very unique architecture for this very problem, called U-Net. The architecture of U-Net is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*OP0mvRcF4uM8bEoU" /><figcaption><a href="https://arxiv.org/pdf/1505.04597v1.pdf">U-Net: Convolutional Networks for Biomedical Image Segmentation</a></figcaption></figure><p>U-Net is an encoder-decoder type model, ensuring that both the input and the output are images. The encoder part of the architecture is normal convolutions + max pool and we eventually get to a very small size feature map. This part of the network tries to capture all the information about the image in a map of a small feature size. Now, this map is fed to the decoder part which tries to unravel the information and then get the desired output. Here, to get near the output image, the size of the feature map needs to increase while decreasing its depth. For this, we use up-conv (up sampling + conv). While using max pool local info was lost and up-conv does not help to retrieve that, hence to provide that information we use crop + copy from the previous map at the same horizontal level. The final output contains two maps. One for background and one for membrane. There is no padding used anywhere in the paper.</p><p>U-Net did really well with very few training images (30) from the <a href="http://brainiac2.mit.edu/isbi_challenge/">EM Segmentation challenge</a>. It improved the sliding window in two ways: skip connections and upsampling. Via upsampling, we no longer had to do pixel-wise segmentation and sacrifice the global context (as in the sliding window). And the skip connections help retain the local context lost while max pooling.</p><p>Since the introduction of U-Net, a number of successful architectures have been proposed which build on top of this as the backbone. In this case study, I have compared the performance of three models: U-Net, <a href="https://arxiv.org/pdf/1807.10165v1.pdf">U-Net++</a>, and <a href="https://arxiv.org/pdf/1908.07919v2.pdf">HRNet</a>. The architectures of the latter two are explained in the “Deep Learning Architectures used” section.</p><h3>Exploratory Data Analysis</h3><p>The dataset consists of images of cells in different background and their corresponding masks of segmented nuclei from the background. The competition was in two stages. For training, a total of 670 images and corresponding masks were provided. Stage 1 test set contained 65 images, and stage 2 test, 3019 images. The masks were provided for the stage 1 test images but not the stage 2 test. Hence in this case study, I use the 670 stage 1 train images as it is for training and the 65 stages 1 test images as the validation images. The masks for the stage 2 test are not provided, hence we can only use them for some manual analysis of how our model is working on unseen data.</p><p>So all in all, in this case, study, I use 670 images for training and 65 for validation. 3019 test images (without masks), will only be used for some manual analysis. A few examples are shown below consisting of the images and their masks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*d-G52xhKtqHhFCV2eZ1Kpg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*wrNfbfu_jNVvMsCGluyAwg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*sEOElKy3k6YQWn0BC0Ul9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*VRXUKO0mLMjl1TFwueaJTg.png" /></figure><p>Looking at only the first two images, you might think, that this can be easily solved through simple thresholding, but that is not the case since the images contain a lot of variety of backgrounds. This is evident looking at the next two. This was the challenging part of the competition. Also what made it more interesting was that the validation images had a lot more variation in images than the train images. So a model had to be designed which can generalize well and not overfit the training set.</p><h4>Dataset Details</h4><p>The following observations were found after some basic data analysis:</p><ul><li>The images are found in different shapes, the most common being 256x256</li><li>There are two types of images found in train and Val data: RGB and RGBA (4 channels). The last channel is called the alpha channel which decides the transparency (255 being opaque and 0 being transparent). Removing this channel will convert our image into an RGB image.</li><li>On close inspection of test data, I found that there is only one image rare grayscale which is a grayscale image. Rest all other images are either RGB or RGBA.</li><li>Images are found in different sizes.</li><li>All images are on a 0–255 scale only. No image is found to be on a 0–1 scale.</li></ul><p><strong>Train Images examples:</strong></p><ul><li>In train data, most images are black and white. Some are colored. The black and white images look easy to segment from the human eye. The colored images are generally purple in color. The train images don’t have a lot of variation in them.</li></ul><p>Below are shown some train images examples with the image on the left and its a mask on the right.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*GmA6VGt-1ZFqTctZmCBvyA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*6h_Y8Xf7NjMZKN_J57g4xQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*Qif59cqfOKxFk6-BPy_JzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*JyZrHSSN3FWqbLnqcGIzJw.png" /><figcaption>Train Images Examples</figcaption></figure><p><strong>Validation Images Examples:</strong></p><ul><li>The validation images seem a little harder to segment as compared to the training images. Some images are similar to the images from the training data and some are very different as seen below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*d0rVh4GKwPbKJ81tJBxNDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*DolkHopmMmkIv8nAhFa5Lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*sQOW4wQQGMEzYpJKeiGn8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*Qki2BXYCz6CmLukc06u6dA.png" /><figcaption>Validation Images Examples</figcaption></figure><p><strong>Validation Images Examples:</strong></p><ul><li>The test images on the other hand contain a lot more variations. Below some examples of test images (no mask) are shown.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZ58O18aMwKPn_G3QtPrrg.png" /><figcaption>Examples of test images</figcaption></figure><ul><li>Thus, the general observation is that the variation in images increases from train to validation to test. Hence, the chances of overfitting are higher, and thus the ability of the model to generalize is a very important part of the model building.</li></ul><h3>Deep Learning Architectures used</h3><p>I have compared the performance of three models: U-Net, U-Net++, and HRNet. I decided the input and output shape of the images to 256x256, since that was the most common shape found, and is a pretty standard input shape in state-of-the-art CNN networks.</p><h4>U-Net</h4><p>The architecture of the model is already explained in the “Literature Review” section. Unlike the standard U-Net with four encoder and decoder blocks each, I kept three of each. For up-sampling, I used the Conv2DTranspose layer. The whole code is written in Keras with Tensorflow backend. The code for U-Net, with the model image, is given below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fd7e0a56892b8c81d8526d932b69e93/href">https://medium.com/media/4fd7e0a56892b8c81d8526d932b69e93/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KTcOZRr0nfqzuOcjmGlIIg.png" /><figcaption>U-Net based architecture with shapes (Image by author)</figcaption></figure><h4><strong>U-Net++</strong></h4><p>In U-Net, the two main ideas were skipped connections and up-sampling. Via up-sampling, we no longer had to do pixel-wise segmentation and sacrifice the global context (as in sliding window). Whereas the skip connections help retain the local context lost while max pooling. In medical segmentation, it is important to have as few errors as possible and capture the most minute of details possible. It is observed that skip connections have been very helpful in recovering fine-grained details of target objects; generating segmentation masks with fine details even on complex backgrounds. This is because the fine-grained low-level information is present in the actual image which gets coarser as we move down and right the network due to max pooling. And skip connections try to provide this information back to the decoder part when it’s trying to reconstruct the image. Without the skip connections the decoder really has no direct idea about the image, but only the final encoded map. With skip connections, the decoder gets to know the actual image and the fine details.</p><p><a href="https://arxiv.org/pdf/1807.10165v1.pdf">U-Net ++</a> builds on this and tries to refine these skip connections. U-Net combines deep, semantic, coarse-grained feature maps from the decoder sub-network with shallow, low-level, fine-grained feature maps from the encoder sub-network. And it combines them directly (crop and copy). U-Net2+ strives to enrich the encoder maps before combining them with the decoder maps i.e. make them more semantically (logically) similar before combining.</p><p>The architecture is inspired by <a href="https://towardsdatascience.com/review-densenet-image-classification-b6631a8ef803">Dense-Net</a>. The advantages of dense-net are 1) Strong gradient flow 2) Fewer parameters and computational efficiency (The network is more complex and hence the network doesn’t need to be that deep) 3) Diversified features as each gets features from different layers.</p><p>The architecture of U-Net++ is given below. (The nodes capital X represent the conv operations and arrows represent the movement of feature maps. Small x i,j represent the outputs of the node X i,j)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*qPuxChYTiCqv-IFjiOOViQ.png" /><figcaption><a href="https://arxiv.org/pdf/1807.10165v1.pdf">UNet++: A Nested U-Net Architecture for Medical Image Segmentation</a></figcaption></figure><p>Two new concepts were introduced here: Dense Skip Pathways and Deep Supervision</p><p><strong>Dense Skip Connections</strong>: Let x i,j denote the output of node X i,j where i indexes the down-sampling layer along with the encoder and j indexes the convolution layer of the dense block along the skip pathway. The stack of feature maps represented by x i,j is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*RRfuaTid2jAlD4PlOLAcnA.png" /><figcaption><a href="https://arxiv.org/pdf/1807.10165v1.pdf">UNet++: A Nested U-Net Architecture for Medical Image Segmentation</a></figcaption></figure><p>where function H(·) is a convolution operation followed by an activation function, U(·) denotes an up-sampling layer, and [ ] denotes the concatenation layer.</p><p>Notice that this is not only different in the way that we are using dense connections between encoder and decoder layers, but also that we are using convolutions instead of simply copying skip connections. In the original U-Net, they just copied. Apart from this the paper also uses <strong>deep supervision</strong> in which the outputs from all the top semantic layers are averaged.</p><p>For its implementation, I reduced the number of convolutions in the encoder and decoder blocks by one layer. The code and the model image for the same are shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e32bf476f33fc26f0e7193ed5ac6b073/href">https://medium.com/media/e32bf476f33fc26f0e7193ed5ac6b073/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4nN0bgO9GFAAWDZMWaTzQ.png" /><figcaption>U-Net++ based architecture with shapes (Image by author)</figcaption></figure><p><strong>HRNet<br></strong>The network was inspired by <a href="https://arxiv.org/pdf/1908.07919v2.pdf">this</a> paper introducing a more densely connected architecture for segmentation, with the idea of extracting a deep representation of the different resolution features and then combining them together. Since the code and the image were pretty large in size, I did not include them here. You can find the code on my GitHub page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQG3y7OlGr49Hl3njP_8Vw.png" /><figcaption><a href="https://arxiv.org/pdf/1908.07919v2.pdf">Deep High-Resolution Representation Learning for Visual Recognition</a></figcaption></figure><h3>Results and Discussion</h3><p>All the models were trained with binary cross-entropy as the loss function. Adam optimizer was used, with a learning rate of 1e-3. I used early stopping with the patience of 20 epochs monitoring validation loss. For metric, I used a custom implementation of Mean IoU.</p><p>The graph for validation loss and Mean IoU is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJWKpAqPUyK3xxr-OVg0tg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NBC-bQ0VrasaEYsJTw2CMA.jpeg" /></figure><p>After training, for each model, the instance of the model at that epoch was selected which had the best Mean IoU. The comparison of all the models is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*x2flUTBizR4hDEWq01WGFg.png" /></figure><p>The performance of all three models is comparable. The training for every model stopped around the 50th epoch mark, meaning all had reached convergence around the 30–35 epoch mark. The best performing one was U-Net with a validation loss of 0.1098 and mean IoU of 0.861. U-Net++ was not far off with a score of 0.8602. HRNet had the lowest performance with a mean IoU of 0.85.</p><p><em>On top of this, I also experimented with increasing the number of encoder-decoder blocks for U-Net and U-Net++ but the models started overfitting and did not give any better results, hence I did not include them here.</em></p><h4>Predicted Masks</h4><p>Below are shown the predicted masks for all the models with the Mean IoU score. Each row contains an image and a true mask followed by a predicted mask by UNet, UNet++, and HRNet in that order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vr9zYmIMiiOk7Ij_tBkxGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FOho9su5GjN-FwJnLYjP3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6T7LsgIu8PsWLiFtNMtRsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QVzAq2fUCE7kd-Ige_2WJQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*17qdMcUT80EJaDROi9HDAA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tk6HSI6Sp1BALhjK-2CAkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beRFyrVYgkcKcTsX3W-TjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WA0xZQzmcRwLAaDVCNaNDQ.png" /></figure><p>Below are some predictions on test images (without true masks and hence without scores)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dy4guuuaG34l_jOYAFtaTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RuGh8ThKx4eSvcyPdg3T8A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*meYuWXAaPFHUDof5ApybLQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VXgA52HBpEtKu-YUOcRGvA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqfcSxckJ7bREbTXSLan3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhErtnDd_MKt-rmsmLCSzQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Kvmx-U-S_gpOWjtWRTDJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UDLUp4GfOcRLn0amuUx3Q.png" /></figure><p>The performance of the models was quite decent for the validation images but the test predictions were not that satisfying. There is still the issue of generalization with such a few training images.</p><p>Finally, I selected the best model as UNet. The model had an <strong>average inference time</strong> of <strong>0.964 seconds</strong>, with the <strong>model size </strong>being <strong>28 MB</strong>.</p><h3>Post-training Quantization</h3><blockquote>“Post-training quantization is a conversion technique that can reduce model size while also improving CPU and hardware accelerator latency, with little degradation in model accuracy.”<br>- TensorFlow Official Documentation</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77528d79f9a03558f5e256749903665f/href">https://medium.com/media/77528d79f9a03558f5e256749903665f/href</a></iframe><p>There are different methods for quantization. For my model, I used Float16 quantization in which all the weights are converted to 16-bit floating-point numbers. This reduces the model size by half: from 28 MB to 14 MB.</p><p>One important check-in quantization is to see if it does not reduce the performance of the model. This is the very reason I could not use other methods like dynamic range quantization, as these methods are not supported in the CPU in my box, and hence getting the outputs was taking a lot of time. Thus, it was not possible to examine if there is any degradation in performance. I could check for Float16 quantization, and there was <strong>no degradation in predictive performance </strong>(Mean IoU score was 0.861 for the quantized model as well) so I went with that.</p><p>So, finally, the <strong>model size</strong> was <strong>14 MB</strong>. There was an <strong>increase in inference time </strong>from <strong>0.964 </strong>seconds to <strong>1.261 </strong>seconds, which is fine, given the decrease in model size.</p><h3>Web App</h3><p>I built a web app for this project and deployed it via streamlit:<br><a href="https://share.streamlit.io/kriz17/mdt_app">https://share.streamlit.io/kriz17/mdt_app</a></p><h3>Summary and Future Work</h3><p>In this case study, I designed Deep CNN-based models for automation of nucleus detection in images under varied conditions. The performance of the total of three models was compared, which were based on U-Net, U-Net++, and HRNet. The best performing model was the U-Net-based one with a Mean IoU score of 0.861. Further, I used Float16 quantization to reduce the model size from 28 MB to 14 MB. The average inference time of the model was 0.126 seconds per sample which is decent for the task at hand. Finally, I deployed the model using streamlit.</p><p>All three models did a pretty decent job with each securing a Mean IoU score of greater than 0.85. They can be used for images with standard backgrounds as in the training data, but not with high variations. The main challenges were the small training dataset and the high variation in the backgrounds of the validation images. The conventional architectures of the models lead to overfitting, and hence I had to decrease the size of the models. In future work, I would like to try data augmentation using elastic deformation as implemented in the original U-Net paper. I would also like to explore Transformer Models based on attention mechanisms since they have proven to be really effective for small datasets [3].</p><h3>Acknowledgments</h3><p>I would like to thank the whole Applied AI Course team and especially Ramana Sir for guiding me throughout this case study.</p><h3>References</h3><p>[1] <a href="https://www.appliedaicourse.com/">https://www.appliedaicourse.com/</a></p><p>[2] <a href="https://www.kaggle.com/c/data-science-bowl-2018/">https://www.kaggle.com/c/data-science-bowl-2018/</a></p><p>[3] <a href="https://arxiv.org/pdf/2102.10662v2.pdf">https://arxiv.org/pdf/2102.10662v2.pdf</a></p><p>[4] <a href="https://arxiv.org/pdf/2009.13120.pdf">https://arxiv.org/pdf/2009.13120.pdf</a></p><p>[5] <a href="https://medium.com/swlh/image-segmentation-using-deep-learning-a-survey-e37e0f0a1489">https://medium.com/swlh/image-segmentation-using-deep-learning-a-survey-e37e0f0a1489</a></p><p>[6] <a href="https://paperswithcode.com/task/medical-image-segmentation">https://paperswithcode.com/task/medical-image-segmentation</a></p><p>[7] <a href="https://en.wikipedia.org/wiki/Image_segmentation">https://en.wikipedia.org/wiki/Image_segmentatio</a>n</p><p>[8]<a href="https://arxiv.org/pdf/1505.04597v1.pdf"> https://arxiv.org/pdf/1505.04597v1.pdf</a></p><p>[9]<a href="https://towardsdatascience.com/review-u-net-biomedical-image-segmentation-d02bf06ca760"> https://towardsdatascience.com/review-u-net-biomedical-image-segmentation-d02bf06ca760</a></p><p>[10] <a href="https://drive.google.com/file/d/1U-SEtD5NExw86Bau7mOW9Lz9F1n7LpF3/view">https://drive.google.com/file/d/1U-SEtD5NExw86Bau7mOW9Lz9F1n7LpF3/view</a></p><p>[11]<a href="https://arxiv.org/ftp/arxiv/papers/1802/1802.06955.pdf"> https://arxiv.org/ftp/arxiv/papers/1802/1802.06955.pdf</a></p><p>[12] <a href="https://arxiv.org/pdf/1802.02427.pdf">https://arxiv.org/pdf/1802.02427.pdf</a></p><p>[13] <a href="https://arxiv.org/pdf/1807.10165v1.pdf">https://arxiv.org/pdf/1807.10165v1.pdf</a></p><p>[14] <a href="https://arxiv.org/ftp/arxiv/papers/2004/2004.08790.pdf">https://arxiv.org/ftp/arxiv/papers/2004/2004.08790.pdf</a></p><p>[15]<a href="https://sh-tsang.medium.com/reading-unet-3-a-full-scale-connected-unet-medical-image-segmentation-ebb5e7f53caa"> https://sh-tsang.medium.com/reading-unet-3-a-full-scale-connected-unet-medical-image-segmentation-ebb5e7f53caa</a></p><p>[16] <a href="https://arxiv.org/pdf/1908.07919v2.pdf">https://arxiv.org/pdf/1908.07919v2.pdf</a></p><p>Full work with code can be found here on my GitHub profile:</p><p><a href="https://github.com/kriz17/Medical-Image-Segmentation">GitHub - kriz17/Medical-Image-Segmentation</a></p><p>Connect with me on LinkedIn:</p><p><a href="https://www.linkedin.com/in/kriz-moses/">Kriz Moses - IIT Indore - Indore, Madhya Pradesh, India | LinkedIn</a></p><p><strong>PS</strong>: Feel free to provide comments/criticisms if you think they can improve the project/blog. I will definitely try to make the required changes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3093bef449a5" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/medical-image-segmentation-2018-data-science-bowl-3093bef449a5">Medical Image Segmentation: 2018 Data Science Bowl</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling Product Search Relevance in e-Commerce: Home Depot Case Study]]></title>
            <link>https://medium.com/data-science/modeling-product-search-relevance-in-e-commerce-home-depot-case-study-8ccb56fbc5ab?source=rss-bbb5020375a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ccb56fbc5ab</guid>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-retrieval]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kriz Moses]]></dc:creator>
            <pubDate>Tue, 29 Jun 2021 14:00:18 GMT</pubDate>
            <atom:updated>2021-11-07T13:07:16.042Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Predict the relevance of search results on homedepot.com using Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n8uaFZduAPr3Yq3whEmKQw.jpeg" /><figcaption>Photo by <a href="https://www.pexels.com/@anete-lusina?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Anete Lusina</a> from <a href="https://www.pexels.com/photo/crop-buyer-with-smartphone-and-coffee-to-go-6331238/?utm_content=attributionCopyText&amp;utm_medium=referral&amp;utm_source=pexels">Pexels</a></figcaption></figure><h3>Table of Contents</h3><ol><li><strong>Introduction<br></strong>• Abstract<br>• Problem Statement<br>• ML Formulation of the Business Problem<br>• Data Overview<br>• Objective and Metric for the ML problem<br>• Real-World Constraints</li><li><strong>Literature Review</strong></li><li><strong>Exploratory Data Analysis: Part 1<br></strong>• Relevance Scores<br>• Product_uid<br>• Attributes<br>• Description<br>• Merging product text data<br>• Filling Null Values</li><li><strong>Data Cleaning<br></strong>• Basic Preprocessing<br>• Standardizing Units<br>• Correcting typos in Search<br>• Further Preprocessing</li><li><strong>Exploratory Data Analysis: Part 2<br></strong>• Basic Text Statistics<br>• Distribution of Unigrams and Bigrams</li><li><strong>Feature Engineering<br></strong>• Set-Theoretic Features<br>• VSM based Features<br>• Probabilistic Features<br>• Query Expansion using Word2Vec</li><li><strong>Performance Chart of the Feature Sets</strong></li><li><strong>Final Modeling and Evaluation<br></strong>• Stacking Regressor<br>• Base Models <br>• Meta Model<br>• Performance on Kaggle (Top 10%)</li><li><strong>Extending into a Real-World Search Engine</strong></li><li><strong>Full Pipeline</strong></li><li><strong>Final Testing and Results<br></strong>• Testing<br>• Run Time<br>• Web Application Demo</li><li><strong>Conclusion and Future Work</strong></li><li><strong>Acknowledgments</strong></li><li><strong>References</strong></li></ol><h3>Introduction</h3><h4>Abstract</h4><p>Everything is moving to digitalization and so is the shopping experience. E-commerce has been on rapid growth over the past few years and hence, online product search has become one of the most important factors in providing the customer with a satisfying shopping experience. In this blog, I propose a robust way of predicting the desired products for a given search query, using techniques involving machine learning, natural language processing, and information retrieval.</p><h4>Problem Statement</h4><p>The task is simple to understand. For any search query that the customer enters, I need to find the most relevant products and show them to the user in order of their relevance. From a business point of view, there are a few points which need to be considered. First, the products need to be ranked, thus, even among the most relevant products, we need to be able to tell which one is more relevant. Second, there is a time constraint involved i.e. the results need to be shown within seconds.</p><h4>Machine Learning Formulation of the Business Problem</h4><p>The task can be formulated as follows: Given a search and a product, find the relevance score between them i.e. how relevant that product is to the search query at hand. So let’s say my machine has learned how to predict the relevance score for a (search-query, product) pair. Now, for any search that the user enters, I can calculate the relevance score for that very search paired with all the products in my database and show the (say) top 10 results to the customer.</p><p>Thus, if I have a lot of labeled data i.e. a lot of (search-query, product) pairs with their relevance scores then I can pose this as a supervised ML problem. And that is exactly what I have done in this case study. The data I have used is provided by Home Depot for the Kaggle competition <a href="https://www.kaggle.com/c/home-depot-product-search-relevance">home-depot-product-search-relevance</a></p><p>Now in a real-world e-commerce search engine, calculating the relevance score for every product for a given search is not possible, because, in any typical e-commerce website, the number of products is very large, and hence it’s computationally expensive and very time-consuming.</p><p>✦ Thus, first, we retrieve a few candidate products using a simpler retrieval model which permits fast query evaluation. And in the second phase, a more accurate but computationally expensive machine-learned model is used to re-rank these products.</p><p>To explain this, consider we have a set of 1,00,000 products. And the search query is “solar lamp”. The first simpler retrieval model will retrieve a few candidate products. This model can be as simple as an AND operator between the words of search and product-text. So here a model governed by AND operator will get all the products with the words “solar” and “lamp”. Say it retrieves some 500 products. Now on top of these 500 products, we can run our complex machine learning algorithm and calculate the relevance score for each (search, product) pair where our search term “solar lamp” remains constant and products vary. And now we re-rank them based on their relevance score and show the top products to the user. This is called <a href="https://en.wikipedia.org/wiki/Learning_to_rank">Learning To Rank</a> (LTOR). In this blog, the focus will be on the learning-to-rank part of the system. In the end, I have also explained how I extended it to make a full-fledged search engine.</p><h4>Data Overview</h4><p>The data is in the form of four CSV files; train.csv, test.csv, attributes.csv, and product_description.csv. You can download it from <a href="https://www.kaggle.com/c/home-depot-product-search-relevance/data">here</a>.</p><p><strong>Training Data: </strong>We have a total of 74067 rows for training in the train.csv file. A part of the data looks as shown below, each row corresponding to a pair of</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*kZYYt-v5i02iw1qHCfRbwg.png" /><figcaption>train.csv file</figcaption></figure><p>(search_term, product_title) and the relevance score telling how much that product is relevant to the search. A relevance score is a real number from 1 to 3 (1 for irrelevant and 3 for perfect match). For each product, we generally have multiple rows i.e. multiple search queries with their relevance score. The product_uid is a unique identifier for each product.</p><p>The product_description.csv contains a text description of each product. And the attributes.csv file contains some additional information about a subset of products.</p><p>The search query is in the form of a string and the product is represented by text data available to us in the form of its title, description, and some attributes. In the real world, we might also have other features for the product like its image, ratings, etc. but here we only have text to work with.</p><p>The relevance scores for the data were manually marked. Each pair (search, product) was evaluated by at least three human raters and then the final relevance score was taken as the average.</p><p><strong>Test Data: </strong>As this is a part of a Kaggle competition, we are also given a test.csv file containing around166,000 rows for which we need to predict the relevance scores. About half of the products in the train set are found in the test set. The Venn diagram for the two sets is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*Z6YThksszjA_iPw5XNmW3Q.png" /></figure><h4>Objective and Metric for the ML problem</h4><p><strong>Objective</strong>: For every given (search-query, product), predict the relevance score. This will be posed as a regression problem as it will be easier for us to rank the products.</p><p><strong>Metric Used: <br></strong>This metric given in the competition is <strong>Root Mean Square Error (RMSE)</strong>. It is the most widely used metric for regression problems and works well in most cases. It is different from MAE (mean absolute error) in the sense that it penalizes large errors more. One issue with RMSE is that its value is difficult to interpret; It can range from zero to infinity. Hence, generally, a baseline score is established using a naive regression model which we can then use as a reference for other models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*SOhwB6kDlgIng4FkoYDKmA.png" /><figcaption>Source: <a href="https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/05/rmse.png">https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/05/rmse.png</a></figcaption></figure><h4>Real-World Constraints</h4><ul><li>Latency constraint: The products need to be shown within a few seconds.</li><li>Ranking of products is necessary.</li><li>Interpretability is partially important: We would want to know why our model has suggested a particular product to the customer.</li></ul><p><strong><em>Note: </em></strong><em>In the blog, I have used the terms ‘search query’, ‘query’, ‘search term’ and ‘search’ interchangeably.</em></p><h3>Literature Review</h3><p>The problem at hand is of search relevance which broadly comes under the field of <a href="https://en.wikipedia.org/wiki/Information_retrieval">Information Retrieval</a>.<em> </em>A ton of research has been published about techniques for free-text querying.</p><p>The traditional non-machine-learned models centered around exploring various ways of modeling language. Set-theoretic models like the <a href="https://www.researchgate.net/profile/Arash-Habibi-Lashkari/publication/232614940_A_Boolean_Model_in_Information_Retrieval_for_Search_Engines_PDF/links/5756c52808aef6cbe35f0f5a/A-Boolean-Model-in-Information-Retrieval-for-Search-Engines-PDF.pdf">Boolean model</a> and <a href="https://www.sciencedirect.com/science/article/abs/pii/S0165011409001080?via%3Dihub">Fuzzy Retrieval</a> are based on representing queries and documented as sets of words and then applying some basic set operations to get the relevance. Algebraic models like the <a href="https://en.wikipedia.org/wiki/Vector_space_model">VSM</a> represent documents and queries as vectors (eg. <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">bag-of-words</a>) and then the similarity between them is calculated as a scalar quantity. More popular models like the <a href="https://www.researchgate.net/profile/Hugo-Zaragoza/publication/220613776_The_Probabilistic_Relevance_Framework_BM25_and_Beyond/links/53d9ce800cf2a19eee8807e8/The-Probabilistic-Relevance-Framework-BM25-and-Beyond.pdf">Okapi BM25</a>, <a href="http://ciir.cs.umass.edu/pubfiles/ir-407.pdf">Indri</a>, and <a href="https://en.wikipedia.org/wiki/Language_model">Language Models</a> are based on <a href="https://en.wikipedia.org/wiki/Probabilistic_relevance_model">probability theory</a>. Although all these models come from different underlying theories, most of them use very similar concepts from the field of <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a> like <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">bag-of-words</a>, <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">tf-idf</a>, cosine similarity, etc.</p><p>The more recent approaches include <a href="https://en.wikipedia.org/wiki/Learning_to_rank">Learning To Rank</a> where typically the query-document pairs are represented by their feature vectors and then a supervised or a semi-supervised machine learning model is trained on them to get the rankings. Here, it&#39;s common to use the traditional retrieval models directly as features. <a href="https://opensourceconnections.com/blog/2016/03/29/semantic-search-with-latent-semantic-analysis/">LSA</a> (or LSI) maps the term-document matrix to a ‘concept’ space and hence getting a dense and better representation of the text. The <a href="https://en.wikipedia.org/wiki/Word2vec">Word2Vec</a> algorithm uses a neural network to learn word associations from a large corpus of text and map the words to high dimensional vectors, which can be used as features. <a href="https://www.researchgate.net/profile/Saar-Kuzi/publication/310823543_Query_Expansion_Using_Word_Embeddings/links/5a94aacf0f7e9ba42970d144/Query-Expansion-Using-Word-Embeddings.pdf">Query Expansion</a> is another effective idea to expand queries with similar words present in the corpus. For measuring similarity, word vectors from the Word2Vec model are commonly used.</p><h3>Exploratory Data Analysis: Part 1</h3><p>In this section, I present the analysis of the data at hand. I have saved the train.csv file as a dataframe in the train_df variable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c6a8475dc9ab1dd6cd56b5adec214dd/href">https://medium.com/media/9c6a8475dc9ab1dd6cd56b5adec214dd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*t06WUkj_gdz36YIyZcreBw.png" /><figcaption>train_df</figcaption></figure><p>The id column is of no use. The product_uid is a unique id of each product and is important while merging the data from other text files. There are no null values in the train data.</p><h4>Relevance Scores</h4><p>The relevance scores were found to be categorical in nature. This was because most of the scores are the average score of 3 raters and each rater could only give an integral score of either 1 or 2 or 3.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/896e40651498a2d0a2d2590257f6266e/href">https://medium.com/media/896e40651498a2d0a2d2590257f6266e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*2I2FFS4cY7obaXWc8thnKw.png" /></figure><p>The majority of the products had a high relevance score i.e. most of the products were relevant to the query. Because of the categorical nature of the relevance scores, a classification model can also be built on this if it helps the final rankings. The scores {1.25, 1.5, 1.75, 2.25, 2.5, 2.75} occurred in very low frequencies.</p><h4>Product_uid</h4><p>A lot of products occur multiple times in the train data i.e. we have multiple search terms and corresponding relevance for the same product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l-EXNARi-DjqU1UJOZWUfQ.png" /></figure><ul><li>More than 10,000 products have multiple occurrences</li><li>The total number of unique products out of 74067 is 54667</li><li>Max times a product has occurred is 21</li></ul><h4>Attributes</h4><p>For the products, we have text information in the form of product title, product description, and attributes. The product title is in the train.csv file. The description and attributes are extracted from the product_description.csv and attributes.csv file respectively.</p><p>An instance of the attributes.csv file is shown below. The field “name” represents the name of the attribute and “value”, the attribute text.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f63c7d6fdd00559ef08630d3f31062b3/href">https://medium.com/media/f63c7d6fdd00559ef08630d3f31062b3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*b51BpY1R400UeSmLd5Mq2w.png" /><figcaption>attributes.csv</figcaption></figure><p>Each product in the attributes.csv file has multiple attributes. The minimum number of attributes for a product is 5 and the maximum is 88.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MTwaidBw39Y2OjjCeFQhrQ.png" /><figcaption>number of attributes vs product-index</figcaption></figure><p>But not all the products in the train.csv file are found in the attributes.csv file. Total 16263 out of a total of 54667 products do not have attributes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*nDAvx7VGqdBKwY8s7vTGNw.png" /><figcaption>Sets of products in the train.csv and attributes.csv files</figcaption></figure><p>Then I explored the most frequently occurring attributes and Brand was the most common one found in 86,250 instances in the attributes.csv file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06a0ec8f75a6d5cdd716d0d9f72cc888/href">https://medium.com/media/06a0ec8f75a6d5cdd716d0d9f72cc888/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*js33cQvQKBnNtZIkov8MyQ.png" /><figcaption>Most frequent attributes</figcaption></figure><p>Not only this but it is also observed that mostly, the product_title contained the brand information. Thus, I added brand as a separate feature in my train_df.</p><h4>Description</h4><p>The product_description.csv has the description of the products. Unlike attributes, we do have descriptions for all the products in the train.csv file.</p><h4>Merging the Attributes, Description, and the Brand feature</h4><p>Finally, I merged all the attributes, the description, and the brand with the train_df. <em>All the attributes for a product were first concatenated and then stored as the ‘combined_attr’ feature</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/329e8324d4e7900ea54539a781345fe8/href">https://medium.com/media/329e8324d4e7900ea54539a781345fe8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SeOJRTLOTXXKAJDO8yEMqA.png" /><figcaption>train_df after merging</figcaption></figure><p><em>Note: From here on, the text fields representing the product, i.e. the product_title, brand, combined_attr, and product_description have been referred to collectively as the ‘product-text’ or ‘document-fields’ or ‘documents’.</em></p><h4>Filling Null Values</h4><p>Before moving onto the cleaning of the text, I found that our train_df has a lot of null values. These occurred when I merged the attributes, as a lot of products in the train data do not have attributes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b11ecc95e842e08419a192bf82510e57/href">https://medium.com/media/b11ecc95e842e08419a192bf82510e57/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*nix1U00-7YwPI78KpCZipg.png" /></figure><p>We can see that the null values occur only in the brand and the combined_attr feature. There are a considerable amount of null values and hence filling them properly was an important step in the cleaning.</p><p><strong>Brand: </strong>It was observed that the brand of a product was generally present in the first few words of the title. For eg., the brand for the product with the product title “Simpson Strong-Tie 12-Gauge Angle” is “Simpson Strong-Tie”. And for “Delta Vero 1-Handle Shower Only Faucet” is “Delta”. Also, 99% of the brands are of length four words or less. So mostly, the brand should be present in the first 4 words of the title.</p><p>So, I first stored all the unique brands in a list “unique_brands”. Now, for any row with a null brand, I checked if the first four words of the title of that product are present in the unique_brands list. If not, then I checked the first three words and so on till the first word. This method worked pretty well and I was able to fill 15003 null brands. For the remaining 2625 values, I filled them with the first word of the title.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d11d9b767f0c64e0f45ea1f0c46e0c8/href">https://medium.com/media/8d11d9b767f0c64e0f45ea1f0c46e0c8/href</a></iframe><p><strong>Description: </strong>Filling the null values for description was not that hard as it was observed that the description text is very similar to the combined attributes text. Hence, all the null values for the combined_attr field were filled with the corresponding product_description values.</p><h3>Data Cleaning</h3><p>This was one of the most important parts of this whole case study. A large amount of time went into observing the text and making the necessary preprocessing steps to make it suitable for modeling. Many techniques from natural language processing were used.</p><p><em>A very simple motivation as to why text preprocessing is so important in this project is because we are in a way trying to find the similarity between the search-query and the product-text. Consider the search as “5gal bucket” and the product-title as “Leaklite 5-Gal Black Bucket”. If we were using a measure like number-of-common-words, then the similarity, in this case, would be zero as there are no words in common. Now say with a bit of preprocessing, our search and title become “5 gal bucket” and “leaklite 5 gal black bucket” respectively. Now our similarity score comes out as 3 because now we have 3 words in common. Thus, little preprocessing can make a huge difference.</em></p><h4>Basic Preprocessing</h4><p>The basic preprocessing steps involved lowercasing, removing special characters, removing stop-words, and stemming.</p><h4>Standardizing Units</h4><ol><li>Many of the product-titles contain the dimensions of the product. A very common way of representation is “8 ft. x 5 ft. x 8 ft.”. The same is observed in product-description.</li><li>In product_title, units are mostly represented as {in. lb. sq. lb. oz. gal. mph}, but in product_description and search_term, units are represented in multiple ways. For eg. inch as {in. in inches} and gallons as {gal. gals gallon}. And hence, these units need to be standardized.</li><li>In search terms, there are some common errors when specifying measurements. Some examples are ‘4shelves’, ‘9x12’, ‘g135’, ‘5gal’, ‘1/2 in by 12’. Hence, it was needed to separate numbers from words.</li></ol><p><strong>Preprocessing Steps:</strong></p><ol><li>Separate numbers from words: ‘9x12’ → ‘9 x 12’, ‘5gal’ → ‘5 gal’ …</li><li>Bring all units of measurements to a consistent representation. For eg. {gal. gals gallon} all are converted to “gallon”. Similarly {ft fts feets foot foots} are converted to “feet”.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57cda75d81eb0e577fc0b1bdea3c8f88/href">https://medium.com/media/57cda75d81eb0e577fc0b1bdea3c8f88/href</a></iframe><h4>Correcting Typos in Search Terms</h4><p>While going through the search terms, it was found that it contains a lot of spelling errors like ‘mower’ was spelled as ‘mowe’, ‘toilet’ as ‘toiled’, etc. This is understandable, as these searches are entered by users and in online search, such spelling errors are common. But this was a big problem, as misspelled words could make features like common-word-count very ineffective. Hence, to correct this, I used a custom spelling corrector. And the results were pretty positive.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9bf3c52ab8819c711a12d9f87085871/href">https://medium.com/media/b9bf3c52ab8819c711a12d9f87085871/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*on-Z2OFzxGlNO_pLuai7wg.png" /><figcaption>Incorrect search→ Corrected search</figcaption></figure><p>The spelling corrector I used was built by <a href="https://norvig.com/spell-correct.html">Peter Norvig</a>. It is based on probability theory but is very simple to understand and very effective. The best thing about it is that you can fit it to your own data. This was very important, as a general-purpose spelling corrector might not be able to recognize brand names that are limited to our data. I encourage you to refer to the link and read more about it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa1a1cf1f044fee9070469043051bf70/href">https://medium.com/media/fa1a1cf1f044fee9070469043051bf70/href</a></iframe><p>I corrected all the search terms and for experimental purposes, I also kept the raw/uncorrected search.</p><h4>Further Preprocessing</h4><p>In the basic preprocessing, I left the stop-word removal and the stemming part as I wanted to fix the incorrect search terms first. Here both these steps are performed and the cleaned text is stored in cleaned_df.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7def6c999dc3818fc4c288cbac6e4dd4/href">https://medium.com/media/7def6c999dc3818fc4c288cbac6e4dd4/href</a></iframe><p>And our final cleaned dataframe looks as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cXpNHI0ywS7o0qgkIB8kwA.png" /><figcaption>cleaned_df</figcaption></figure><p><em>I also created a dataframe cleaned_df2 where I store the text data without stemming. This was done to be able to use pre-trained encodings like Word2Vec later.</em></p><h3>Exploratory Data Analysis: Part 2</h3><p>After cleaning, I was able to make more accurate findings from the text data analysis. There was a lot of analysis done, and I’ll go over the most important findings/conclusions in this section.</p><p>I divided the relevance scores into two classes for easier analysis: “High” with scores &gt; 2 and “Low” with scores ≤ 2. They are also referred to as the positive and the negative class respectively. Although, this is a regression problem, dividing the scores into two classes helped a lot in the analysis; A feature that helps separate the High from the Low class would mostly be helpful in the final regression problem as well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/260619b0d435f2b0b834ca89f549b1e8/href">https://medium.com/media/260619b0d435f2b0b834ca89f549b1e8/href</a></iframe><p><em>We have a total of 6 text fields and the analysis was done separately on each.</em></p><p><strong><em>NOTE: <br></em></strong><em>After close inspection, I observed that the description and attributes text are very similar and thus, I did not consider the attributes text in any further analysis.</em></p><h4>Basic Text Statistics</h4><p>Analysis of a number of basic stats on the text was done. Namely the length of the text string, word count, and average word length. For corrected_search, the distribution of these stats is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b792ebe09bb9c6b6847931e242189337/href">https://medium.com/media/b792ebe09bb9c6b6847931e242189337/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dsOD-blkH8kf_KQdBD0CWA.png" /><figcaption>Distributions of basic stats</figcaption></figure><p><strong>Observations :</strong></p><ol><li>Most search queries have a word count of between 2 to 5 words.</li><li>The distributions of these stats are very similar for both positive and negative classes.</li><li>The most important stat from the above was found to be word-count.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*G68UpeQCU5IUwYcmEAdTBA.png" /></figure><p>Notice that as we move towards the right, the ratio of height/density of positive to negative class starts decreasing or we can say that negative class starts becoming more prominent. That means as word-count increases, the probability of relevance being High decreases. Thus, word-count gives some information about the relevance score.</p><p>4. And the above statement can be justified by looking at the correlation of word-count with the relevance. There is a little correlation and it’s negative in nature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8324c240c2a93b9bce453c8fda578f5/href">https://medium.com/media/b8324c240c2a93b9bce453c8fda578f5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*tSYOAtDyr0pLC3HfXGKfDg.png" /></figure><p>5. So word-count might not be that bad a feature.</p><p>6. number-of-characters and average-word-length had very low correlation<br>scores of -0.092 and 0.084 respectively, with relevance, hence might not be helpful.</p><p><em>All these stats were analyzed for title, description, and brand text as well. And the same pattern was found. The only stat which had any noticeable correlation with the relevance was word-count.</em></p><p><em>Apart from this, I also experimented on two more stats: readability-index and sentiment-score. The correlation scores for the two were -0.035 and 0.019 which are very low. And they did not help distinguish the two classes.</em></p><h4>Most common Unigrams and Bigrams</h4><p><strong>Search Text</strong>: Most common unigrams for corrected_search are as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5329c620b67af1ef43e8c2157690be79/href">https://medium.com/media/5329c620b67af1ef43e8c2157690be79/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jfGTn0DgolRNXICcnE6YFQ.png" /></figure><p>‘inch’ and ‘x’ are the most frequent terms. Also, a lot of numbers are very frequently occurring. This means that a lot of search terms contain the dimensions of the products needed. These numbers and units are important as they can be a good indication as to what kind of product the user desires. Hence I didn’t remove the numbers while data cleaning and will be keeping them for the final featurization part as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*C_SnRHoQoqrE2GA747zYpA.png" /><figcaption>Most frequent unigrams for each class</figcaption></figure><p>However, the most frequent words for both classes are very similar. On the left, you see the most frequent words in search for each class with their frequencies.</p><p>Even the most common bigrams for both classes were very similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uC59eiZ9w01MFYmuLiuOGw.png" /></figure><p>Hence, from the above observations, for search-text, we saw that most frequent unigrams and bigrams are very similar in both classes. Thus, vectorizations like simple BOW or TF-IDF BOW might not work that well.</p><p><strong>Description Text</strong>: Even in the description text, it was found that the most common word is inch, with numbers being prominent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stZTThIXmOZy7yghNgSSng.png" /></figure><p>The most frequent unigrams and bigrams for both classes were found to be very similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aaSpH2IK0U1-lAeJngraxQ.png" /></figure><p>The same analysis was done for brand and title text and a similar trend was found. The text fields in themselves did not give a lot of information about the relevance score.</p><h4>Conclusions from EDA2</h4><ol><li>Among all the stats on text, word-count was the most effective in separating the two classes. We could not get a lot of information from other stats like average-word-length and length-of-string, both of which showed very low correlation values. Hence in the final featurization, I will only include the word-count feature.</li><li>The unigrams and bigrams distributions of all the text fields (search, title, description, brand) are very similar for both classes. And hence simple vectorizations like BOW and TF-IDF BOW might not work that well on their own.</li></ol><h3>Feature Engineering</h3><p>The features for this problem were mostly derived from the field of<a href="https://en.wikipedia.org/wiki/Information_retrieval"> Information Retrieval</a>. Note that here, the features need to be arbitrary functions of both the search and the product combined, as we are trying to find the relevance for the pair of (search, product). Some basic retrieval models were used directly as features. I have divided the features into four sets:</p><ul><li>Set 1: Set-Theoretic Features</li><li>Set 2: Vector Space Models (VSM) based features</li><li>Set 3: Probabilistic Features</li><li>Set 4: Query Expansion using Word2Vec</li></ul><h4>Feature Set 1: Set-Theoretic Features</h4><p>These features are derived from simple set-theoretic retrieval models like the <a href="https://en.wikipedia.org/wiki/Boolean_model_of_information_retrieval">Boolean model</a>, <a href="https://en.wikipedia.org/wiki/Fuzzy_retrieval">Fuzzy Retrieval</a>, etc. which are based on representing queries and documents as sets of words and then applying some operation on them to get the relevance score. The features include:</p><p>➤ The count of common words between search and document-fields (title, brand, description)</p><p>➤ Cosine coefficient and Jaccard coefficient between search and document fields</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zj5JNCiF3H8r-FiY-L8dwA.png" /></figure><p>➤ Length of search and the document fields</p><p>➤ Whether the last word in the search is in the document fields or not</p><p><em>The last feature was taken intuitively because, in a multiple word search-query, the initial words generally describe the attributes of the product you need but the last word represents the very noun of product. Like in “black jacket”, the first word describes the color of the jacket but the last word is the very noun of the product you need.</em></p><p>The above features were created for both the corrected_search and the raw_search.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7200774ce948ca125990c1c57d25aabc/href">https://medium.com/media/7200774ce948ca125990c1c57d25aabc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BRir7srES7s07-R3uFcxUA.png" /><figcaption>Comparison between corrected_search and raw_search</figcaption></figure><ul><li>It is observed that there is a considerable amount of rows in which the num-common-words feature is more for corrected_search than the raw_search. Hence, correcting the search terms did help.</li></ul><h4>EDA on Feature Set 1</h4><p>A total of 29 features were created here. The PDF (Probability Density Function) plots of a few are given below.</p><p><em>Note: The features suffixed with ‘ST’ represent that between the corrected_search and the title, ‘SB’ is for brand and ‘SD’ for description. For raw_search, I have used suffix in the form of ‘r_ST’. For eg. ‘cosine_ST’ represents the cosine coefficient between corrected_search and title, whereas ‘cosine_r_ST’ represents that between raw_search and title.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_w25kUj6CnupFxmUrzL0Q.png" /><figcaption>PDF plots of set 1 features</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*RFgQtHd_5TR0E9_LmAhGQg.png" /></figure><p>It’s difficult to make sense out of them unless you look closely. Take cosine_SD as an example. Generally, in all the PDFs, we see that the curve for the positive or “High” class is above the negative or “Low” class. This is plainly due to data imbalance as there are more points belonging to the positive class than the negative. You can see that at the mark 0, the curve for the positive class is a little above negative. As you move towards the right, the positive curve still remains above but the ratio of the height of the positive curve to the negative curve increases. That is, as cosine_SD increases, the ratio of points in the positive class to the negative class increases. Hence the probability of the point belonging to positive class increases. Now if you give me two points, then I can say that the point having a larger value of cosine_SD should have a higher relevance score.</p><p>Thus, from the above observation, we can conclude that cosine_SD does provide some information about the relevance score for a (search, product-text) pair.</p><p>The same analysis was done for all the features in this set including the ones for raw_search and the <strong>key observations</strong> are:</p><ol><li>The features num_common, cosine, Jaccard showed considerable differences in distributions of positive and negative class. Among them, cosine_ST was the most effective.</li><li>The features suffixed with ST i.e. the features between search and title showed more distinction in distributions of the positive and negative class as compared to SD and SB.</li><li>Length features didn’t show a lot of difference in distributions.</li><li>One of the most interesting features was the islast feature. A pretty considerable difference in distribution was observed. For eg., in islast_ST the density of the negative class seems to be similar at both 0 and 1 values but the density of the positive class increases considerably from 0 to 1.</li></ol><p>✦ So as per the observations, cosine and islast features are the most important features followed by Jaccard. Among them, ST features should show the best results.</p><p>The correlation matrix for these features was also constructed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DEph445178KiCNVaS73hsw.png" /><figcaption>correlation matrix for set 1 features</figcaption></figure><p>And their correlation scores with relevance were found</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de3419f95cd020394ce93af6e0eaa218/href">https://medium.com/media/de3419f95cd020394ce93af6e0eaa218/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aXrAKcwTBFL70F-Ih0sPQQ.png" /><figcaption>Most correlated set 1 features to the relevance score</figcaption></figure><p><strong><em>Note:</em></strong><em> A desired feature set would be a set in which all the features have a high correlation score with relevance and low correlation scores with each other.</em></p><p><strong>Observations and Conclusions:</strong></p><ol><li>There are a few features that have a high correlation with each other. These correlated features can be redundant and hence can be removed while modeling.</li><li>The correlation scores wrt. relevance are not very high but some correlation does exist. Hence, these features can be useful.</li><li>We can see that correlation of cosine_ST, jaccard_ST, and islast_ST are highest, exactly as we predicted.</li><li>These values were also calculated for the raw_search and in general, it’s seen that correlation of corrected_search features with relevance is more than that for raw_search features.</li><li>I also applied <a href="https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html">T-SNE</a> on these features to visualize them in 2-D space. I projected the embedded features into 2-D space and colored them with their individual classes. I tried it with the two classes from before and also with seven classes. But it did not give any insight at all. None of the classes showed any clustering.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcNlZ8EnYAQSd0094wxkjQ.png" /><figcaption>Visualizing TSNE features with seven and two classes respectively.</figcaption></figure><h4>Feature Set 2: VSM based Features</h4><p><a href="https://en.wikipedia.org/wiki/Vector_space_model">VSM</a> based features are typically based on representing documents and queries as vectors and then calculating the similarity between them as a scalar quantity.</p><p>Here, to convert the text into vectors, I used <a href="https://en.wikipedia.org/wiki/Latent_semantic_analysis">Latent semantic indexing</a> where <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">Truncated-SVD</a> is applied to the term-document matrix such that the queries and documents are converted to vectors in the “concept space”. Basically, here I first vectorized the text using TF-IDF BOW and then used Truncated-SVD to reduce the dimensions of the vectors and create a more dense and meaningful representation. The features include:</p><p>➤ Apply LSI on the term-document matrices of search, title, and description separately and use the low-rank approximations as features directly.</p><p>➤ Apply LSI on full text (combined title and description). Then, transform the search query into the ‘concept’ space and calculate cosine similarity between them.</p><h4>EDA on Feature Set 2</h4><p>The PDFs for the cosine-similarity features are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*JEjj3HVXoVd3GEIarxNBsg.png" /><figcaption>PDFs of set 2 cosine-similarity features</figcaption></figure><p>As you can see, the similarity features provide a considerable distinction between the two classes and were very useful. The correlation values of each of these three with the relevance class were found to be 0.32 which is pretty good.</p><h4>Feature Set 3: Probabilistic Features</h4><p>These features are based on the core concepts in <a href="https://en.wikipedia.org/wiki/Probabilistic_relevance_model">probability theory</a> like the Bayes Theorem. The features used are:</p><p>➤ <a href="https://en.wikipedia.org/wiki/Language_model">Language Models</a> with Dirichlet, Absolute, and Jelinek Miller smoothing</p><p>➤ <a href="https://en.wikipedia.org/wiki/Okapi_BM25">Okapi BM25</a> ranking function</p><p>➤ Query and fields represented as TF-IDF Word2Vec</p><p>➤ (sum, min, max) of (tf, idf, tf-idf) for the search query in each of the text field</p><p><strong>The BM25 Ranking Function: <br></strong>Out of these features, I would like to focus on the BM25 ranking function, as it on its own was used for text querying in the traditional search engines and is still very popular for simple free-text querying. For a given (query, document) pair, it calculates a similarity score which is as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/0*h-AFYojKv8DmHQlL" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/Okapi_BM25">https://en.wikipedia.org/wiki/Okapi_BM25</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W43hvVXlY0zVrpj2nY6UiQ.png" /><figcaption>Image by Author, inspired by the <a href="https://www.youtube.com/playlist?list=PLBv09BD7ez_56YwzSyQOOsrAE4Js7XuAB">lectures by Victor Lavrenko</a> on Probabilistic Models for IR</figcaption></figure><p>Above is an attempt at explaining the whole equation. It is basically a modified form of TF-IDF weighting where the TF value has been modified to incorporate the varying length of the documents and to bring non-linearity in the increase of the score wrt term-frequency. For understanding it better, I encourage you to go through the following <a href="https://www.youtube.com/watch?v=e3hSc6SC29Q&amp;list=PLBv09BD7ez_56YwzSyQOOsrAE4Js7XuAB">lecture</a> by Victor Lavrenko.</p><h4>EDA on Feature Set 3</h4><p>A similar analysis was done and all the features had a decent correlation value with the relevance score but a little lower compared to set 1 and set 2. Interestingly enough, the feature with the highest correlation was found to be min_tf_ST i.e the minimum term-frequency of search words in the title text. And intuitively also, it should be a good feature; min_tf_ST = 0 means that there are words in search which do not occur in that title and min_tf_ST ≠ 0 would mean that all the words in the search are found in the title thus signifying high relevance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*FE7bJJNHZ67XdXRgfHDm3w.png" /><figcaption>most correlated set 3 features to relevance</figcaption></figure><p>The PDF plots of the ST features i.e between search and title are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8FdAQRNt6kkxt9c-6Ruk2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKAEhumTrqJr_n_3t2JBAQ.png" /><figcaption>PDFs of set 3 features</figcaption></figure><p><strong>Observations and Conclusions:</strong></p><ol><li>Among the tf, idf, tf-idf features, the tf features i.e. min_tf, max_tf and sum_tf are the most effective. Whereas the idf features are not able to separate the two classes and don&#39;t show a high correlation with relevance score as well.</li><li>The bm25 and the language model features (JM, AD, Dir) do show distinction between the two classes. Note that the language models which I designed have a negative correlation with relevance hence lower values would point to high relevance</li><li>The only caveat with all these features was that they had a very high correlation value with each other hence there was a lot of redundancy which might affect the linear models like Linear Regression and SVM. To tackle this, one can remove one of each of these highly correlated features.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05467ea79b00156cd746129a1860c605/href">https://medium.com/media/05467ea79b00156cd746129a1860c605/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*vkPB5V_L43vzefJQoP5aJw.png" /><figcaption>High correlation between AD_SB and Dir_SB features</figcaption></figure><h4>Feature Set 4: Query Expansion</h4><p><a href="https://en.wikipedia.org/wiki/Query_expansion">Query Expansion</a> is a very simple concept to understand.</p><ul><li>First, all the words in the search query are converted to word vectors using Word2Vec.</li><li>Then, let q be a user query represented by a bag of terms, q = (t1, t2, . . . , t|q| )</li><li>To expand a query q, we follow • For each t ∈ q, collect the k-most similar terms to t using w2v. • Include the new terms into the query set giving us q.</li></ul><p>After query expansion, we can use the expanded search queries to recalculate all the basic features from the feature set1.</p><p>I did perform a bit of EDA on this set and results observations were very similar to the search without expansion, i.e. expanding the terms did not prove to be very helpful. That might be because I trained a custom word2vec model on my corpus and the model might not have been trained well. With a well-trained model, I believe query expansion could be very helpful.</p><h3>Performance Chart of the Feature Sets</h3><p>On these feature sets, I trained an <a href="https://xgboost.readthedocs.io/en/latest/python/python_api.html">XGBoost</a> model with some hyperparameter tuning. I kept on adding the feature sets, checking for any improvements and the results were very positive. The training was done on the whole train data with 74067 data points. For evaluation, the metric used was RMSE with 5-fold cross-validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0uAEBSEzDTX2m2RvaBn1Zw.png" /><figcaption>Performance chart of feature sets</figcaption></figure><h4>Observations and Conclusions :</h4><ol><li>The error continuously reduced as we added more features.</li><li>The error for f1_raw was 0.4805 which improved a little with f1_corrected. It improved considerably to 0.4763 when we used both the feature sets combined. Hence both raw and corrected feature sets do improve the overall performance.</li><li>Adding the f2_lsi set also saw a considerable drop in error to 0.471.</li><li>The most improvement was observed when we added the feature set 3 where the error dropped from 0.471 to 0.4547. Not only this, feature set 3 alone performed very well with the error of 0.4569. And hence this was the most important set of all.</li><li>The features from query expansion that is f4 did not bring a lot of improvement.</li><li>The best score observed was 0.4541 on features from all the sets combined.</li></ol><h3>Final Modeling and Evaluation</h3><h4>Stacking Regressor</h4><p>The XGBoost models in the previous section were built mainly to observe and compare the performance of different feature sets. For the final modeling part, I built a custom Stacking Regressor with 17 base models and a Ridge Regression model as the meta-model. For this, I proceeded as follows:</p><ol><li>Split the whole data from the train.csv file (74067 points) into train and test set(80–20)</li><li>Now, I split the 80% train set into D1 and D2.(50–50). So D1 and D2 have about 29k points each.</li><li>From D1, I did sampling with replacement to create 17 samples d1,d2,d3….d17 for my 17 base models. The sampling ratio used was between 0.85 and 0.9. Thus to train each of the base models, around 25k data points were used.</li><li>Then I created 17 models and train each of these models with each of these 17 samples.</li><li>I passed the D2 set to each of these 17 models and got 17 predictions for D2 from each of these models.</li><li>Now, using these 17 predictions, I created a new dataset, and for D2, I already knew its corresponding target values, so then I trained my Meta-Model with these 17 predictions as features. The Meta Model hence was trained with 29k points each having 17 features.</li><li>For model evaluation, I use the 20% data that I kept as the test set in the beginning. I passed that test set to each of the base models, got the 17 predictions, created a new dataset with these 17 predictions and finally, passed it to the metamodel to get the final prediction. Using this final prediction and the targets for the test set, I calculated the model’s performance score.</li></ol><h4>Base Models</h4><p>I divided the base models into three sets M1, M2, and M3. Mainly tree-based and linear models were used.</p><ul><li>M1 models were trained on a total of 71 base features. These are f1_comb(29) + f2_lsi(3) + f3(39). A variety of models (total 7) were trained on this set.</li><li>M2 models were trained on a total of 971 features. 71 of these were the base features. The other 900 features are from Word2Vec embeddings for search, title, and description.</li><li>M3 models were trained on a total of 4071 features: 71 base features and the rest 4000 from Truncated-SVD in feature set 3. For this set, due to its high dimensionality, I stuck to linear models only.</li></ul><p>For cross-validation, I generally used 5-fold cross-validation. The performance report of the base models is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*sExlXLEcOPn4zNA1-NLUVg.png" /><figcaption>Performance of base models</figcaption></figure><h4>Meta Model</h4><p>For the meta-model, I used a simple ridge regression model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58d3710d794ef2dae4a8bcfeb3113523/href">https://medium.com/media/58d3710d794ef2dae4a8bcfeb3113523/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*ktOVvz-Maect_jDSwNlInQ.png" /></figure><p>After hyperparameter tuning, I got a final RMSE of <strong>0.4539 </strong>on the test set. On the train set, I got an RMSE of 0.4531<strong> </strong>hence the model was not overfitting also.</p><h4>Performance on Kaggle</h4><p>I tested my final model on the test set provided by Kaggle containing over 160,000 datapoints and it got a Private Score of <strong>0.46525. </strong>It ranked in the <strong>top 10% </strong>out of more than two thousand teams that participated in the competition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJgG-rSmNs0qTq_Ua7uS4Q.png" /><figcaption>Kaggle: Top 10%</figcaption></figure><h3>Extending into a Real-World Search Engine</h3><p>As mentioned earlier in the article, in a real-world search engine, for any search-query, we would first retrieve a few candidate products using a simpler retrieval model. And then, a machine-learned model is used to re-rank these products.</p><p>We have already built the machine-learned model in the form of a stacking regressor. For the initial retrieval part, I first created a database out of all the products in the train.csv and the test.csv files. The database contained 124428 unique products with the fields being product_uid, title, description, and attributes. For the model, I use <a href="https://en.wikipedia.org/wiki/Okapi_BM25">BM25</a>, which we have also used as a feature for our machine learning model. Given a search query, the BM25 model retrieves the top 100 products based on its custom ranking function.</p><p>Now on top of these 100 products, I run my stacking regressor to which as input I give 100 pairs of (search, product-text) where the search remains constant but the products vary. And so my model then ranks them and we can then show the products to the customer with the highest relevance scores.</p><h3><strong>Full Pipeline</strong></h3><p>The full pipeline of the model is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*baCt3v3AVPS67Mlkg_TbIg.png" /><figcaption>Full Pipeline</figcaption></figure><h3>Final Testing and Results</h3><p>So now the whole search engine was complete. I tested it for a bunch of queries. And the results were very interesting. I am attaching the screenshots of some results below.</p><p>First, a simple search “pan” and the results are good enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dxbW8M4FjJKeVmfTIuF20g.png" /></figure><p>Then I searched for “air conditioner” and then with a spelling error “aor condiotioner”. And the results for both are exactly the same. This means that our spelling corrector is working fine!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*CeL-VxQCacphEKqNlLSRdQ.png" /><figcaption>‘air conditioner’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*Sw8adQ6YVuvLjitFsqjT4Q.png" /><figcaption>‘aor condiotioner’</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FENXT_5q8KADhkoHMD0_3A.png" /><figcaption>‘warm water’</figcaption></figure><p>For “warm water”, the results are very interesting. None of the product titles have both the words ‘warm’ and ‘water’ in them. In the top results, you can see that all the titles have either of the words present. Then what is the reasoning for the first title to be given the first rank? This is due to the presence of the word ‘Icicle’ for which the definition is “a hanging, tapering piece of ice formed by the freezing of dripping water”. The word embeddings that we used must have encoded it in such a way that the encoding is similar to the word ‘water’. Hence our model found an association of ‘icicle’ with ‘water’ and displayed the results in such a manner. Also notice, in the next three results, we have gotten more desired results in the form of water heaters. Here also, the word embeddings much have made the model know that the word ‘warm’ is associated with the word ‘heat’ and hence the results.</p><p>Results of a few more searches are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*FUEzTtAk9G86Ad06MtQCLA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*KmbpDQ8ihnU-YZYLIeVWtw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*A_eM15IijjmcYsak63Dc9w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*Nwi0f3L-icrJhSwHXJxPAA.png" /></figure><p><strong>Run Time<br></strong>The run time varied for different searches. For one-word simple searches like “pan”, the average run-time was around 5.09 seconds but for long searches and the ones requiring spelling correction, the run time was between 5.5 and<strong> </strong>5.7 seconds.</p><p><strong>Note: </strong>The main purpose of this case study was to model product search, mainly focusing on coming up with useful features on which we can apply our ML algorithms. We were not focused so much on the design and optimization part. For real-world use, this search engine can be optimized like crazy and it wouldn&#39;t take long for the run time to be reduced to below one second.</p><p><strong>Web Application Demo</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8Ygff8fEJzU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8Ygff8fEJzU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8Ygff8fEJzU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7197b97be0f5d3382b4095ee41805bcc/href">https://medium.com/media/7197b97be0f5d3382b4095ee41805bcc/href</a></iframe><h3>Conclusion and Future Work</h3><p>In this case study, I explored a variety of techniques from the field of Information Retrieval to extract features from the data provided. The techniques ranged from simple set operators to Latent Semantic Indexing. LSI-based similarities and traditional retrieval models such as BM25 and Language Models have proven to be extremely effective. For modeling, I built a Stacking Regressor with 17 base models and Ridge Regressor as the meta-model which got a final score of 0.4652 on Kaggle. Further, I extended this to a full-fledged search engine by adding an initial retrieval model in the form of BM25.</p><p>There is still a significant difference between my score and the top-ranked team’s (0.4319). Thus, it’s clear that much more work needs to be done to improve the model. Here are a few suggestions:</p><ol><li>The text data we have is pretty different from the general text found on the internet. Hence, whenever it&#39;s possible to train a model on our own custom data then we should explore that possibility. For example, we can train our own Word2Vec model. With this, Para2Vec and Sent2Vec models can also be explored.</li><li>A better spelling corrector can be implemented as this could make the model much more robust.</li><li>The attributes text was not used in any of the featurization. It can be included.</li><li>In the preprocessing part, lemmatization can be performed.</li><li>There must brands which have similar types of products, hence the brands can be clustered, and then the cluster numbers can be used as a categorical feature itself.</li><li>Only the “count” of common words was used as a feature. We can also try to find a way to embed these common words to tell our model which kind of common words are present.</li><li>In query expansion, for calculating similar words, better vectorization techniques can be explored</li><li>For the modeling part deep learning models like LSTMs, GRUs, BERT, CNNs can be explored.</li></ol><h3>Acknowledgments</h3><p>I would like to thank the whole <a href="https://www.appliedaicourse.com/">AppliedAICourse</a> team for guiding me throughout the Case Study.</p><h3>References</h3><ol><li><a href="https://www.kaggle.com/c/home-depot-product-search-relevance">https://www.kaggle.com/c/home-depot-product-search-relevance</a></li><li><a href="https://en.wikipedia.org/wiki/Information_retrieval">https://en.wikipedia.org/wiki/Information_retrieval</a></li><li><a href="https://www.researchgate.net/publication/232614940_A_Boolean_Model_in_Information_Retrieval_for_Search_Engines_PDF">Boolean Model in Infomation Retrieval for Search Engines</a></li><li><a href="https://www.sciencedirect.com/science/article/abs/pii/S0165011409001080?via%3Dihub">Fuzzy information retrieval model revisited</a> by SławomirZadrożny and KatarzynaNowacka</li><li><a href="https://arxiv.org/abs/0911.5046">Integrating the Probabilistic Model BM25/BM25F into Lucene</a></li><li><a href="https://en.wikipedia.org/wiki/Learning_to_rank">https://en.wikipedia.org/wiki/Learning_to_rank</a> by Joaquín Pérez-Iglesias, José R. Pérez-Agüera, et al.</li><li><a href="http://ciir.cs.umass.edu/pubfiles/ir-407.pdf">Indri: A language-model-based search engine for complex queries</a> by Trevor Strohman, Donald Metzler, Howard Turtle, and W. Bruce Croft.</li><li><a href="http://delab.csd.auth.gr/~dimitris/courses/ir_spring06/page_rank_computing/01cc99333c00501ddab030.pdf">Using Linear Algebra for Information Retrieval</a> by Michael W. Berry, et al.</li><li><a href="https://www.researchgate.net/publication/310823543_Query_Expansion_Using_Word_Embeddings">Query Expansion Using Word Embeddings</a> by Saar Kuzi, et al.</li><li><a href="https://arxiv.org/pdf/1803.05127.pdf">Feature Selection and Model Comparison on Microsoft Learning-to-Rank Data Sets</a> by Sen Lei, et al.</li><li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.104.3479&amp;rep=rep1&amp;type=pdf">The Vector Space Model in Information Retrieval: Term Weighting Problem</a></li><li><a href="https://opensourceconnections.com/blog/2016/03/29/semantic-search-with-latent-semantic-analysis/">https://opensourceconnections.com/blog/2016/03/29/semantic-search-with-latent-semantic-analysis/</a></li><li><a href="http://www.ccs.neu.edu/home/jaa/CSG339.06F/Lectures/vector.pdf">http://www.ccs.neu.edu/home/jaa/CSG339.06F/Lectures/vector.pdf</a></li><li><a href="https://www.youtube.com/watch?v=3DC5hlge_bs&amp;list=PLBv09BD7ez_7Ke6U7yGBvfP4_Hau3ZGj2&amp;index=2">Video Lecture on Language Models for IR</a> by Victor Lavrenko</li><li><a href="http://datasciencebar.github.io/blog/home-depot-produce-relevance-review/">http://datasciencebar.github.io/blog/home-depot-produce-relevance-review/</a></li><li><a href="https://medium.com/kaggle-blog/home-depot-product-search-relevance-winners-interview-2nd-place-thomas-sean-qingchen-nima-68068f9f9ffd">https://medium.com/kaggle-blog/home-depot-product-search-relevance-winners-interview-2nd-place-thomas-sean-qingchen-nima-68068f9f9ffd</a></li><li><a href="https://arxiv.org/pdf/1803.05127.pdf">https://arxiv.org/pdf/1803.05127.pdf</a></li><li><a href="http://billy-inn.github.io/papers/cmput690.pdf">http://billy-inn.github.io/papers/cmput690.pdf</a></li><li><a href="https://arxiv.org/pdf/2001.04980.pdf">https://arxiv.org/pdf/2001.04980.pdf</a></li><li><a href="https://www.appliedaicourse.com/">https://www.appliedaicourse.com</a></li></ol><p>Full work with code can be found here on my Github profile:</p><p><a href="https://github.com/kriz17/Home-Depot-Product-Search-Relevance">kriz17/Home-Depot-Product-Search-Relevance</a></p><p>Connect with me on LinkedIn:</p><p><a href="https://www.linkedin.com/in/kriz-moses/">Kriz Moses - IIT Indore - Indore, Madhya Pradesh, India | LinkedIn</a></p><p><strong>PS</strong>: Feel free to provide comments/criticisms if you think they can improve this blog, I will definitely try to make the required changes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ccb56fbc5ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/modeling-product-search-relevance-in-e-commerce-home-depot-case-study-8ccb56fbc5ab">Modeling Product Search Relevance in e-Commerce: Home Depot Case Study</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Encoder-Decoder Seq2Seq Models, Clearly Explained!!]]></title>
            <link>https://medium.com/analytics-vidhya/encoder-decoder-seq2seq-models-clearly-explained-c34186fbf49b?source=rss-bbb5020375a9------2</link>
            <guid isPermaLink="false">https://medium.com/p/c34186fbf49b</guid>
            <category><![CDATA[sequence-to-sequence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-translation]]></category>
            <category><![CDATA[encoder-decoder]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kriz Moses]]></dc:creator>
            <pubDate>Fri, 12 Mar 2021 05:23:41 GMT</pubDate>
            <atom:updated>2024-07-30T17:39:37.907Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step guide to understanding Encoder-Decoder Sequence-to-Sequence models in detail!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzVwaKGaKqAgVCW4MH08sQ.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/coxinhafotos-3726685/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=5233295">CoxinhaFotos</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=5233295">Pixabay</a></figcaption></figure><h3>Table of Contents</h3><ol><li>Introduction</li><li>Motivation</li><li>Prerequisites</li><li>Sequence Modelling problems</li><li>The Architecture of Encoder-Decoder models</li><li>Training and Testing Phase: Getting into the tensors</li><li>The “Sutskever Model”</li><li>Image Captioning Intuition</li><li>Further Reading</li><li>References</li></ol><h3>Introduction</h3><blockquote>The traditional Deep Neural Networks (DNNs) are powerful machine learning models that achieve excellent performance on difficult problems such as speech recognition and visual object recognition. But they can only be used for large labeled data where inputs and targets can be sensibly encoded with vectors of fixed dimensionality. They cannot be used to map sequences-to-sequences (for eg. machine translation)</blockquote><blockquote><a href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural Networks</a> [1]</blockquote><p>This was the motivation behind coming up with an architecture that can solve general sequence-to-sequence problems and so encoder-decoder models were born.</p><p>In this article, I aim to explain the encoder-decoder sequence-to-sequence models in detail and help build your intuition behind its working. For this, I have taken a step-by-step approach and tried to explain the concepts <strong>visually</strong>. I will mainly refer to the seminal research paper <a href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural Networks </a><a href="https://arxiv.org/abs/1409.3215">by Ilya Sutskever, et al</a> which implemented encoder-decoder models for neural machine translation. In the end, I will also give the intuition behind a modified version of encoder-decoder used for image captioning in the paper<em> </em><a href="https://arxiv.org/abs/1412.2306">Deep Visual-Semantic Alignments for Generating Image Descriptions by Andrej Karpathy, Li Fei-Fei, et al</a></p><h3>Motivation</h3><p>The Encoder-Decoder architecture is relatively new and had been adopted as the core technology inside Google’s translate service in late 2016. It forms the basis for advanced sequence-to-sequence models like Attention models, GPT Models, Transformers, and BERT. Hence, understanding it can be very crucial before moving onto advanced mechanisms.</p><h3>Prerequisites</h3><p>This post assumes that you are familiar with neural networks and, in particular, the working of RNNs/LSTMs. If you are unsure, then I strongly encourage you to check the blogs <a href="https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9">Illustrated Guide to Recurrent Neural Networks</a><em> </em>by<em> </em>Michael Phi<em> and</em> <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">Understanding LSTM Networks</a> by Christopher Olah.</p><h3>Sequence Modelling Problems</h3><p>Sequence Modelling problems refer to the problems where either the input and/or the output is a sequence of data (words, letters…etc.)</p><p>Consider a very simple problem of predicting whether a movie review is positive or negative. Here our input is a sequence of words and output is a single number between 0 and 1. If we used traditional DNNs, then we would typically have to encode our input text into a vector of fixed length using techniques like BOW, Word2Vec, etc. But note that here the sequence of words is not preserved and hence when we feed our input vector into the model, it has no idea about the order of words and thus it is missing a very important piece of information about the input.</p><p>Thus to solve this issue, RNNs came into the picture. In essence, for any input X = (x₀, x₁, x₂, … xₜ) with a variable number of features, at each time-step, an RNN cell takes an item/token <strong>xₜ</strong> as input and produces an output <strong>hₜ</strong> while passing some information onto the next time-step. These outputs can be used according to the problem at hand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*HLoBU9zzXITTJWK8aBvzFQ.png" /><figcaption><a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">Understanding LSTM Networks</a> by Christopher Olah.</figcaption></figure><p>The movie review prediction problem is an example of a very basic sequence problem called many to one prediction. There are different types of sequence problems for which modified versions of this RNN architecture are used.</p><p>Sequence problems can be broadly classified into the following categories:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kje9TpsSvMGeBF6vIv6k5Q.jpeg" /><figcaption><a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">The Unreasonable Effectiveness of Recurrent Neural Networks</a> by Andrej Karpathy</figcaption></figure><blockquote>Each rectangle is a vector and arrows represent functions (e.g. matrix multiply). Input vectors are in red, output vectors are in blue and green vectors hold the RNN’s state (more on this soon). From left to right: <strong>(1)</strong> Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g. image classification). <strong>(2)</strong> Sequence output (e.g. image captioning takes an image and outputs a sentence of words). <strong>(3)</strong> Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment). <strong>(4)</strong> Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French). <strong>(5)</strong> Synced sequence input and output (e.g. video classification where we wish to label each frame of the video). Notice that in every case are no pre-specified constraints on the lengths sequences because the recurrent transformation (green) is fixed and can be applied as many times as we like.</blockquote><blockquote>— Andrej Karpathy, <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">The Unreasonable Effectiveness of Recurrent Neural Networks</a> [5]</blockquote><h4>Sequence-to-Sequence Problems</h4><p>Sequence-to-Sequence (Seq2Seq) problems is a special class of Sequence Modelling Problems in which both, the input and the output is a sequence. Encoder-Decoder models were originally built to solve such Seq2Seq problems. In this post, I will be using a many-to-many type problem of Neural Machine Translation (NMT) as a running example. Later, we also see how they can also be used for image captioning (one-to-many example).</p><h3>The Architecture of Encoder-Decoder models</h3><p>My main goal is to help you understand the architecture used in the paper <a href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural Networks<em> </em>by Ilya Sutskever, et al</a>. This was one of the first papers to introduce the Encoder-Decoder model for machine translation and more generally sequence-to-sequence models. The model was applied to English to French translation.</p><p><strong><em>NOTE: To get to the final model, I will attempt to simplify things and introduce the concepts one by one to hopefully make it easier for people to understand without too much in-depth knowledge of the subject matter, filling in the gaps as and when needed.</em></strong></p><h4>The Neural Machine Translation Problem</h4><p>To help you understand better, I will be taking Neural Machine Translation as a running example throughout this post. In neural machine translation, the input is a series of words, processed one after another. The output is, likewise, a series of words.</p><p><strong>Task</strong>: Predict the French translation for every English sentence used as input.</p><p>For simplicity let’s consider that there is only a single sentence in our data corpus. So let our data be:</p><ul><li><strong>Input: English sentence: “nice to meet you”</strong></li><li><strong>Output: French translation: “ravi de vous rencontrer”</strong></li></ul><h4><strong><em>Terms Used</em></strong></h4><p>To avoid any confusion:</p><ul><li>I will refer to the Input sentence <strong>“nice to meet you” </strong>as<strong> X/input-sequence</strong></li><li>I will refer to the output sentence <strong>“ravi de vous rencontrer” </strong>as<strong> Y_true/target-sequence </strong>→ This is<strong> </strong>what we want our model to predict (the ground truth).</li><li>I will refer to the predicted output sentence of the model as <strong>Y_pred/predicted-sequence</strong></li><li>The individual words of the English and French sentence are referred to as <strong>tokens</strong></li></ul><p>Hence, given the input-sequence “nice to meet you”, we want our model to predict the target-sequence/Y_true i.e “ravi de vous rencontrer”</p><h4>High-Level Overview</h4><p>At a very high level, an encoder-decoder model can be thought of as two blocks, the encoder and the decoder connected by a vector which we will refer to as the ‘context vector’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*GvN7ekDa8rFPAZIc_xUmIw.png" /><figcaption>Image by author</figcaption></figure><ul><li><strong>Encoder</strong>: The encoder processes each token in the input-sequence. It tries to cram all the information about the input-sequence into a vector of fixed length i.e. the ‘context vector’. After going through all the tokens, the encoder passes this vector onto the decoder.</li><li><strong>Context vector</strong>: The vector is built in such a way that it&#39;s expected to encapsulate the whole meaning of the input-sequence and help the decoder make accurate predictions. We will see later that this is the final internal states of our encoder block.</li><li><strong>Decoder</strong>: The decoder reads the context vector and tries to predict the target-sequence token by token.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*6yI-Ecx36JYixgomc-inPg.png" /><figcaption>Image by author</figcaption></figure><h4><strong>What’s under the hood?</strong></h4><p>The internal structure of both the blocks would look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0aHodc667UfSyZj-UY8OQw.png" /><figcaption>Image by author</figcaption></figure><p>As far as architecture is concerned, it&#39;s quite straightforward. The model can be thought of as two LSTM cells with some connection between them. The main thing here is how we deal with the inputs and the outputs. I will be explaining each part one by one.</p><h4>The Encoder Block</h4><p>The encoder part is an LSTM cell. It is fed in the input-sequence over time and it tries to encapsulate all its information and store it in its final internal states<strong> hₜ </strong>(hidden state)<strong> </strong>and <strong>cₜ </strong>(cell state). The internal states are then passed onto the decoder part, which it will use to try to produce the target-sequence. This is the ‘context vector’ which we were earlier referring to.</p><p>The outputs at each time-step of the encoder part are all discarded</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*W02VVwUd0aDiitTnfopx0g.png" /><figcaption>Image by author</figcaption></figure><p><em>Note: Above diagram is what an LSTM/GRU cell looks like when we unfold it on the time axis. i.e. it is the single LSTM/GRU cell that takes a single word/token at each timestamp.</em></p><p><em>In the paper, they have used LSTMs instead of classical RNNs because they work better with long-term dependencies. I have seen people use GRUs as well.</em></p><h4>The Decoder Block</h4><p>So after reading the whole input-sequence, the encoder passes the internal states to the decoder and this is where the prediction of output-sequence begins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*yYL1eM6UWbBZa9N975ShpA.png" /><figcaption>Image by author</figcaption></figure><p>The decoder block is also an LSTM cell. The main thing to note here is that the initial states <strong>(h₀, c₀)</strong> of the decoder are set to the final states <strong>(hₜ, cₜ) </strong>of the encoder. These act as the ‘context’ vector and help the decoder produce the desired target-sequence.</p><p>Now the way decoder works, is, that its output at any time-step <strong>t</strong> is<strong> </strong>supposed to be the <strong>tᵗʰ </strong>word in the target-sequence/Y_true (“ravi de vous rencontrer”). To explain this, let&#39;s see what happens at each time-step.</p><p><strong>At time-step 1</strong></p><p>The input fed to the decoder at the first time-step is a special symbol <strong>“&lt;START&gt;”</strong>. This is used to signify the start of the output-sequence. Now the decoder uses this input and the internal states <strong>(hₜ, cₜ) </strong>to produce<strong> </strong>the output in the 1st time-step which is supposed to be the 1st word/token in the target-sequence i.e. <strong>‘ravi’</strong>.</p><p><strong>At time-step 2</strong></p><p>At time-step 2, the output from the 1st time-step <strong>“ravi”</strong> is fed as input to the 2nd time-step. The output in the 2nd time-step is supposed to be the 2nd word in the target-sequence i.e. <strong>‘de’</strong></p><p>And similarly, the output at each time-step is fed as input to the next time-step. This continues till we get the “&lt;END&gt;” symbol which is again a special symbol used to mark the end of the output-sequence. The final internal states of the decoder are discarded.</p><p><em>Note that these special symbols need not necessarily be “</em>&lt;START&gt;” and “&lt;END&gt;” <em>only. These can be any strings given that these are not present in our data corpus so the model doesn’t confuse them with any other word. In the paper, they have used the symbol </em>“&lt;EOS&gt;” <em>and in a slightly different manner</em>. <em>I will be talking a little bit more about this later.</em></p><p><strong>NOTE: </strong>The process mentioned above is how an <strong>ideal </strong>decoder will work in the testing phase. But in the training phase, a slightly different implementation is required, to make it train faster. I have explained this in the next section.</p><h3>Training and Testing Phase: Getting into the tensors</h3><h4>Vectorizing our data</h4><p>Before getting into details of it, we first need to vectorize our data.</p><p>The raw data that we have is</p><ul><li><strong>X =</strong> <strong>“nice to meet you” → Y_true = “ravi de vous rencontrer”</strong></li></ul><p>Now we put the special symbols “&lt;START&gt;” and “&lt;END&gt;” at the start and the end of our target-sequence</p><ul><li><strong>X =</strong> <strong>“nice to meet you” → Y_true = “&lt;START&gt; ravi de vous rencontrer &lt;END&gt;”</strong></li></ul><p>Next the input and output data is vectorized using one-hot-encoding (ohe). Let the input and output be represented as</p><ul><li><strong>X = (x1, x2, x3, x4) → Y_true = (y0_true, y1_true, y2_true, y3_true, y4_true, y5_true)</strong></li></ul><p>where xi’s and yi’s represent the ohe vectors for input-sequence and output-sequence respectively. They can be shown as:</p><p><strong>For input X</strong></p><p>‘nice’ → x1 : [1 0 0 0]</p><p>‘to’ → x2 : [0 1 0 0 ]</p><p>‘meet’ →x3 : [0 0 1 0]</p><p>‘you’ → x4 : [0 0 0 1]</p><p><strong>For Output Y_true</strong></p><p>‘&lt;START&gt;’ → y0_true : [1 0 0 0 0 0]</p><p>‘ravi’ → y1_true : [0 1 0 0 0 0]</p><p>‘de’ → y2_true : [0 0 1 0 0 0]</p><p>‘vous’ → y3_true : [0 0 0 1 0 0]</p><p>‘rencontrer’ → y4_true : [0 0 0 0 1 0]</p><p>‘&lt;END&gt;’ → y5_true : [0 0 0 0 0 1]</p><p><em>Note: I have used this representation for easier explanation. Both the terms ‘true sequence’ and ‘target-sequence’ have been used to refer to the same sentence “ravi de vous rencontrer” which we want our model to learn.</em></p><h4>Training and Testing of Encoder</h4><p>The working of the encoder is the same in both the training and testing phase. It accepts each token/word of the input-sequence one by one and sends the final states to the decoder. Its parameters are updated using backpropagation over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5nD2k6cRx1GJwB_qTtypEw.png" /><figcaption>Image by author</figcaption></figure><h4>The Decoder in Training Phase: Teacher Forcing</h4><p>The working of the decoder is different during the training and testing phase, unlike the encoder part. Hence we will see both separately.</p><p>To train our decoder model, we use a technique called “<strong>Teacher Forcing” </strong>in which we feed the <strong>true</strong> output/token (and <strong>not the predicted</strong> output/token) from the previous time-step as input to the current time-step.</p><p>To explain, let’s look at the 1st iteration of training. Here, we have fed our input-sequence to the encoder, which processes it and passes its final internal states to the decoder. Now for the decoder part, refer to the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3gCQBomC_AolBeF1klw_ig.png" /><figcaption>Image by author</figcaption></figure><p>Before moving on, note that in the decoder, at any time-step <strong>t</strong>, the output <strong>yt_pred</strong> is the probability distribution over the entire vocabulary in the output dataset which is generated by using the Softmax activation function. The token with the maximum probability is chosen to be the predicted word.</p><p>For eg. referring to the above diagram, y1_pred = [0.02 0.12 0.36 0.1 0.3 0.1] tells us that our model thinks that the probability of 1st token in the output-sequence being ‘&lt;START&gt;’ is 0.02, ‘ravi’ is 0.12, ‘de’ is 0.36 and so on. We take the predicted word to be the one with the highest probability. Hence here the predicted word/token is <strong>‘de’ </strong>with a probability of 0.36</p><p>Moving on...</p><p><strong>At time-step 1</strong></p><p>The vector [1 0 0 0 0 0] for the word ‘&lt;START&gt;’ is fed as the input vector. Now here I want my model to predict the output as y1_true=[0 1 0 0 0 0] but since my model has just started training, it will output something random. Let the predicted value at time-step 1 be y1_pred=[0.02 0.12 0.36 0.1 0.3 0.1] meaning it predicts the 1st token to be <strong>‘de’</strong>. Now, should we use this y1_pred as the input at time-step 2?. We can do that, but in practice, it was seen that this leads to problems like slow convergence, model instability, and poor skill which is quite logical if you think.</p><p>Thus,<strong> teacher forcing</strong> was introduced to rectify this. in which we feed the true output/token (and not the predicted output) from the previous time-step as input to the current time-step. That means the input to the time-step 2 will be y1_true=[0 1 0 0 0 0] and not y1_pred.</p><p>Now the output at time-step 2 will be some random vector y2_pred. But at time-step 3 we will be using input as y2_true=[0 0 1 0 0 0] and not y2_pred. Similarly at each time-step, we will use the true output from the previous time-step.</p><p>Finally, the loss is calculated on the predicted outputs from each time-step and the errors are backpropagated through time to update the parameters of the model. The loss function used is the categorical cross-entropy loss function between the target-sequence/<strong>Y_true </strong>and the predicted-sequence/<strong>Y_pred </strong>such that</p><ul><li>Y_true = [y0_true, y1_true, y2_true, y3_true, y4_true, y5_true]</li><li>Y_pred = [‘&lt;START&gt;’, y1_pred, y2_pred, y3_pred, y4_pred, y5_pred]</li></ul><p>The final states of the decoder are discarded</p><h4>The Decoder in Test Phase</h4><p>In a real-world application, we won&#39;t have Y_true but only X. Thus we can’t use what we did in the training phase as we don’t have the target-sequence/Y_true. Thus when we are testing our model, the predicted output (and not the true output unlike the training phase) from the previous time-step is fed as input to the current time-step. Rest is all same as the training phase.</p><p>So let’s say we have trained our model and now we test it on the single sentence we trained it on. <strong>Now If we trained the model well and that too only on a single sentence then it should perform almost perfectly but for the sake of explanation say our model is not trained well or partially trained and now we test it. Let the scenario be depicted by the diagram below</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QaJKYwjg06vzooJVGKOIEw.png" /><figcaption>Image by author</figcaption></figure><p><strong>At time-step 1</strong></p><p>y1_pred = [0 0.92 0.08 0 0 0] tells that the model is predicting the 1st token/word in the output-sequence to be ‘ravi’ with a probability of 0.92 and so now at the next time-step this predicted word/token will only be used as input.</p><p><strong>At time-step 2</strong></p><p>The predicted word/token <strong>“ravi”</strong> from 1st time-step is used as input here. Here the model predicts the next word/token in the output-sequence to be <strong>‘de’ </strong>with a probability of 0.98 which is then used as input at time-step 3</p><p>And the similar process is repeated at every time-step till the ‘&lt;END&gt;’ token is reached</p><p>Better visualization for the same would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*PdckBUuynt8wCVJ6f197Rw.png" /><figcaption>Image by author</figcaption></figure><p>So according to our trained model, the predicted-sequence at test time is “ravi de rencontrer rencontrer”. Hence though the model was incorrect on the 3rd prediction, we still fed it as input to the next time-step. The correctness of the model depends on the amount of data available and how well it has been trained. The model may predict a wrong output but nevertheless, the same output is only fed to the next time-step in the test phase.</p><h4>The Embedding Layer</h4><p>One little detail that I didn’t tell you before was that the input-sequence in both the decoder and the encoder is passed through an embedding layer to reduce the dimensions of the input word vectors because in practice, the one-hot-encoded vectors can be very large and the embedded vectors are a much better representation of words. For the encoder part, this can be shown below where the embedding layer reduces the dimensions of the word vectors from four to three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OWrsfe6vtTUU--cuYWSrtA.png" /><figcaption>Image by author</figcaption></figure><p><em>This embedding layer can be pre-trained like Word2Vec embeddings or can be trained with the model itself.</em></p><h4>The Final Visualization at test time</h4><p>Somehow tried to capture everything in one diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B5pqh5hTgTaNiuUQhtxBQA.png" /><figcaption>Image by author</figcaption></figure><p><em>You might find some slight differences in implementations here and there but the main idea remains the same.</em></p><h3>The “Sutskever Model”</h3><p>The paper <a href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural Networks<em> </em>by Ilya Sutskever, et al</a> was one of the pioneer papers introducing the encoder-decoder model for machine translation and in general Seq2Seq modelling.</p><p>Here, I will mention the important points from the paper and how their implementation was different from the toy example that we learned from.</p><ul><li>The model was applied to English to French translation</li><li>Each sentence was made to end with a special end-of-sentence symbol “&lt;EOS&gt;”</li></ul><blockquote>“Note that we require that each sentence ends with a special end-of-sentence symbol “&lt;EOS&gt;”, which enables the model to define a distribution over sequences of all possible lengths.” [1]</blockquote><p>In the paper, to explain, they took input-sequence as “A B C” and target-sequence as “W X Y Z”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*8oJefm50Ic3Tv7yxAYM9Sw.png" /><figcaption><a href="https://www.slideshare.net/quangntta/sequence-to-sequence-learning-with-neural-networks">https://www.slideshare.net/quangntta/sequence-to-sequence-learning-with-neural-networks</a></figcaption></figure><ul><li><strong>Dataset Details</strong></li></ul><blockquote>“We trained our models on a subset of 12M sentences consisting of 348M French words and 304M English words, which is a clean “selected” subset from [29]. We chose this translation task and this specific training set subset because of the public availability of a tokenized training and test set together with 1000-best lists from the baseline SMT [29]. As typical neural language models rely on a vector representation for each word, we used a fixed vocabulary for both languages. We used 160,000 of the most frequent words for the source language and 80,000 of the most frequent words for the target language. Every out-of-vocabulary word was replaced with a special “UNK” token.”[1]</blockquote><ul><li><strong>Input-sequences were reversed.</strong></li></ul><blockquote>“While the LSTM is capable of solving problems with long term dependencies, we discovered that the LSTM learns much better when the source sentences are reversed (the target sentences are not reversed)” [1]</blockquote><ul><li>A 1000-dimensional word embedding layer was used to represent the input words. Softmax was used on the output layer.</li><li>The input and output models had 4 layers with 1,000 units per layer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*eh20nFQQxwip4bkkkc_98g.png" /><figcaption><a href="https://www.slideshare.net/quangntta/sequence-to-sequence-learning-with-neural-networks">https://www.slideshare.net/quangntta/sequence-to-sequence-learning-with-neural-networks</a></figcaption></figure><ul><li>A BLEU score of 34.81 was achieved.</li></ul><blockquote>“… we obtained a BLEU score of 34.81 […] This is by far the best result achieved by direct translation with large neural networks. For comparison, the BLEU score of an SMT baseline on this dataset is 33.30” [1]</blockquote><p>Here is a talk on the same paper for your reference.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-uyXE7dY5H0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-uyXE7dY5H0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-uyXE7dY5H0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/50def6f5d5bf7c10fe7e0f00cb10c836/href">https://medium.com/media/50def6f5d5bf7c10fe7e0f00cb10c836/href</a></iframe><h3>Image Captioning Intuition</h3><p>The paper <a href="https://arxiv.org/abs/1412.2306">Deep Visual-Semantic Alignments for Generating Image Descriptions by Andrej Karpathy, Li Fei-Fei, et al</a><strong> </strong>used a different version of the encoder-decoder model for image captioning. Image Captioning is the process of generating a textual description of an image.</p><p>In the model proposed by them, the image is fed into a CNN network of your choice. Then the last Dense layer is fed into the LSTM network. This layer in some sense acts as a context vector as it captures the essence of the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*krQS1HdwYlu9IwrX_V6Jhw.png" /><figcaption><a href="https://arxiv.org/pdf/1412.2306.pdf">Deep Visual-Semantic Alignments for Generating Image Descriptions</a></figcaption></figure><h3>Further Reading</h3><p>If you’ve found your way here, then congratulations!! I hope you’ve found this useful. Since this is my first attempt at blogging, I hope that the readers will be forgiving and overlook any slight errors I may have made. I am mentioning more resources on the topic if you are looking to go deeper.</p><ol><li>I would suggest you to go through the paper we have discussed <a href="https://arxiv.org/abs/1409.3215"><strong>Sequence to Sequence Learning with Neural Networks by Ilya Sutskever, et al</strong></a><strong> </strong>by yourself once.</li><li>Nothing helps you understand a concept better than implementing it yourself. I found this amazing blog <a href="https://towardsdatascience.com/neural-machine-translation-using-seq2seq-with-keras-c23540453c74"><strong>Neural Machine Translation with Keras</strong></a><strong> </strong>where an Encoder-Decoder model is trained from scratch for English to French translation.</li><li>As mentioned before, there are a lot of advanced techniques which have been developed over encoder-decoder models for Seq2Seq modelling. For them I strongly suggest you to refer to the following in order</li></ol><ul><li>Attention Mechanism → <em>Jay Alammar, </em><a href="https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/"><strong>Visualizing A Neural Machine Translation Model</strong></a><strong> </strong>blog post, 2018.</li><li>Transformer → <em>Jay Alammar, </em><a href="https://jalammar.github.io/illustrated-transformer/"><strong>The Illustrated Transformer</strong></a> blog post<em>, </em>2018.</li><li>BERT → Jay Alammar, <a href="http://jalammar.github.io/illustrated-bert/"><strong>The Illustrated BERT</strong></a><strong> </strong>blog post, 2018.</li></ul><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/1409.3215">Sequence to Sequence Learning with Neural Networks<em> </em>by Ilya Sutskever, et al</a></p><p>[2] <a href="https://arxiv.org/abs/1412.2306">Deep Visual-Semantic Alignments for Generating Image Descriptions by Andrej Karpathy, Li Fei-Fei, et al</a></p><p>[3] <a href="https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9">https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9</a></p><p>[4] <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">https://colah.github.io/posts/2015-08-Understanding-LSTMs/</a></p><p>[5] <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">http://karpathy.github.io/2015/05/21/rnn-effectiveness/</a></p><p>[6] <a href="https://machinelearningmastery.com/encoder-decoder-recurrent-neural-network-models-neural-machine-translation/">https://machinelearningmastery.com/encoder-decoder-recurrent-neural-network-models-neural-machine-translation/</a></p><p>[7] <a href="https://www.slideshare.net/quangntta/sequence-to-sequence-learning-with-neural-networks">https://www.slideshare.net/quangntta/sequence-to-sequence-learning-with-neural-networks</a></p><p>The diagrams in the post have been made using <a href="https://creately.com/">creately</a></p><p><strong>PS</strong>: Feel free to provide comments/criticisms if you think they can improve this blog, I will definitely try to make the required changes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c34186fbf49b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/encoder-decoder-seq2seq-models-clearly-explained-c34186fbf49b">Encoder-Decoder Seq2Seq Models, Clearly Explained!!</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>