<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikita Sharma on Medium]]></title>
        <description><![CDATA[Stories by Nikita Sharma on Medium]]></description>
        <link>https://medium.com/@nikitasharma_43692?source=rss-2fa4208c113d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vEIrEpXU4y727YADsJSNaA.jpeg</url>
            <title>Stories by Nikita Sharma on Medium</title>
            <link>https://medium.com/@nikitasharma_43692?source=rss-2fa4208c113d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:52:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikitasharma_43692/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring Sign Language Recognition techniques with Machine Learning]]></title>
            <link>https://heartbeat.comet.ml/exploring-sign-language-recognition-techniques-with-machine-learning-d564262d87d3?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d564262d87d3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[sign-language]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Thu, 14 Jan 2021 14:01:06 GMT</pubDate>
            <atom:updated>2021-10-06T00:00:41.106Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding Indian Sign Language recognition and its techniques with a focus on the state-of-the-art hierarchical neural network approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*xKRUNRaSKlukqBCouhAbKA.jpeg" /></figure><p>In this post, we’re going to investigate the field of sign language recognition and its applications in the real world. We are going to discuss the approaches adopted by a <a href="https://link.springer.com/article/10.1007/s00521-020-05448-8">research paper</a> on Indian Sign Language Recognition and try to understand the merits and demerits of these methods from a practical point of view.</p><p>So, let’s jump right in!</p><p>Kindly refer to the link given below for the complete research paper. The models in this post will be based on the approach used in this paper. I am thrilled to let my readers know that the research paper here discussed was written by me and my teammates and has been published in the International Journal of Neural Computing and Applications.😁</p><p><a href="https://link.springer.com/article/10.1007/s00521-020-05448-8">Research Paper Link</a></p><p><strong>Note: </strong>Though the research paper focuses on Indian Sign Language recognition, it uses a generic approach that can be applied to other sign languages as well.</p><h3>What is Sign Language Recognition?</h3><p>Sign language is the mode of communication which uses visual ways like expressions, hand gestures, and body movements to convey meaning. Sign language is extremely helpful for people who face difficulty with hearing or speaking. Sign language recognition refers to the conversion of these gestures into words or alphabets of existing formally spoken languages. Thus, conversion of sign language into words by an algorithm or a model can help bridge the gap between people with hearing or speaking impairment and the rest of the world.</p><h4>How is Indian Sign Language different?</h4><p>Indian Sign Language (ISL), unlike other sign languages, is a two hand gesture sign language, though some of the letters of the alphabet (c,i,j,l,o,u,v) only require one hand representation.</p><h4>Issue with Indian Sign Language Recognition</h4><p>The problem with ISL recognition is that, due to the involvement of both hands, there is feature occlusion and computer vision techniques like <a href="https://en.wikipedia.org/wiki/Convex_hull_algorithms">convex hull</a> fail to capture the gesture effectively.</p><p>For example, let us imagine an algorithm that detects the location of the centre of palm and fingers to detect a gesture. Now, in the case of a gesture involving two hands, the algorithm would get confused since there would be two palms and 10 fingers and thus, result in a wrong prediction.</p><h3>Techniques for Sign Language Recognition</h3><p>Various techniques have been explored for the purpose of efficient sign language recognition over time. Some of the most effective ones are discussed below:</p><h4>Hardware based recognition techniques</h4><p>Some sign language recognition techniques consist of two steps: detection of hand gesture in the image and classification into a respective alphabet. Several techniques involve hand tracking devices (Leap Motion and Intel Real Sense) and use machine learning algorithms like SVM (Support Vector Machines) to classify the gestures. Hardware devices like kinetic sensors (by Microsoft) develops a 3D model of the hand and observes the hand movements and their orientations. A glove-based approach was another technique wherein the user was required to wear a special glove that recognized the position and orientation of the hand. Hardware techniques can be fairly accurate, but cannot be vastly used due to the initial setup cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5pDkblxcXavnqWVGd2qCfQ.jpeg" /></figure><h4>Machine learning based recognition techniques</h4><p>A few earlier techniques used computer vision techniques like the <a href="https://en.wikipedia.org/wiki/Convex_hull_algorithms">convex hull</a> method used to determine the convexities in the image and detect the edge of the hand in the image. There are also contour based techniques which look for skin-like contours in the image to detect a hand. The detection is later followed by machine learning algorithms trained for the task of classifying these gestures into alphabets.</p><p><em>The technique used in the research paper being discussed in this post focuses on Neural Network based recognition.</em></p><h3>Dataset</h3><p>The dataset was created manually due to the lack of availability of a complete dataset for ISL which included all the 26 letters of the Roman/Latin alphabet. The dataset includes almost 10,000 images for each letter in different backgrounds and lighting conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFZW6WwrZxOHOUKOxDeSHg.png" /></figure><h4>Data Augmentation</h4><p>The images for each letter were augmented to enhance the size and efficiency of the dataset. The images were zoomed in and out, brightened, darkened, rotated, and shifted so as to provide a set of images for each letter which were diverse with respect to the background, orientation, and lighting conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-7EvdvyEDttKFIEi64KHjQ.png" /></figure><h3>Approaches for Indian Sign Language Recognition</h3><p>Three approaches were developed for the problem statement at hand and a comparative analysis was performed on these models based on their training statistics and results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*G2SUgr-LUA0KA7LEC4Ns5g.png" /></figure><h4>Pre-trained and Fine-tuned models</h4><p>A pre-trained VGG16 model was employed for training on the dataset of ISL alphabets. The training was done using transfer learning and fine-tuning techniques, thereby establishing a baseline for comparing the models. The last layer (the softmax layer) of the initial VGG16 model was removed since it was designed as per the Imagenet dataset. Two fully-connected layers were then appended followed by a new softmax layer having 26 classes, one for each letter of the alphabet.</p><h4>Natural Language based model</h4><p>As a part of this model, a Deep Convolutional Neural Network (DCNN) with 26 classes was developed. Later, the output was fed to an English Corpora-based model for eradicating any errors during classification due to noise or model error. This process was based on the probability of the occurrence of the particular word in the English vocabulary. For example: if a person makes a gesture for “C-A-T” and it gets classified as “C-X-T”, then the model can convert this X into A based on the fact that CAT is a much more frequent word and there isnt’ any word like CXT. Only the top three accuracy scores provided by the neural network were considered in this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THawk_oFUcNg4otETZkONQ.png" /></figure><h3>Hierarchical Neural Network Model</h3><p>To eradicate the feature occlusion issue due to the presence of both one-handed and two-handed gestures, a segmented approach was adopted. An SVM model was used to classify the images into two categories: images having one hand or two hands, thus performing binary classification. After that, these images were fed into their respective neural networks designed for one and two hand features separately.</p><h4>Model Architecture</h4><ol><li><strong>Extracting HOG Feature matrix: </strong>Binary classification is performed by an SVM model to classify into images that contain one-handed or two-handed gestures. Before feeding the images into the SVM, we applied Histogram of Oriented Gradients (HOG) on these images which gave us a feature representation of these images. HOG is a feature descriptor and extractor which works on gradient-based edge detection and can be very helpful to detect hand features in noisy and dark backgrounds. The gradients along x and y directions of an image can be very important around edges and corners and helps us to identify sudden intensity changes. To learn about HOG in depth, please refer to <a href="https://heartbeat.comet.ml/introduction-to-basic-object-detection-algorithms-b77295a95a63">this</a> article.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*vY4vaciVNF_Ha3uGeRyGzQ.png" /><figcaption>HOG feature representation of images</figcaption></figure><p><strong>2. Data Pre-Processing for SVM:</strong> After extracting the HOG features, we got a 24339*2028 matrix which is quite huge to be fed to SVM. Thus, standardization and scaling was performed and then Principal Component Analysis (PCA) was done to get the most information out of this huge matrix and compact it into 1500 features for each image. To learn more about PCA, please refer to <a href="https://heartbeat.comet.ml/understanding-the-mathematics-behind-principal-component-analysis-efd7c9ff0bb3">this</a> article.</p><p><strong>3. SVM for Binary Classification: </strong>The principal component features were finally fed into the SVM model with a linear kernel.<strong> </strong>The output of the SVM depicted two classes, ‘0’ for one-hand and ‘1’ for two-hand gestures. To learn more about SVM, kindly go through <a href="https://heartbeat.comet.ml/understanding-the-mathematics-behind-support-vector-machines-5e20243d64d5">this</a> article. The model produced an accuracy of 96.79%</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*WhXQATfFkPBkSqrQ-bbHig.png" /><figcaption>Confusion matrix for SVM</figcaption></figure><p><strong>4. Neural Network for one hand features: </strong>If the output of the SVM turns out to be “0,” it means it is a one hand feature and is thus fed to a convolutional neural network based on VGG19 architecture. At last, two fully connected layers with 128 nodes each were appended followed by a softmax having 7 classes, one for each one-handed gesture (C,I,J,L,O,U,V).</p><p><strong>5. Neural Network for two hand features: </strong>If the output of the SVM turns out to be “1,” it means it is a two hand feature and is thus again fed to a different convolutional neural network based on VGG19 architecture. At the end of the VGG19 architecture, two fully connected layers with 256 nodes each were appended which were followed by a softmax layer of 19 classes, one for each two-handed gesture (A,B,D,E,F,G,H,K,M,N,P,Q,R,S,T,W,X,Y,Z).</p><h4>Training Parameters</h4><p>The Hierarchical Model was trained on 150,000 images, approximately. The SVM model was trained on PCA features extracted from 6085 images. For each of the neural networks following the SVM, Adam optimizer was used with a learning rate of 0.0001 in the case of one hand, and 0.0005 in case of two hand gestures.</p><h3>Results</h3><p>The results of all the three models are summarized in the next subsection. However, we are interested in the Hierarchical Network approach, which has proven to be a significant improvement over the existing architectures for Indian Sign Language recognition. The hierarchical model provided a training loss of 0.0016, thus translating to a training accuracy of 99% and a validation accuracy of 98.52% for categorizing one-handed features. For two-handed features, the training and validation accuracies were 99 and 97%, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*uv2U6KICIwzd1jPZ-I7x0A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*ZZnOdPvNxxTEHClZ1HpQAA.png" /></figure><h4>Comparative Analysis</h4><p>Here is a tabular comparison between the three models discussed in the research paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*-QJWqVrfXzCNXMlmewGsNw.png" /><figcaption>Accuracies for a few alphabets with different approaches</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*qcaeQVcjQFlbZvSgr8-tdQ.png" /><figcaption>Comparative analysis for the three approaches</figcaption></figure><h3>Conclusion</h3><p>In this post, we discussed Sign Language Recognition techniques and focused on a Hierarchical Neural Network based approach. Sign language recognition can help bridge the gap between people with hearing or speech impairment or who use sign language, and the rest of society who haven’t learned sign language. This leads to enhanced communication for all.</p><p>All feedback is welcome and appreciated — I’d love to hear what you think of this article!</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d564262d87d3" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/exploring-sign-language-recognition-techniques-with-machine-learning-d564262d87d3">Exploring Sign Language Recognition techniques with Machine Learning</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Graph Neural Networks]]></title>
            <link>https://heartbeat.comet.ml/introduction-to-graph-neural-networks-c5a9f4aa9e99?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5a9f4aa9e99</guid>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[graph-neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Wed, 28 Oct 2020 13:37:16 GMT</pubDate>
            <atom:updated>2021-09-24T13:50:05.544Z</atom:updated>
            <content:encoded><![CDATA[<h4>Graph neural networks — their need, real-world applications, and basic architecture with the NetworkX library</h4><figure><img alt="An image of a light fixture with multiple bulbs. All of the lights are connected with straight or bent lines to the plug." src="https://cdn-images-1.medium.com/max/1024/1*gkanLNHVltDw3AjHv54JWg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@bracht?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Fabio Bracht</a> on <a href="https://unsplash.com/s/photos/connection?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post, we are going to investigate a relatively newer field in deep learning which involves graphs — a very important and widely used data structure. This post encompasses the basics of graphs, the amalgamation of graphs and deep learning, and a basic idea about graph neural networks and their applications. We will also briefly discuss on how to build graphs with a Python library called NetworkX</p><p><em>So, let’s dive right in!</em></p><h3>What are graphs?</h3><p>In the world of computer science, graphs are a type of data structure having two components: Nodes (or vertices) and edges, which connect two nodes. Thus, a graph can be defined as a collection of loosely inter-connected nodes via edges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ASEWlSzqsGJdDwht4Pi0Q.jpeg" /><figcaption>Image Source: <a href="https://medium.com/data-structures-and-algorithms/graph-dd2b72c32f1f">https://medium.com/data-structures-and-algorithms/graph-dd2b72c32f1f</a></figcaption></figure><p>The<strong> </strong>nodes of a graph can be homogenous with all nodes having a similar structure, or heterogenous nodes having different types of structure. The edges define the relationship one node has with another. Edges can be bidirectional (from one node u to another v and vice versa), or unidirectional (from one node u to another node v). Edges can also be weighted — having a weight assigned to the edge that might depict the edge’s cost or importance.</p><p><em>An example:</em> Let us suppose a graph to be considered as a network of cities — the cities under observation being nodes and the roads connecting them being edges. Now, there can be various types of relevant problems that can be solved with graphs, such as finding out the shortest distance between cities (where roads can also be weighted as per the condition of the roads or traffic), or finding the cities which are well-connected to each other, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*ao-FSJHJ-cGU_yYCxTOA3A.png" /><figcaption>Image Source: <a href="https://www.raywenderlich.com/773-swift-algorithm-club-graphs-with-adjacency-list">https://www.raywenderlich.com/773-swift-algorithm-club-graphs-with-adjacency-list</a></figcaption></figure><h4>What are Graph Neural Networks (GNN)?</h4><p>Graphs have tremendous expressive powers and are therefore gaining a lot of attention in the field of machine learning. Every node has an embedding associated with it that defines the node in the data space. Graph neural networks refer to the neural network architectures that operate on a graph. The aim of a GNN is for each node in the graph to learn an embedding containing information about its neighborhood (nodes directly connected to the target node via edges). This embedding can then be used for different problems like node labelling, node prediction, edge prediction, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7VzFWjVoAdqrDBRQWCPvnA.png" /><figcaption>Each node and its neighborhood</figcaption></figure><p>Thus, after having embeddings associated with each node, we can convert edges by adding feed forward neural network layers and combine graphs and neural networks.</p><h4>Need for Graph Neural Networks</h4><p>The need for graph neural networks arose from the fact that a lot of data available to us is in an unstructured format. Unstructured data is data that has not been processed or does not have a pre-defined format which makes it difficult to analyze. Examples of such data are audio, emails, and social media postings. To make sense of this data and to derive inferences from it, we need a structure that defines a relationship between these unstructured data points. The existing machine learning architectures and algorithms do not seem to perform well with these kinds of data. The primary advantages of graph neural networks are:</p><ol><li>The graph data structure has proven tremendously successful in the field of computer science while working with unstructured data.</li><li>Graphs are helpful in defining concepts which are abstract, like relationships between entities. Since each node in the graph is defined by its connections and neighbors, graph neural networks can capture the relationships between nodes in an efficient manner.</li></ol><p>Thus, developing GNNs for handling data like social network data, which is highly unstructured, is an exciting amalgamation of graphs and machine learning which holds a lot of potential.</p><h4>Real-Life Applications of Graph Neural Network</h4><p>Being introduced recently in 2018, the GNNs still have a lot of real-life applications because their architecture resonates with the irregularity in data collected from various sources. Currently, GNNs have been the hot topic for:</p><p><strong>Social Network Analysis</strong> — Similar posts prediction, tags prediction, and recommending content to users.</p><p><strong>Natural Sciences </strong>— GNNs have also gained popularity in dealing with molecular interactions like protein-protein interactions.</p><p><strong>Recommender Systems </strong>— A heterogenous graph can be used to capture relationships between users and items to recommend relevant items to a buyer.</p><blockquote>Deep learning — For experts, by experts. <a href="https://www.deeplearningweekly.com/?utm_campaign=dlweekly-newsletter-expertise4&amp;utm_source=heartbeat">We’re using our decades of experience to deliver the best deep learning resources to your inbox each week</a>.</blockquote><h3>How do Graph Neural Networks work?</h3><p>After we have the basic structure of the graph neural network (nodes with their embeddings and edges with feed forward layers), we can move forward to understanding how GNNs actually work.</p><p>The basic idea is to learn neighborhood embeddings by aggregating information from a node’s neighbors via edges using neural networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPaI-f9FdVIQCmui2ljn2g.png" /><figcaption>Image source: <a href="http://snap.stanford.edu/proj/embeddings-www/files/nrltutorial-part2-gnns.pdf">http://snap.stanford.edu/proj/embeddings-www/files/nrltutorial-part2-gnns.pdf</a></figcaption></figure><h4>Neighborhood Aggregation or Message Passing</h4><p>Message passing refers to passing and receiving information between nodes about its neighborhood. Consider a target node having its initial embeddings: It receives information from its neighbors passed via edge neural networks. Data from these edges are aggregated (many techniques are used, like max pooling, averaging, etc.,) and passed to the activation unit of a node to get a new set of embeddings for the node. Every node in the initial setup has features x_v. The embeddings for each node after message passing can be defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*XI7j33ktxcDbdqlHiUSBoQ.png" /><figcaption>taken from the research paper: <a href="https://arxiv.org/pdf/1812.08434.pdf">https://arxiv.org/pdf/1812.08434.pdf</a></figcaption></figure><p>Where <em>x_ne[v]</em> denotes the features of the neighbors of <em>v, x_co[v]</em> is the edge features connected to <em>v, h_ne[v] </em>is the embedding of the neighbors of v.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ysh3sKgHjKAsPWaoX_TA7g.jpeg" /><figcaption>Image Source: <a href="http://snap.stanford.edu/proj/embeddings-www/files/nrltutorial-part2-gnns.pdf">http://snap.stanford.edu/proj/embeddings-www/files/nrltutorial-part2-gnns.pdf</a></figcaption></figure><p>In the above figure, hₐ⁽¹⁾ is the initial embedding of the node, hNₐ⁽¹⁾ is the aggregated embeddings of its neighbors. Combining these and passing to the node’s activation unit or filter will provide the new embedding for node A, which will also contain information about its neighbors. In this manner, each node gets a new set of embeddings for itself which determines its position in the graph. With various iterations or K layers of message passing, a node learns more and more about its neighborhood and its distant neighbors as well.</p><p>Eventually, each node has a rough idea about the complete graph (or a part of it, depending on the number of iterations and node-node distance/path or layers considered).</p><p><em>An example:</em> Consider a graph with social media posts as nodes. Now, if these nodes have embeddings and are labelled with tags like romance, science, comedy, etc., we get a new post and we need to provide it with a tag. Using the existing network and embeddings, neighborhood aggregation will help us predict the labels and embeddings for the unseen node.</p><p><strong>Advantage:</strong> Rather than running the entire algorithm again, we can just use embeddings of the neighbors to determine the locality of the new post. Therefore, GNN-based recommendation can be more efficient and scalable than other traditional machine learning recommendation algorithms out there for dealing large datasets.</p><h3>Types of Graph Neural Networks</h3><p>Graphs are used with various existing neural network architectures to yield promising results for various machine learning problems. The two most dominant networks are discussed briefly below.</p><h4>Graph Convolutional Networks (GCNs)</h4><p>Convolutional neural networks(CNNs) have been vastly used for image classification and segmentation problems. Convolutional operation refers to applying a spatial filter to the input image and getting a feature map as a result.</p><p><em>You can read more about CNNs </em><a href="https://en.wikipedia.org/wiki/Convolutional_neural_network"><em>here</em></a><em>.</em></p><p>GCNs refer to applying a spatially moving filter over the nodes of the graph which contains embeddings or data relevant to each node to get a feature representation of each node. Stacking a number of convolutional layers like a regular CNN can also be done to incorporate information from larger neighborhoods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gzid1GA297zfpZm1IWIDOQ.png" /><figcaption>Image source: <a href="https://www.experoinc.com/post/node-classification-by-graph-convolutional-network">https://www.experoinc.com/post/node-classification-by-graph-convolutional-network</a></figcaption></figure><h4>Graph Auto-Encoder Networks</h4><p>Auto-encoders are neural networks which consist of two networks combined via a bottleneck layer: An encoder, which downsamples the input by passing it through convolutional filters to provide the compact feature representation of the image, and a decoder<strong> </strong>which takes the representation provided by the encoder as input and tries to re-construct the input according to the same.</p><p>You can read more about auto-encoders <a href="https://en.wikipedia.org/wiki/Autoencoder">here</a>.</p><p>Graph auto-encoders try to learn a compact representation of the graph and then re-construct the graph using the decoder. They can be used to learn graph embeddings and hence can be used for predicting embeddings for un-seen nodes and to classify newer nodes into existing categories within the graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*ywcpaQbVybK3BYm7gO_xAw.jpeg" /><figcaption>Image source: <a href="https://www.frontiersin.org/articles/10.3389/fdata.2019.00002/full">https://www.frontiersin.org/articles/10.3389/fdata.2019.00002/full</a></figcaption></figure><p><em>Other kinds of graph neural networks like spatial and temporal graph neural networks, generative graph neural networks, recurrent graph neural networks, etc., have also been developed.</em></p><h3>Constructing Graphs with NetworkX</h3><p>NetworkX, as mentioned in its documentation, is a “Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.”<em> </em>Let’s learn how to build graphs with NetworkX:</p><p><strong>Install NetworkX:</strong></p><pre>pip install networkx</pre><p><strong>Creating a graph:</strong></p><pre><strong>import</strong> <strong>networkx</strong> <strong>as</strong> <strong>nx</strong><br>G = nx.Graph()<br>#defines an empty graph</pre><p><strong>Adding nodes to the graph:</strong></p><pre>#adds node 1<br>G.add_node(1)<br>#adds nodes from any iterable like list<br>G.add_nodes_from([2, 3])</pre><p><strong>Adding edges to the graph:</strong></p><pre>#adding an edge from first node to another<br>G.add_edge(1, 2)<br>#adding edges from a list<br>G.add_edges_from([(1, 2), (1, 3)])<br>#adding a weighted edge<br>G.add_edge(1, 2, weight=4.5)</pre><p>In this manner, we can construct a graph that we want to work on. Here is a simple example to find the shortest path between two nodes:</p><pre>import networkx as nx<br>G = nx.Graph()<br>G.add_edge(&#39;A&#39;, &#39;B&#39;, weight=5)<br>G.add_edge(&#39;B&#39;, &#39;D&#39;, weight=3)<br>G.add_edge(&#39;A&#39;, &#39;C&#39;, weight=4)<br>G.add_edge(&#39;C&#39;, &#39;D&#39;, weight=5)<br>nx.shortest_path(G, &#39;A&#39;, &#39;D&#39;, weight=&#39;weight&#39;)<br>[&#39;A&#39;, &#39;B&#39;, &#39;D&#39;]</pre><h3>Libraries that Support Graph Neural Networks</h3><p>For the purpose of creation and analysis of graph structure, the following Python libraries can be used:</p><ul><li><a href="https://networkx.github.io/">NetworkX - NetworkX documentation</a></li><li><a href="https://igraph.org/python/">python-igraph</a></li></ul><p>For the purpose of creating graph neural networks, the following libraries have been gaining a lot of attention and support:</p><ul><li><a href="https://github.com/dmlc/dgl">GitHub - dmlc/dgl: Python package built to ease deep learning on graph, on top of existing DL frameworks.</a></li><li><a href="https://github.com/deepmind/graph_nets">GitHub - google-deepmind/graph_nets: Build Graph Nets in Tensorflow</a></li></ul><h3>Conclusion</h3><p>In this post, we went through a journey of understanding graphs and graph neural networks. Graph neural networks are an intuitive solution to making sense of unstructured data and are useful for many real-world applications. We also learned how to build graphs from scratch with a library called NetworkX. Building an efficient graph is influential to the output of the network.</p><p>Hope you like this article and please let me know what you think.</p><p><strong><em>Until next time!</em></strong></p><p><em>Editor’s Note:</em><a href="http://heartbeat.fritz.ai/"><em> Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5a9f4aa9e99" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/introduction-to-graph-neural-networks-c5a9f4aa9e99">Introduction to Graph Neural Networks</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semantic Image Inpainting with Context Encoders]]></title>
            <link>https://heartbeat.comet.ml/semantic-image-inpainting-with-context-encoders-b1b1b01b90c7?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1b1b01b90c7</guid>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Tue, 20 Oct 2020 13:33:29 GMT</pubDate>
            <atom:updated>2021-10-05T20:20:54.078Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding Image Inpainting and its Techniques with a Focus on the State-of-the-Art Context-Encoders and a Generative Model-Based Approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1PYPaiWteblFe8Hp410RQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@cloudyaaa?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Klaudia Piaskowska</a> on <a href="https://unsplash.com/s/photos/paintings?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post, we are going to understand the underlying concepts of semantic image inpainting and various techniques developed for the same. However, we are going to focus on an effective and elegant state-of-the-art context encoder and generative model based approach for image inpainting.</p><p><em>Let’s get going!</em></p><p>The model and architecture discussed in this post is referred from the research paper link given below. Kindly refer to the research paper for the complete implementation:</p><p><a href="https://arxiv.org/pdf/1604.07379.pdf"><strong>Research Paper Link</strong></a></p><h3>What is Semantic Image Inpainting?</h3><p>Imagine having an old but precious photograph and finding out a few parts of it had corrupted. You want to cherish the moment captured by the photograph forever and recover the image. Image inpainting can save the day!</p><p>Given a corrupted image with parts of the image missing or distorted, semantic image inpainting refers to the filling up of these regions with the help of the available visual data. Semantic image inpainting, as its name suggests, also takes into account the context of the image and pixels surrounding the missing regions while filling these regions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*HX77gwpVxo-RAr-s_1egaA.jpeg" /><figcaption>An example of Image Inpainting for Reviving Historical Paintings</figcaption></figure><p><strong><em>A few real world applications of semantic image inpainting:</em></strong></p><ol><li><strong>Restoring History: </strong>Semantic image inpainting can be helpful to restore historical artifacts and give them new life. It can also be helpful in recovering ancient hieroglyphics, books, and images of high heritage value without any human effort or involvement of a subject-matter expert.</li><li><strong>Reviving Deteriorating Art: </strong>Semantic image inpainting can prove to be a great tool for museums to revive their art. Restoring art requires great expertise and time if performed manually by artists.</li></ol><h3>Techniques Used for Semantic Image Inpainting</h3><p>Various techniques have emerged over time to fulfill the need of reviving distorted images and providing satisfactory results close to the ground truth.</p><p>A few of them are discussed below:</p><h4><strong>Computer-Vision Based Techniques</strong></h4><p>Most of the computer vision techniques developed earlier were based on scene completion. The model had a huge database of millions of images. When encountering a corrupt image, it matched the rest of the images with the dataset, and upon finding a high similarity, it picked up a patch from the image in the dataset and fit it in the corrupted region. This technique only works for small regions.</p><h4><strong>Pixel-Based Neighborhood Techniques</strong></h4><p>Given an image with a few regions missing, pixel-based techniques at their core apply KNN (K-nearest neighbors) to fill these corrupted pixels with the pixel having the highest vote. A point to be noted here is that only pixels close to the corrupted region are given a vote, since it is highly likely that the pixels close to the region are more similar in context than the entire image. These non-semantic techniques, however, do not work when an entire scene or object is missing.</p><h4><strong>Context-Based Machine Learning Techniques</strong></h4><p>Context-based techniques form the base of the technique discussed in the following sections of this post. While trying to perform semantic image inpainting, let us think about how our brain might perceive the same task. When we look at a picture and its distorted regions, we take into account the context of the image and the visual information surrounding the missing region and try to recreate it in our brain. Something similar is the idea of semantic image inpainting that we will look at here.</p><p>For reconstructing an image, we need two types of information:</p><ol><li><strong>Perceptual Information: </strong>Understanding the content of the image.</li><li><strong>Contextual Information:</strong> Identifying and relating contents in the image as a complete picture.</li></ol><p>Below we are going to focus on a technique which involves a combination of context-based techniques and auto-encoder networks.</p><h3>Auto-Encoders</h3><p>Auto-encoders are neural networks which take an input image and try to learn its feature representation — they try to retain all possible information about an image by passing it through a low-dimensional “bottleneck” layer so as to gain a compact representation of the image and the scene presented in the image. These networks are made up of:</p><p><strong>Encoder:</strong> Which downsamples the image by passing it through convolutional filters and has a bottleneck layer at last which provides the compact feature representation of the image.</p><p><strong>Decoder: </strong>Which takes the representation provided by the encoder as input and tries to reconstruct the image according to the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*iiRWMjjQm8N8LAnOq0tZTg.png" /><figcaption>Image Source: <a href="https://medium.com/@birla.deepak26/autoencoders-76bb49ae6a8f">https://medium.com/@birla.deepak26/autoencoders-76bb49ae6a8f</a></figcaption></figure><h4>Context Encoders</h4><p>The model used in the above research paper is known as context encoders since the network consists of two parts: An encoder for capturing the context of an image into feature representation and a decoder which uses the representation provided by the encoder to fill up the missing content.</p><h4>Issue with Conventional Auto-Encoders</h4><p>Conventional auto-encoders are not fit for the problem of image inpainting because they essentially just compress and decompress the image without learning any semantic information about the image. The network is not conditioned to gain an understanding about the context of the image.</p><h3>Dataset</h3><p>The images for training the context encoder network are taken from the ImageNet dataset and Paris Street View Dataset. Images are masked with the dropped out or corrupt region set to zero. The masks can be of arbitrary shape and size, but three different strategies were used:</p><p><strong>Central Region:</strong> Removing 1/4th of the image part from the centre.</p><p><strong>Random Block:</strong> Overlapping blocks at arbitrary positions covering 1/4th of the image removed.</p><p><strong>Random Region:</strong> Removing arbitrary shapes, covering 1/4th of the image. This approach seemed to work well with the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yLae2pDGNvNG5G2QvpjrLA.png" /></figure><h3>Network Architecture</h3><h4>Encoder</h4><p>The encoder is inspired from the AlexNet architecture with five convolutional layers followed by max pooling. The encoder, upon being provided an image of 227*227, provides a 6*6*256 dimensional feature representation. The weights of the model are randomly initialized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsJ_H02EBzfCqQTUXvMKiQ.png" /></figure><p><em>Note: For the purpose of using an adversarial loss, a discriminator network is also used. To learn more about generative models, refer to </em><a href="https://developers.google.com/machine-learning/gan"><em>this</em></a><em> series.</em></p><h4>Channel-wise fully-connected layer</h4><p>The propagation of information within the feature representation is handled by channel-wise fully-connected layer. The fully-connected layer with groups has all the nodes connected to each other. This layer joins the output of the encoder to the input of the decoder. If the input layer has m feature maps of size n×n, this layer will output m feature maps of dimension n×n followed by stride 1 convolution. This restriction of being fully-connected only in groups has lesser parameters (mn⁴) as compared to the conventional fully-connected layers (m²n⁴) and thus saves computation time.</p><p>This layer allows each unit of the decoder to perceive the complete image content.</p><h4>Decoder</h4><p>The output from the channel-wise fully-connected layer is fed to the decoder, which reconstructs the image and the missing content. The decoder has five up-convolution layers each followed by a ReLU activation function so as to reach the target image size. Up-convolution can be thought of as an image upsampled and convolved which results in an image with higher resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tI8JSEOk5wLvFIPv3e3OUA.png" /></figure><h3>Loss Function</h3><p>To understand how well/poorly our model is working, we monitor the value of loss functions for several iterations. Loss functions are essentially the difference between the predicted and expected values. They give us an idea of the accuracy of the model. To learn more about loss functions, refer to <a href="https://heartbeat.fritz.ai/exploring-activation-and-loss-functions-in-machine-learning-39d5cb3ba1fc">this</a> article.</p><p>Designing the loss function for semantic image inpainting is a tricky task. There are two factors around which the loss function should be constructed:</p><ol><li>The overall image should be as realistic as possible after filling the missing region.</li><li>The context and the overall structure of the missing content must be captured.</li></ol><p>Thus, there is a need for a loss function which is a combination of two losses, known as the joint loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*qihk9_PecSXysJYB9LFWjA.png" /></figure><h4>Re-Construction Loss</h4><p>The re-construction loss is essentially a masked and normalized L2 loss that is the sum of the squared differences in predicted and expected values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6QjaLqAM5VWZkPkldVcHag.png" /></figure><p>Here, x is the image, F is the context encoder, and M is a binary mask depicting 0 for corrupt regions and 1 for the rest of the image. The re-construction loss is able to identify a rough outline of the missing object but does not give any fine details. Reason being, L2 works on pixel-based difference in values, and minimizing the value of the loss function (in this case, L2 distance) will lead to overly smooth textures which are undesirable. Thus, there is a need to address this problem by using another loss function in addition to the re-construction loss.</p><h4>Adversarial Loss</h4><p>This is derived from generative adversarial networks (GANs). The learning procedure takes two networks — an adversarial discriminator D that is provided with real images and samples generated by another network called Generator G, and tries to distinguish between the real and fake ones, while G tries to fool the discriminator by generating images that appear realistic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yyjl_tgHRLNGxUM-b48udw.jpeg" /></figure><p>For more information on GANs, refer to <a href="https://developers.google.com/machine-learning/gan">this</a> series.</p><p>For accommodating adversarial loss, the context encoder F is taken as G, and the loss function favors that not only the missing content but the entire output image looks realistic. Thus, adversarial loss keeps a check on the images generated by the decoder and helps favor the generation of sharper and realistic images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*FDLvfgEE1DLS5WAKeK8asA.png" /></figure><p>In the above formula, we try to increase the probability of the discriminator being right, so that the generator is compelled to perform better with time and generates realistic images to try and fool the discriminator.</p><h3>Training Parameters</h3><p>The model was developed in Caffe and PyTorch and was trained to minimize the value of loss function discussed in the previous section. Results are slightly better if the images are resized to 128×128. The network uses Adam for optimization. The co-efficient values for the combined loss function are λrec = 0.999 and λadv = 0.001. A learning rate of 1/10⁴ was used for the images with random region corruption. The network was run for almost 100k iterations and took 14 hours on a Titan X GPU.</p><p><strong><em>Note:</em></strong><em> For sharper inpainting, the pooling layers of the entire network were replaced with convolutions of the same kernel size and stride, thus resulting in a pool-free encoder.</em><strong><em> </em></strong><em>Thus, for re-construction based problems, pooling should be avoided.</em></p><h3>Results</h3><p>The context-encoder model produced significantly better and sharper results for filling up the missing regions with contextual information available to it. These results were satisfactorily better than other image inpainting models. A few glimpses of the results are shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFY4HJU1a9rLWFwGpfxXsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*pbppa4EH8YSzkH_Xs4vVQg.png" /></figure><h3>Conclusion</h3><p>In this post, we discussed semantic image inpainting as a real-world problem and discussed a state-of-the-art approach at length. Image inpainting is an interesting and widely popular problem and the context based auto-encoders seem to work satisfactorily for the same.</p><p>All feedback is welcome and appreciated — I’d love to hear what you think of this article.</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1b1b01b90c7" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/semantic-image-inpainting-with-context-encoders-b1b1b01b90c7">Semantic Image Inpainting with Context Encoders</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Optimizers in Machine Learning]]></title>
            <link>https://heartbeat.comet.ml/exploring-optimizers-in-machine-learning-7f18d94cd65b?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f18d94cd65b</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Tue, 29 Sep 2020 13:27:08 GMT</pubDate>
            <atom:updated>2021-10-05T20:44:18.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide to the widely used optimizer functions and a breakdown of their benefits and limitations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L0dz0dgFcJXRZ2gmef3qrw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mylestan?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Myles Tan</a> on <a href="https://unsplash.com/s/photos/swing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post we’re going to embark on a journey to explore and dive deep into the world of optimizers for machine learning models. We will also try and understand the foundational mathematics behind these functions and discuss their use cases, merits, and demerits.</p><p><em>So, what are we waiting for? Let’s get started!</em></p><h3>What is an Optimizer?</h3><p>Don’t we all love when neural networks work their magic and provide us with tremendous accuracy? Let’s get to the core of this magic trick by understanding how our networks find the most optimal parameters for our model.</p><p>We know that loss functions are used to understand how good/bad our model performs on the data provided to it. Loss functions are essentially the summation of the difference between the predicted and calculated values for given training samples. For training a neural network to minimize its losses so as to perform better, we need to tweak the weights and parameters associated with the model and the loss function. This is where optimizers play a crucial role.</p><p>Optimizers associate loss function and model parameters together by updating the model, i.e. the weights and biases of each node based on the output of the loss function.</p><h4>An example:</h4><p>Let us imagine a person hiking down the hill with no sense of direction. He doesn’t know the right way to reach the valley, but, he can understand whether he is moving closer (going downhill) or further away (uphill) from his destination. If he keeps taking steps in the correct direction, he will reach the valley.</p><p>This is the intuition behind optimizers — to reach a global minima with respect to loss function.</p><h3>Types of Optimizers</h3><p>Let’s discuss the most frequently used and appreciated optimizers in machine learning:</p><h3>Gradient Descent</h3><p>Well, of course we need to start off with the biggest star of our post — gradient descent. Gradient descent is an iterative optimization algorithm. It is dependent on the derivatives of the loss function for finding minima. Running the algorithm for numerous iterations and epochs helps to reach the global minima (or closest to it).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZzu43KoxDamVpWMVW0zfw.png" /><figcaption>Image Source: <a href="https://towardsdatascience.com/gradient-descent-algorithm-and-its-variants-10f652806a3">https://towardsdatascience.com/gradient-descent-algorithm-and-its-variants-10f652806a3</a></figcaption></figure><h4>Role of Gradient:</h4><p>Gradient refers to the slope of the equation in general. Gradients are partial derivatives and can be considered as the<strong> </strong>small change reflected in the loss function with respect to the small change in weights or parameters of the function. Now, this slight change can tell us what to do next to reduce the output of loss function — reduce this weight by 0.02 or increase this parameter by 0.005.</p><h4>Learning Rate:</h4><p>Learning rate is the size of the steps our algorithm takes to reach the global minima. Taking very large steps may skip the global minima, and the model will never reach the optimal value for loss function. On the other hand, taking very small steps will take forever to converge. Thus, size of the step is also dependent on the gradient value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*m1KQOLl-qB0mgRq_IWivnQ.png" /></figure><p>In the above formula, α is the learning rate, J is the cost function, and ϴ is the parameter to be updated. As you can see, partial derivative of J with respect to ϴj gives us the gradient. Note that, as we get closer to the global minima, the slope/gradient of the curve becomes less and less steep, which gives us a smaller value of derivative, which in turn reduces the step size automatically.</p><h4>Instances of Gradient Descent</h4><ul><li><strong>Vanilla/Batch Gradient Descent- </strong>gradient calculation<strong> </strong>with all training samples (slow and heavy computation)</li><li><strong>Stochastic Gradient Descent- </strong>gradient calculation with each training sample (unstable and may shoot even after getting global minima, but is memory efficient)</li><li><strong>Mini Batch Gradient Descent- </strong>dividing the training data into mini batches (32 , 64… usually in the powers of 2) and gradient calculation for the same (fast and memory efficient with lesser variance)</li></ul><p><strong>Advantages of Gradient Descent</strong></p><p>It is quite fast and is easy and intuitive to understand and compute.</p><p><strong>Disadvantages of Gradient Descent</strong></p><p>It may get stuck at a local minima and running gradient descent on a complete dataset takes too long if the dataset is very large (always use batches).</p><h3>Momentum Based Optimizers</h3><p>These optimizers help models converge by focusing on movement in the correct direction (downhill) and reduces the variance in every other insignificant direction. Thus, adding another parameter ‘Y’ to our optimization algorithm to move steeply in the relevant direction helps us move faster towards convergence. The momentum parameter ‘Y’ typically has a value around 0.9. But, at the same time, one needs to hit and try multiple values and choose manually for the most optimal results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EkA12En6SnXnt_Og-ACjAg.jpeg" /></figure><h3>Nesterov Accelerated Gradient (NAG)</h3><p>If the value of the momentum is too high, our model can even cross the minima and again start going up. Thus, just like a ball, if it has too much energy, it would not stop on the flat surface after rolling down, but will continue to move up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*zwRxp8muf36qu5lqi6ZDVA.gif" /></figure><p>But, if our algorithm knows when to slow down, it will reach the minima. Thus, in NAG algorithms, the gradient for time step t also takes into account the estimated gradient for time step t+1, which gives a better idea whether to slow down or not. It works better than the conventional momentum algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*x4wXCTmOGyHqTjnRRJyg9g.png" /></figure><p>In the above equations, (ϴ -ϒvt-1) is the next approximated values of the parameters and ∇J(ϴ -ϒvt-1) is the gradient with respect to the future values.</p><p><strong>Advantages of NAG</strong></p><p>It has an understanding of motion or future gradients. Thus, it can decrease momentum when the gradient value is lesser or the slope is low and can increase momentum when the slope is steep.</p><p><strong>Disadvantages of NAG</strong></p><p>NAG is not adaptive with respect to the parameter importance. Thus, all parameters are updated in a similar manner.</p><h3>AdaGrad — Adaptive Gradient Algorithm</h3><p>How cool would it be if our algorithm adapts different learning rates for different parameters? This is the intuition behind AdaGrad — some weights will have different learning rates based on how frequently or by how much their values are updated so as to perform optimally. Larger learning rate updates are made for less frequent parameters and smaller ones for the frequent parameters. The learning rate however, always decreases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*KqopEWrVIPfF44CYHPjZGQ.png" /></figure><p>Here, as you can see, AdaGrad uses the sum of the square of past gradients to calculate the learning rate for each parameter (∇L(ϴ) is the gradient or partial derivative of cost function with respect to ϴ).</p><p><strong>Advantages of AdaGrad</strong></p><p>Works great for datasets which have missing samples or are sparse in nature.</p><p><strong>Disadvantages of AdaGrad</strong></p><p>The learning might be very slow, since, according to the formula above, division by bigger numbers (sum of past gradients become bigger and bigger with time) means that the learning rate is decreasing over time — therefore the pace of learning would also decrease.</p><h3><strong>RMS-Prop — </strong>Root Mean Square Propagation</h3><p>RMS-Prop is almost similar to AdaGrad, with just a minute difference: It uses exponentially decaying average instead of the sum of gradients<em>. </em>Thus, RMS-Prop basically combines momentum with AdaGrad. Also, instead of using all the gradients for momentum, it takes into account only the most recent gradients for momentum calculation. This gives the model an understanding of whether it should increase or decrease its learning rate, given the current scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*vyuagFZ04ZvclsPTPRaJSQ.png" /></figure><p><strong>Advantages of RMS-Prop</strong></p><p>AdaGrad decreases the learning rate with each time step, but RMS-Prop can adapt to an increase or a decrease in the learning rate with each epoch.</p><p><strong>Disadvantages of RMS-Prop</strong></p><p>The learning can be pretty slow, same reason as in AdaGrad.</p><h3>Adam — Adaptive Moment Estimation</h3><p>Adam optimizer also uses adaptive learning rate technique to calculate current gradient from the first and second moments of the past gradients. Adam optimizer can be viewed as a combination of momentum and RMS-prop and is the most widely used optimizer for a wide range of problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*Ti-cvetTBXnTsM6rHUhmlg.png" /></figure><p>Rather than influencing the learning rate with respect to first moments as in RMSProp, Adam also uses the average of the second moments of the gradients. Also, Adam calculates an exponential moving average of the gradient and the squared gradient. Thus, Adam can be considered as a combination of AdaGrad and RMS-Prop.</p><p>In the above formula, α is the learning rate, β1(usually ~0.9) is the exponential decay rate with respect to the first moments, β2 is the exponential decay rate with respect to the second moments (usually ~0.999). ∈ is just a small value to avoid division by zero.</p><p><strong>Advantages of Adam</strong></p><p>Adam optimizer is well suited for large datasets and is computationally efficient.</p><p><strong>Disadvantages of Adam</strong></p><p>There are few disadvantages as the Adam optimizer tends to converge faster, but other algorithms like the Stochastic gradient descent focus on the datapoints and generalize in a better manner. Thus, the performance depends on the type of data being provided and the speed/generalization trade-off.</p><h3>Race to the Global Minima</h3><p>Here is an animation of how different optimizers help the model reach the global minima for a particular data space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m7-otgfbxiAvSipInHliXw.gif" /><figcaption>Image Source:<a href="https://github.com/ilguyi/optimizers.numpy">https://github.com/ilguyi/optimizers.numpy</a></figcaption></figure><h3>Conclusion</h3><p>In this post we discussed about various optimizers like gradient descent and its variations, Nesterov accelerated gradient, AdaGrad, RMS-Prop, and Adam along with their primary use cases. These are the most widely-used optimizer functions and are essential for developing efficient neural networks. Choosing an optimizer is influential to building efficient and fast neural networks.</p><p>I hope this article has helped you learn and understand more about these fundamental ML concepts. All feedback is welcome. Please help me improve!</p><p>Until next time!</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f18d94cd65b" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/exploring-optimizers-in-machine-learning-7f18d94cd65b">Exploring Optimizers in Machine Learning</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Image Super-Resolution Using Generative Adversarial Networks]]></title>
            <link>https://heartbeat.comet.ml/image-super-resolution-using-generative-adversarial-networks-86283cd29d28?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/86283cd29d28</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[super-resolution]]></category>
            <category><![CDATA[heartbeat]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Wed, 09 Sep 2020 14:57:58 GMT</pubDate>
            <atom:updated>2021-10-05T22:36:03.518Z</atom:updated>
            <content:encoded><![CDATA[<h3>Image Super-Resolution using Generative Adversarial Networks</h3><h4>Exploring image super-resolution techniques, with a focus on the state-of-the-art GAN-based approach</h4><p>In this post, we’re going to investigate the field of image super-resolution and its applications in real world. We’ll discuss a brilliant state-of-the-art model involving <a href="https://heartbeat.comet.ml/introduction-to-generative-adversarial-networks-gans-35ef44f21193">generative adversarial networks</a> (GANs) for this task and try to understand the underlying logic behind the approach.</p><p><strong><em>So let’s jump right in!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJ0H8KuGxy7D3WJ7VOyjdA.png" /></figure><p>Kindly refer to the link given below for the complete research paper. The model discussed in this post will be based on the approach used in this paper.</p><p><a href="https://arxiv.org/pdf/1609.04802.pdf"><strong>Research Paper Link</strong></a></p><h3>What Is Image Super-Resolution?</h3><p>As the name suggests, super-resolution refers to the art (yes, data science is an art, people!😉) of converting an image into higher resolution using its lower resolution image. It has various real-world applications like:</p><ol><li><strong>Storing images:</strong> For a data-intensive service, storing images in their high resolution format can be pretty expensive, since it takes up a lot of space. For example: Hospitals handle terabytes of data relevant to patients, injuries, MRIs etc.—thus, storing all of this data in high resolution is not economically feasible. With the help of image super-resolution techniques, one can convert the low resolution formats into higher resolution on a need-to-do basis, thus making storage more efficient.</li><li><strong>Reviving history: </strong>Ancient artifacts and historical images and documents can benefit from image super-resolution, as well. Such documents can be revived by converting them into their higher resolution counterpart, and thus preserved in a better quality.</li></ol><h3>Techniques Used for Image Super-Resolution</h3><h4><strong>Single Image Super-Resolution (SISR)</strong></h4><p>is one the most frequently used techniques to increase the resolution of images<strong>. </strong>Various techniques have been developed to enhance the resolution of images,<strong> </strong>Some of them are briefly discussed below:</p><ul><li><strong>Prediction-based methods: </strong>These<strong> </strong>were one of the initial model types for SISR. These algorithms work by interpolating pixel values in the image. Example of such filters are: linear, bicubic, or Lanczos. These filter-based approaches however, reduces contrast or sharp edges of an image and overly simplifies the output image. As a result, the image is too smooth and such results are undesirable in SISR. To learn more, refer to <a href="https://en.wikipedia.org/wiki/Image_scaling">this</a> article.</li><li><strong>Nearest neighbor methods: </strong>These techniques are widely used in computer vision. And in this case, the low resolution patches of similar types are upsampled, and thus the image is reconstructed based on the pixel values of the nearest pixels.</li></ul><h4>GAN-Based Approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HKfsgfm-1HLKnXKM7Zseiw.png" /></figure><p>GANs have proven to be a substantial improvement for generative modeling problems like <a href="https://heartbeat.comet.ml/guide-to-image-inpainting-using-machine-learning-to-edit-and-correct-defects-in-photos-3c1b0e13bbd0">image inpainting</a> and SISR. In generative adversarial networks, two networks train and compete against each other, resulting in mutual improvisation. The generator misleads the discriminator by <strong>creating compelling fake inputs </strong>and tries to fool the discriminator into thinking of these as real inputs<strong> </strong>. The discriminator tells us if an input is real or fake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*XiAXk60ur-NeZm2c49s7nA.png" /></figure><p>This optimization equation helps to increase the probability of the discriminator being able to differentiate between fake and real images and pushes the generator to create compelling real-like images to fool the discriminator more effectively.</p><blockquote><strong>For a detailed discussion on GANs, refer to </strong><a href="https://developers.google.com/machine-learning/gan"><strong>this</strong></a><strong> series. Also, visit my </strong><a href="https://heartbeat.comet.ml/my-mangagan-building-my-first-generative-adversarial-network-2ec1920257e3"><strong>post</strong></a><strong> for an interesting application of GANs.</strong></blockquote><p>The research paper we are going to discuss proposes a <strong>super-resolution generative adversarial network</strong> (SRGAN), which uses a deep residual network (ResNet) with skip-connections, and loss functions that are tweaked to address the problem at hand.</p><h4>Why adopt a GAN-based approach?</h4><p>Despite there being various other techniques for SISR and numerous advancement in different convolutional neural networks, a GAN-based approach seems to perform the best, given current research efforts.</p><p>The reason for this is that the other techniques mentioned above tend to perform operations on the image that result in either: an image being upsampled completely based on the pixel values of the other pixels in it’s vicinity; or based on averaged out squared errors or filters, which smoothen edges and details in an image.</p><p>The GAN-based approach on the other hand, depends on generative modeling and thus attempts to build images that are natural and closely resemble the real ones so as to fool the discriminator. In an attempt to do so, the sharpness of the edges and the contrast is preserved.</p><h4>Images</h4><p>For training purposes, the images below were downsampled by a factor of <strong>r </strong>after applying a Gaussian filter for a blur smoothing effect. In terms of width (W), height (H), and number of color channels (C),image tensors would be:</p><ul><li><strong>for low resolution I</strong>ᴸᴿ: (W*H*C)</li><li><strong>for high resolution I</strong>ᴴᴿ: (rW*rH*C)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*fYHXLY8lqi56hc52380cGw.png" /><figcaption>comparison between original and down-sampled images</figcaption></figure><h3>Network Architecture</h3><h4>Generator</h4><p>The generator network is created by appending B identical residual blocks, each consisting of 2 Convolutional layers with 3×3 kernels and 64 feature maps, followed by a batch normalization layer to reduce the effects of internal co-variate shifts. Residual blocks are used to better pass low-level information to the higher layers. A parametric ReLU is used as an activation function. To learn more about activation functions, refer to <a href="https://heartbeat.comet.ml/exploring-activation-and-loss-functions-in-machine-learning-39d5cb3ba1fc">this</a> article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ym2m1k3oCIcCPpNjMoG6eQ.png" /></figure><h4>Discriminator</h4><p>The discriminator network contains 8 convolutional layers of 3 × 3 filter kernels, increasing by a factor of 2 from 64 to 512 kernels. Strided convolution layers are used instead of max pooling layers.</p><p>Strided convolutions are preferred over pooling because they are a kind of convolution, and hence let the model learn new parameters and increase its expressiveness. Conversely, max pooling just selects the maximum value for a given group of pixels as output.</p><p>The resulting 512 feature maps are followed by 2 fully connected layers and a final sigmoid activation function for classifying real and fake images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yTuV0dp--eKMpER1nrW7Rg.png" /></figure><h3>Loss function</h3><p>To understand how well/poorly our model is working, we monitor the value of loss functions for several iterations. This helps us measure the accuracy of our model and understand how our model behaves for certain inputs. To learn more about loss functions, refer to <a href="https://heartbeat.comet.ml/exploring-activation-and-loss-functions-in-machine-learning-39d5cb3ba1fc">this</a> article.</p><h4>Why not MSE (Mean Sqaured Error)?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*mIiu7YrEgwEaMHAOCR8Dtg.png" /><figcaption>pixel wise MSE between original and generated image</figcaption></figure><p>MSE is usually the go-to loss function for various machine learning problems. The problem of SISR cannot use MSE as its loss function for monitoring results. The reason being, MSE works on pixel-based difference in values, and minimizing the value of the loss function (in this case, MSE) will lead to overly-smooth textures, which are not desirable.</p><p><strong>Thus the overall perceptual loss function has to be a linear combination of two losses:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*PbePjOP6hhXbwL2E2YTx2A.png" /></figure><h4>Content Loss</h4><p>Instead of relying on pixel-wise losses, a part of the loss function should be dedicated to perceptual similarity. This content loss is thus based on pixel-wise differences between the activation layers of a VGG-10 Model. The VGG loss is defined as the Euclidean distance between the feature representations of a reconstructed image Gθ (<strong>I</strong>ᴸᴿ) and the reference image <strong>I</strong>ᴴᴿ. The activations between different combinations of layers can be tried and tested for to optimize results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*yHwo2lNb1w18SyG00JcU1A.png" /></figure><p>φi,j indicates the feature map obtained by the j-th convolution (after activation) before the i-th maxpooling layer within the VGG-10 network.</p><h4>Adversarial Loss</h4><p>The adversarial loss serves as a check on the image generated by the generator. Since the conventional generator generates an image from random or Gaussian noise, for the output image to resemble the input image, the low resolution image <strong>I</strong>ᴸᴿ is passed to the generator. Adversarial loss favors the generation of natural-looking images. Thus, it’s defined by the discriminator probabilities over all training images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*xFKXyeHSnY1X5KKWWo-5AQ.png" /></figure><h3>Training Parameters</h3><p>The network was trained using a sample of 350,000 images from ImageNet. The low resolution images were obtained by downsampling the original images (BGR, C = 3) with a factor r = 4. The pixel values of <strong>I</strong>ᴸᴿ images were scaled to [0, 1] and pixel values of <strong>I</strong>ᴴᴿ images were scaled to [−1, 1]. The MSE loss was calculated on images of intensity range [−1, 1].</p><p>The learning rate was kept as 10⁻⁴ for 105 iterations, and then 10⁻⁵ for another 105 iterations. An Adam optimizer was employed with β1 = 0.9.</p><h3>Results</h3><p>The SRGAN model produced promising results, which were, visually-speaking, significantly better than any of the prior research done in the field of SISR. Some glimpses of the results are shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oe-jYp-_vCMI08tZkfHIHQ.png" /></figure><h3>Conclusion</h3><p>In this post, we discussed image super-resolution as a real-world problem and discussed a state-of-the-art approach at length. Image super-resolution continues to be one of the most intriguing problems in deep learning and computer vision. The GAN-based approach seems to work well for the same.</p><p>Let me know if you want me to write more posts discussing research papers for interesting machine learning problems.</p><p>All feedback is welcome and appreciated — I’d love to hear what you think of this article.</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86283cd29d28" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/image-super-resolution-using-generative-adversarial-networks-86283cd29d28">Image Super-Resolution Using Generative Adversarial Networks</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Activation and Loss Functions in Machine Learning]]></title>
            <link>https://heartbeat.comet.ml/exploring-activation-and-loss-functions-in-machine-learning-39d5cb3ba1fc?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/39d5cb3ba1fc</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Tue, 11 Aug 2020 13:07:09 GMT</pubDate>
            <atom:updated>2021-10-07T17:22:03.864Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide to the most frequently used activation and loss functions, and a breakdown of their benefits and limitations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dI78udLwgVIOc7Pna4YQXg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@kvtepov?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Vlad Kutepov</a> on <a href="https://unsplash.com/s/photos/turn-on?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post, we’re going to discuss the most widely-used activation and loss functions for machine learning models. We’ll take a brief look at the foundational mathematics of these functions and discuss their use cases, benefits, and limitations.</p><p><strong><em>Without further ado, let’s get started!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*F8SVj1DckQ70wix1oklS8A.png" /><figcaption>Source:<a href="https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/">https://missinglink.ai/guides/neural-network-concepts/7-types-neural-network-activation-functions-right/</a></figcaption></figure><h3>What is an Activation Function?</h3><p>To learn complex data patterns, the input data of each node in a neural network passes through a function the limits and defines that same node’s output value. In other words, it takes in the output signal from the previous node and converts it into a form interpretable by the next node. This is what an activation function allows us to do.</p><h4>Need for an Activation function</h4><ol><li><strong>Restricting value:</strong> The activation function keeps the values from the node restricted within a certain range, because they can become infinitesimally small or huge depending on the multiplication or other operations they go through in various layers (i.e. the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem">vanishing</a> and <a href="https://machinelearningmastery.com/exploding-gradients-in-neural-networks/">exploding</a> gradient problem).</li><li><strong>Add non-linearity: </strong>In the absence of an activation function, the operations done by various functions can be considered as stacked over one another, which ultimately means a linear combination of operations performed on the input.<strong> </strong>Thus, a <strong>neural network without an activation function is essentially a linear regression model</strong>.</li></ol><h3>Types of Activation functions</h3><p>Various types of activation functions are listed below:</p><h4>Sigmoid Activation function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*BwZmaMBaVW8f8lBAQUoytw.png" /></figure><p>The sigmoid function was traditionally used for binary classification problems (goes along the lines of “if x≤0.5, y=0 else y=1”). But, it tends to cause <strong>vanishing gradients problem</strong>, and if the values are too close to 0 or +1, the curve or gradient is almost flat and thus the learning would be too slow.</p><p>It’s also computationally expensive, since there are a lot of complex mathematical operations involved.</p><h4>Tanh Activation Function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*IcQ-P-cKAyt6N4vkwgCrdg.png" /></figure><p>The tanh function was also traditionally used for binary classification problems (goes along the lines of “if x≤0, y=0 else y=1”).</p><p>It’s different than sigmoid in the sense that it’s <strong>zero-centred</strong>, and thus restricts input values between -1 and +1. It’s even more computationally expensive than sigmoid since there are a lot of complex mathematical operations involved, which need to be performed for every input and iteration, repeatedly.</p><h4>ReLU Activation Function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*t2qFY1xGn2xpo1Ezw7ivWQ.png" /></figure><p>ReLU is a famous, widely-used non-linear activation function, which stands for Rectified Linear Unit (goes along the lines of “if x≤0, y=0 else y=1”).</p><p>Thus, it’s only activated when the values are positive. ReLU is <strong>less computationally expensive</strong> than tanh and sigmoid because it involves simpler mathematical operations.</p><p>But it faces what’s known as the “dying ReLU problem”—that is, when inputs approach zero, or are negative, the gradient of the function becomes zero, and thus the model <strong>learns slowly</strong>. ReLU is considered a go-to function if one is new to activation function or is unsure about which one to choose.</p><h4>Leaky-ReLU Activation function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*SBOLp0q8YT7uznb07zFTmA.jpeg" /></figure><p>The leaky ReLU function proposes a solution to the dying ReLU problem. It has a small positive slope in the negative plane, so it enables the model to learn, even for negative input values.</p><p>Leaky ReLUs are widely used with <a href="https://heartbeat.comet.ml/my-mangagan-building-my-first-generative-adversarial-network-2ec1920257e3">generative adversarial networks.</a> Parametric leaky ReLUs use a value “alpha”, which is usually around 0.1, to determine the slope of the function in the negative plane.</p><h4>Softmax Activation Function</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*_IDMoFnoJT916hhUREiFAQ.png" /><figcaption>Source:<a href="https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d">https://medium.com/data-science-bootcamp/understand-the-softmax-function-in-minutes-f3a59641e86d</a></figcaption></figure><p>The softmax function is a function that helps us represent inputs in terms of a discrete probability distribution. According to the formula, we need to apply an exponential function to each element of the output layer and normalize the values to ensure their summation is 1. The output class is the one with the highest confidence score.</p><p>This function is mostly used as the last layer in classification problems—especially multi-class classification problems—where the model ultimately outputs a probability for each of the available classes, and the most probable one is chosen as the answer.</p><h3>What is a Loss Function?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*en9e94qMlUpgjJcWzOED0A.jpeg" /><figcaption>Source: <a href="https://www.analyticsvidhya.com/blog/2019/08/detailed-guide-7-loss-functions-machine-learning-python-code/">https://www.analyticsvidhya.com/blog/2019/08/detailed-guide-7-loss-functions-machine-learning-python-code/</a></figcaption></figure><p>To understand how well/poorly our model is working, we monitor the value of loss functions for several iterations. It helps us measure the accuracy of our model and understand how our model behaves for certain inputs. Thus, <strong>it can be considered as the error or deviation in the prediction from the correct classes or values</strong>. The larger the value of the loss function, the further our model strays from making the correct prediction.</p><h4>Types of loss functions</h4><p>Depending on the type of learning task, loss functions can be broadly classified into 2 categories:</p><ol><li><strong>Regression loss functions</strong></li><li><strong>Classification loss functions</strong></li></ol><h3>Regression Loss Functions</h3><p>In this sub-section, we’ll discuss some of the more widely-used regression loss functions:</p><h4><strong>Mean Absolute Error or L1 Loss (MAE)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*jX730ljnUb9Acs_vpYI-yw.png" /></figure><p>The mean absolute error is the average of absolute differences between the values predicted by the model and the actual values. There’s an issue with MAE though—if some values are underestimated (negative value of error) and some are almost equally overestimated (positive value of error), they might cancel each other out, and we may get the wrong idea about the net error.</p><h4><strong>Mean Squared Error or L2 Loss(MSE)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*TCE9Kui4fbyZl5u3ARRBJw.png" /></figure><p>The mean squared error is the average of the squared differences between the values predicted by the model and the actual values. Squaring the error also helps us avoid the nullification issue faced by MAE.</p><p>MSE is also used to emphasize the error terms in cases where the input and output values have small scales. Thus, due to squaring the error terms, large errors have relatively greater influence when using MSE than smaller errors.</p><p>However, this can be a gamble when there are a lot of outliers in your data. Since the outliers would have greater weight due to higher error values being squared, it can make the error or loss function biased. Thus, outlier eradication should be performed before applying MSE.</p><h4>Huber loss</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PjIaI3ZswNWl_II5fB2gPg.png" /></figure><p>Huber loss is an absolute error, and as you can see from the formula above, it becomes quadratic as the error grows smaller and smaller. In the above formula, y is the expected value, xᵗ ϴ is the predicted value, and tₕ is a user-defined hyper-parameter.</p><h3>Classification Loss Functions</h3><p>In this sub-section, we’ll discuss some of the more widely-used loss functions for classification tasks:</p><h4>Cross-Entropy loss</h4><p>This loss is also called log loss. To understand cross-entropy loss, let’s first understand what entropy is. Entropy refers to the disorder or uncertainty in data. The larger the entropy value, the higher the level of disorder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*sKQ1ItgIBKsIcArMAOFksA.png" /></figure><p>As you can see in the above formula, the entropy is basically the negative summation of the product of the probability of occurrence of an event with its log over all possible outcomes. Thus, cross entropy as a loss function signifies reducing entropy or uncertainty for the class to be predicted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*lxV_trk6D5mLkbiq21qe9w.png" /></figure><p>Cross-entropy loss is therefore defined as the negative summation of the product of the expected class and the natural log of the predicted class over all possible classes. The negative sign is used because the positive log of numbers &lt; 1 returns negative values, which is confusing to work with while evaluating model performance.</p><p>For example, if the problem at hand is binary classification, the value of y can be 0 or 1. In such a case, the above loss formula reduces to:</p><blockquote>-(ylog(p)+(1−y)log(1−p))</blockquote><p>where p is the value of predicted probability that an observation O is of class C.</p><p>Thus, the loss function over the complete set of samples would be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*PDtIfRHpMfbbXbhj26I-OA.png" /></figure><h4>Hinge loss</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*elgfhmlvH027Rvm0OXedWg.jpeg" /></figure><p>Hinge loss helps in penalizing the wrongly-predicted values, as well as the values that were correctly predicted but with a lower probability score. Hinge loss is primarily used with Support Vector Machines (SVMs), since it supports the formation of a large-margin classifier by penalizing wrongly-predicted values, as well as the correctly-predicted ones with low probability.</p><h3>Conclusion</h3><p>In this post we discussed about various activation functions like sigmoid, tanh, ReLU, leaky-ReLU and softmax, along with their primary use cases. These are the most widely-used activation functions and are essential for developing efficient neural networks.</p><p>We also discussed a few major loss functions like mean squared error, mean absolute error, huber loss, cross-entropy loss, and hinge loss.</p><p>I hope this article has helped you learn and understand more about these fundamental ML concepts. All feedback is welcome. Please help me improve!</p><p>Until next time!😊</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39d5cb3ba1fc" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/exploring-activation-and-loss-functions-in-machine-learning-39d5cb3ba1fc">Exploring Activation and Loss Functions in Machine Learning</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guide to Ensemble Learning]]></title>
            <link>https://heartbeat.comet.ml/a-guide-to-ensemble-learning-390027fe38b8?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/390027fe38b8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Mon, 15 Jun 2020 14:03:04 GMT</pubDate>
            <atom:updated>2021-10-07T17:48:10.197Z</atom:updated>
            <content:encoded><![CDATA[<h4>Ensemble learning: Need, benefits, and detailed coverage of bagging, boosting, and stacking</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gd6zVHzVex_zhgVCNPoT9A.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@larisabirta?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Larisa Birta</a> on <a href="https://unsplash.com/s/photos/quartet?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post, we’ll have explore the fundamentals of ensemble learning. Specifically, we’ll discuss techniques like bagging, boosting, and stacking in detail and take a brief look at various foundational algorithms that adopt these techniques.</p><p><strong><em>Without further ado, let’s get started!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-DDBQ2fhyHHVIxgrLRiYA.png" /><figcaption>Source: <a href="https://analyticsindiamag.com/basics-of-ensemble-learning-in-classification-techniques-explained/">https://analyticsindiamag.com/basics-of-ensemble-learning-in-classification-techniques-explained/</a></figcaption></figure><h3>What is Ensemble Learning?</h3><p>This article presents techniques for improving classification accuracy by aggregating various classifiers in different ways. These techniques are known as the ensemble methods.</p><p>An <strong>ensemble method</strong> involves constructing a set or a group of base classifiers or individual learners (that may or may not follow the same algorithm) from training data. The final classification that we predict is, hence, the aggregation of all these classifiers’ outputs.</p><p>Error in any machine learning model can be described as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*DS4XJpIyShMoBRXVYJ5wuw.png" /></figure><p>Ensemble techniques are designed to help reduce these errors, and thus enhance the performance of the model.</p><h4>Necessary Conditions for Better Performance</h4><p>There are two necessary conditions for an ensemble classifier to perform better than a single classifier:</p><ol><li>The base classiﬁers should be independent of each other. If there’s a dependency, then these dependent classifiers will introduce a bias towards their favored class.</li><li>The base classiﬁers should individually perform better than random guessing. Any model that provides an accuracy of less than 50% is considered worse than random guessing. Thus, accumulating such classifiers will result in even worse performance.</li></ol><h3>Methods for Constructing an Ensemble Classifier</h3><ol><li><strong>By manipulating the training set:</strong> Various training sets are constructed by sampling data from the original dataset. Example: bagging and boosting.</li><li><strong>By manipulating the input features: </strong>In this approach, a subset of input features is chosen to form each training set. Example: random forest.</li><li><strong>By manipulating the learning algorithm:</strong> In this approach, several models are applied to the same training dataset. For example, an <strong>artiﬁcial neural network</strong> can produce diﬀerent models by changing its network topology or initial weights. Similarly, <strong>an ensemble of decision trees</strong> can be constructed by taking different nodes as root or splitting criteria.</li></ol><h3>Basic Ensemble Techniques</h3><p>In this section, we’re going to dive into the details of the three most commonly used ensemble techniques: Bagging, boosting, and stacking.</p><h3>Bagging</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JqSKm8ITz1Wxwn4QWCZNgw.png" /><figcaption>Source:<a href="https://www.mdpi.com/2076-3417/8/7/1198/htm">https://www.mdpi.com/2076-3417/8/7/1198/htm</a></figcaption></figure><p>To help us understand the concept of bagging, let’s use an analogy. Suppose that you’re a patient seeking a diagnosis for a medical condition or illness, given that you’ve recently noticed some symptoms. Instead of asking one doctor, you may choose to take the advice of several. If a certain result occurs more than any of the others, you may choose this as the ﬁnal or best diagnosis.</p><p>Thus, it can be inferred that the final outcome will be a combined majority vote, where each doctor<strong> gets an equal vote</strong>. The intuition for bagging is similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*rhJk9uSNwQ4uGbkigL6jHg.png" /></figure><p>Bagging, which is also known as bootstrap aggregating, is a technique that <strong>repeatedly samples data (with replacement) from a dataset</strong>.</p><p>Every individual dataset will have the same size as the original dataset, but the data points are chosen either randomly or following a sampling distribution. Because the sampling is done with replacement, a few data points may appear several times in the same training set, while others may never be a part of that training set.</p><p>Think of sampling with replacement as taking out balls of different colors like blue and red from a bag. Let’s assume there are 10 blue and 10 red balls. Initially, the probability of picking a blue or red ball is equal—that is, 10/20= 0.5.</p><p>If we take out a blue ball once, we put another blue ball in the bag so that total number of both blue and red balls will be same always, and thus, the probability, too. So one dataset might have 15 blue and 5 red, another might have 13 red and 7 blue, and so on.</p><p>Each classifier’s output is aggregated, and the class with the majority of votes is selected as the output class.</p><h4>Benefits of Bagging</h4><ol><li>The effect of outliers is minimized by applying bagging. Since every data point has an equal probability of being selected, bagging does not focus on any particular output class of the training data. It’s therefore less susceptible to overﬁtting in cases involving noisy data.</li><li>Bagging improves generalization error by reducing the variance of the base classiﬁers. The performance of bagging depends on the stability of the base classiﬁer. If a base classiﬁer is unstable, bagging helps to reduce the errors associated with random ﬂuctuations in the training data. For example, choosing a different splitting criteria in decision tree induction can result in entirely different trees.</li></ol><h4>Implementation in Scikit-learn</h4><p>Implementing bagging is fairly simple using the sklearn library. We just need to import the bagging aggregator and the base classifier that we prefer:</p><pre><strong>from</strong> <strong>sklearn.ensemble</strong> <strong>import</strong> BaggingClassifier<br><strong>from</strong> <strong>sklearn.neighbors</strong> <strong>import</strong> KNeighborsClassifier<br>bagging = BaggingClassifier(KNeighborsClassifier(),<br>                            max_samples=0.5, max_features=0.5)</pre><h3>Boosting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*DwvwMlOcT1T9hZwIJvMfng.png" /><figcaption>Source:<a href="https://towardsdatascience.com/understanding-adaboost-for-decision-tree-ff8f07d2851">https://towardsdatascience.com/understanding-adaboost-for-decision-tree-ff8f07d2851</a></figcaption></figure><p>Let’s understand the concept of boosting by tweaking the previous analogy a little. Suppose again that you’re a patient seeking a diagnosis for new symptoms. Instead of asking one doctor, you may choose to take the advice of several. Let’s say you weigh the advice of a few doctors more than the others, due to their expertise and experience.</p><p>The ﬁnal output will then be a combination of the weighted diagnoses. This is the essence behind boosting. Unlike bagging, every doctor doesn’t get an equal vote. They are all assigned weights.</p><p>Boosting is an iterative procedure that changes the training set according to the data points that are falsely classified. It tweaks the weights of the data points in such a way that the base classiﬁers will <strong>focus on examples that are hard to classify.</strong></p><p>In the above image, as you can see, the weights and parameters are updated in such a manner that the falsely predicted data points are given more importance or weight.</p><h4>AdaBoost Classifier</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*kbX0lhZy6YdTXoB7Q-MReQ.png" /></figure><p>Given above is the algorithm for AdaBoosting. We’re going to walk through it step-by-step. Let’s say we have a dataset whose data points are represented as{(xj, yj) | j =1 ,2,…,N}, where x is the set of features and y is the class label. In the AdaBoost algorithm, every classifier is given importance according to it’s error rate, given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*_CSzCoPkGJzGaQcM43SQrg.png" /></figure><p>where I(p) = 1 if the predicate p is true, and 0 otherwise. The importance of a classiﬁer Ci is given by the following parameter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*bEr3gnkWLf8Ac8LKfnZUkA.png" /></figure><p>The αi parameter is used to update the weights of the data points based on how well they were classified. The weight update mechanism for AdaBoost is given by the equation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*9N8n13rqEltCnXMKik-HTQ.png" /></figure><p>Thus, wi reduces for a correctly classified point and increases for a falsely predicted data point. In addition, if any intermediate rounds produce an error rate higher than 50%, the weights are reverted back to their original uniform values, wi =1/N.</p><h4>Implementation in Scikit-learn</h4><p>We can implement the AdaBoost algorithm in sklearn by using these few lines of code:</p><pre><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> cross_val_score<br><strong>from</strong> <strong>sklearn.datasets</strong> <strong>import</strong> load_iris<br><strong>from</strong> <strong>sklearn.ensemble</strong> <strong>import</strong> AdaBoostClassifier<br><br>X, y = load_iris(return_X_y=<strong>True</strong>)<br>clf = AdaBoostClassifier(n_estimators=100)<br>scores = cross_val_score(clf, X, y, cv=5)<br>scores.mean()</pre><h3>Stacking</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*TcGmOuMTARvshOY5bDN81Q.png" /><figcaption>Source:<a href="https://medium.com/@rrfd/boosting-bagging-and-stacking-ensemble-methods-with-sklearn-and-mlens-a455c0c982de">https://medium.com/@rrfd/boosting-bagging-and-stacking-ensemble-methods-with-sklearn-and-mlens-a455c0c982de</a></figcaption></figure><p>With stacking, a learner is used to combine the output from all the base classifiers. All of these are combined via a meta classifier or regressor. In stacking, a single model is used to learn how to best combine the predictions from the contributing models. These models need not be similar (like random forest), and each of them are trained on the complete dataset.</p><p>Thus, two things need to be pre-defined while making our stacking model:</p><ol><li>The k base classifiers that we fit our training data on.</li><li>The meta classifier to combine the output from these k base classifiers.</li></ol><p>For example, if we take 3 models, namely, k-nearest neighbors, an SVM, and a naive Bayes classifier, and we select a neural network as our meta classifier—the neural network will then take the outputs of these 3 models as input and will return final predictions based on their combination.</p><h4>Implementation in Scikit-learn</h4><p>Implementing stacking in sklearn is quite easy. You just have to enlist the models you want to accumulate together, along with their parameters, and then a meta-estimator to combine them all:</p><pre><strong>from</strong> <strong>sklearn.linear_model</strong> <strong>import</strong> RidgeCV, LassoCV<br><strong>from</strong> <strong>sklearn.svm</strong> <strong>import</strong> SVR<br>estimators = [(&#39;ridge&#39;, RidgeCV()),<br>              (&#39;lasso&#39;, LassoCV(random_state=42)),<br>              (&#39;svr&#39;, SVR(C=1, gamma=1e-6))]</pre><h3>Algorithms Based on Ensemble Techniques</h3><p>A few famous algorithms based on the ensemble techniques discussed above are listed here:</p><ol><li><strong>Random Forest:</strong> A random forest can be thought of as an ensemble of several decision trees, which are made using either a subset of the training data or a subset of the features. Then bagging is applied to aggregate and find the final output. Refer to <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html">this</a> article to learn more.</li><li><strong>AdaBoost: </strong>AdaBoost is a boosting-based technique that can be found in almost all popular machine learning libraries.<strong> </strong>Refer to <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html">this</a> article to learn more.</li><li><strong>XGBoost: </strong>XGBoost is based on a gradient boosting algorithm that works on an ensemble of decision trees.<strong> </strong>It’s tremendously successful for certain use cases and is very popular these days in the community. Refer to <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html">this</a> article to learn more.</li></ol><h3>Sources to Get Started with Ensemble Learning</h3><ul><li><a href="https://www.kaggle.com/flennerhag/ml-ensemble-scikit-learn-style-ensemble-learning">ML-Ensemble: Scikit-learn style ensemble learning</a></li><li><a href="https://www.kaggle.com/moghazy/ensemble-learning-with-feature-engineering">Ensemble Learning with Feature Engineering</a></li><li><a href="https://mlwave.com/kaggle-ensembling-guide/">Kaggle Ensembling Guide</a></li></ul><h3>Conclusion</h3><p>In this post, we explored the why<em>s </em>and hows of ensemble learning. As we can see, ensemble techniques help in preventing overfitting and reducing the bias and variance of the resultant model, while increasing accuracy.</p><p>There are a lot of algorithms that employ ensemble learning, and we discussed a few. But there’s much more to learn if you’re so inclined. Hope this post was informative. All feedback is welcome.</p><p><strong><em>Until next time! :)</em></strong></p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=390027fe38b8" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/a-guide-to-ensemble-learning-390027fe38b8">A Guide to Ensemble Learning</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Mathematics behind Linear Regression Part II]]></title>
            <link>https://heartbeat.comet.ml/understanding-the-mathematics-behind-linear-regression-part-ii-64fb734d758b?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/64fb734d758b</guid>
            <category><![CDATA[gradient-descent]]></category>
            <category><![CDATA[data-science-for-ml]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Fri, 29 May 2020 14:02:01 GMT</pubDate>
            <atom:updated>2021-10-15T16:51:15.443Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring linear regression: Mathematical foundations and benefits and limitations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ngij4nll0VOBpQmJsfEn6w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@thomasq?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Thomas Q</a> on <a href="https://unsplash.com/s/photos/clothesline?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In part two of this look at the foundations of linear regression, we’ll pick up where we left off. If you haven’t read the <a href="https://heartbeat.comet.ml/understanding-the-mathematics-behind-linear-regression-part-1-4390b7367787">part 1</a>, I highly encourage you to do so.</p><p>In part 1, we discussed regression at a high level and performed feature engineering on the Facebook-Comment-Prediction dataset. The goal at hand is to predict the number of comments on a post, given various features. We selected the final 14 features after pre-processing. The dataset description can be found <a href="https://github.com/nikitaa30/Facebook-Comment-Prediction-with-Linear-Regression/blob/master/dataset/dataset-description">here</a>.</p><p>Here, we’re going to further discuss the underlying mathematics of linear models, apply them on our dataset, and learn about some of its benefits and limitations.</p><p><em>So let’s get started!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*eRqtnWb6MRPnA22BIekSEw.png" /></figure><h3>Cost Function</h3><p>Cost functions in machine learning are used to inspect the performance of the model by determining the relationship between features and the target variable. In other words, they tell us how good/bad our model is at making predictions.</p><p>Mathematically, the value of the cost function is calculated in terms of the <strong>distance between the original values of the target variable in samples and the values predicted by the model</strong>.</p><p>The most frequently used cost function for linear regression is the <strong>mean</strong> <strong>squared error (MSE) function.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*3HBJ3P5c8PUL0hjZtCAUyg.png" /><figcaption>Source:<a href="https://i.stack.imgur.com/O752N.png">https://i.stack.imgur.com/O752N.png</a></figcaption></figure><ul><li>hθ(x(i))=θ0+θ1x(i)</li><li>(x(i),y(i))(x(i),y(i)) is the ith training data.</li><li>m is the number of training samples.</li><li>1/2 is a constant that helps provide an average value for the derivative of the cost function.</li></ul><p>(hθ(x(i))−y(i))² is essentially the squared difference of the predicted and calculated values of the target variable. We’ll use this formula in the next section in a Python code snippet to calculate gradient descent.</p><p><strong><em>The objective of a machine learning model is to find optimal values of certain parameters to minimize the value of the cost function.</em></strong></p><h3>Gradient Descent</h3><p>As we all know, the lower the difference between predicted and calculated values of target variables, the better.</p><p><em>But how do we reach such a stage?</em></p><p>The answer is gradient descent. It’s an optimization algorithm that helps us reach the minima (preferably global) of the cost function iteratively. Let’s imagine an example—let’s say you’re standing at the top of a mountain and you want to reach the valley. Now, there may be several troughs in your path (local minima) but the valley is your ultimate goal (global minima).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*iNPHcCxIvcm7RwkRaMTx1g.jpeg" /><figcaption>Source: <a href="https://saugatbhattarai.com.np/what-is-gradient-descent-in-machine-learning/gradient-descent-2/">https://saugatbhattarai.com.np/what-is-gradient-descent-in-machine-learning/gradient-descent-2/</a></figcaption></figure><p>In this image, as we can see, the size of the steps taken matters. If you take large steps, you might end up missing the global minima. Whereas, too little steps would make the algorithm slow. So, we need to update the size of our steps according to the distance from the minima.</p><p><strong>How do we do that?</strong></p><p>If you think back on high school math, you might recall that derivatives help us find the maxima and minima of a function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*PALSF085qeje7OLMgIUniA.png" /><figcaption>Source: <a href="https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated">https://stats.stackexchange.com/questions/278771/how-is-the-cost-function-from-logistic-regression-derivated</a></figcaption></figure><p>Here, as we can see from the illustration and the formula above , more distance from the minima implies steeper steps, since the derivative value would be larger, (the larger the slope, the higher the magnitude of the derivative) and as we get closer and closer to the minima, the step size decreases, since the slope also decreases. Thus, continuously updating the step value ensures we can reach a global minima.</p><h4>Implementing gradient descent in Python</h4><p>Let’s have a first glance at the gradient descent algorithm, and then examine it step-by-step.</p><pre>#saving the value of cost function at each step<br>costs = list()<br>#regularised gradient descent with lambda user-provided</pre><pre>def gradient_descent(alpha, x, y, numIterations, lmbda):<br>    m = x.shape[0] #number of samples<br>    x = np.c_[ np.ones(m), x]<br>    theta = np.ones(14)<br>    x_transpose = x.transpose()<br>    <br>    for iter in range(0, numIterations):<br>        hypothesis = np.dot(x, theta)<br>        loss = hypothesis - y<br>        J = (np.sum(loss**2)//(2*m))<br>        costs.append(J)<br>        gradient = np.dot(x_transpose, loss)/m   <br>        theta = theta - alpha * gradient <br>        <br>    theta1 = theta<br>    y_train_pred = np.dot(x,theta1)<br>    # root mean square error<br>    rms_train = ((((y_train_pred - np.array(y))**2).sum())/m)**0.5<br>    <br>    x1 = np.c_[ np.ones(X_test.shape[0]), X_test]<br>    y_test_pred = np.dot(x1,theta)<br>   <br>    rms_test = ((((y_test_pred - np.array(Y_test))**2).sum())/(x1.shape[0]))**0.5 ## root mean square error<br>    xyz = (alpha,rms_train,rms_test)<br>    return xyz</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*8tFzvPyrDELLipSg0lfUWw.png" /></figure><p>In this loop, we calculate the predicted value and then its difference from the target value. J, or the cost function, is the summation of this difference over all the samples.</p><p>theta = theta - alpha * gradient updates the value of the step according to the derivative (gradient=slope= derivative). With the help of the final value of theta, we calculate y_pred and then rms_test and rms_train values. The root mean square error values provide us with an intuition about the fit of our model. Since it’s essentially the squared difference of the predicted and calculated target value (root squared), the lower the rms values, the better the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*akWH2ou8p8ExWVMgsPkDUg.png" /></figure><h3>Regularized Linear Regression</h3><p>Regularization refers to providing additional information to the cost function to smoothen the learning curve, avoid overfitting, and stabilize the estimates when there’s collinearity among the features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*eblqjloAjCPX4CbEWLqMVQ.png" /><figcaption>Source: <a href="https://www.globalsoftwaresupport.com/regularization-machine-learning/">https://www.globalsoftwaresupport.com/regularization-machine-learning/</a></figcaption></figure><h4>Types of Regularization</h4><ol><li><strong>L1 regularization:</strong> In L1 norm (normalization), we use the <strong><em>squared magnitude</em></strong> of the coefficient as a penalty term to the cost function. The linear model employing L1 regularization is also called ridge regression.</li><li><strong>L2 Regularization: </strong>In L2 norm, we use the absolute value of magnitude as a penalty term to the cost function. The linear model employing L2 regularization is also called lasso (Least Absolute Shrinkage and Selection Operator) regression.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pv8LrA_E7Npr3FXz9UvBmg.png" /><figcaption>Source: <a href="https://towardsdatascience.com/only-numpy-implementing-different-combination-of-l1-norm-l2-norm-l1-regularization-and-14b01a9773b">https://towardsdatascience.com/only-numpy-implementing-different-combination-of-l1-norm-l2-norm-l1-regularization-and-14b01a9773b</a></figcaption></figure><h4>Difference between lasso and ridge techniques</h4><p>One major difference between the two is that lasso reduces the value of less important features to zero, thereby helping in feature selection for sparse data space. Ridge, on the other hand, leads to very small but non-zero coefficient values for the less important features.</p><h4>Implementing regularization in Python</h4><p>Next, let’s implement ridge regression in our model. The cost function J needs to be tweaked just a little to accommodate regularization. Regularizing the model in our case is extremely necessary because we’re performing multiple regression. We have finalized 14 features, and these many features can lead to overfitting or bias since they may end up having complex relationships with the target variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*OVb40Bx05LxKlNakSb6dlg.png" /></figure><p><em>There we go!</em></p><h3>Knowing when to stop training</h3><p>There are numerous ways in which we can determine when to stop the algorithm from training on the train_set. We do so according to our problem statement and requirements. For example, we sometimes terminate the algorithm after a few thousand iterations, when the validation error is found to be lowest, or when the cost function’s value is minimized. In our case, we let the model run for a number of iterations.</p><blockquote><strong>Note</strong>: The code applies multiple regression from scratch; hence, it may not be as optimized as sklearn regressors. It’s written for understanding the concepts and implementing things as we prefer. In general, we always use validation error to stop the algorithm.</blockquote><h3>Metrics for evaluating performance of the model</h3><p>We used rms_train and rms_test values for evaluating our model for a certain alpha value. Here are the cumulative results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*5V-SpRzrXHYoxHreqTHF_Q.png" /></figure><p>As you can see, we got the best results for our dataset by using 0.30 as an alpha value.</p><p>A few other metrics that are widely used include:</p><ol><li><strong>Mean squared error (MSE): </strong>The average of the squared difference between the target value and the value predicted by the model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*JCGJpM4sleClep1wyeWqXQ.png" /></figure><p>2. <strong>Mean absolute error (MAE): </strong>The absolute difference between the target value and the value predicted by the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*zTj6MJBgYcvaqcVrwzqJMA.png" /></figure><p>3.<strong> R² (R squared):</strong> This technique compares our model with a constant baseline. This constant baseline is depicted by sketching the mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JwEiZQSkL4I710994WaY4w.png" /></figure><h3>Advantages of Linear Regression</h3><p>The advantages of linear regression are as follows:</p><ol><li><strong>Simplicity: </strong>Linear regression provides fair results when there is a linear relationship between the covariates and the target variable.</li><li><strong>Mathematically sound:</strong> Given all the assumptions of the linear regression model are met by the data, mathematically, it’s straightforward to estimate the weights, and you have a guarantee of finding the optimal weights. Also, there are several measures of errors and values to determine the model’s fit.</li><li><strong>Fast: </strong>The structure of the model and the underlying calculations for the regression model can be computed in very less time. Thus, the response time for the model is slower, even with large amounts of data.</li></ol><h3>Disadvantages of Linear Regression</h3><p>The following are a few disadvantages of linear regression:</p><ol><li><strong>Over-simplification: </strong>The model over-simplifies real-world problems where variables exhibit complex relationships among themselves. Fitting a linear model on such data will result in high R² score.</li><li><strong>Feature correlation: </strong>This may adversely affect a linear model. A feature with high correlation to the target may be given negative weights, due to its highly-negative correlation with another feature. Completely correlated features make it impossible to find a unique solution for the linear equation.</li></ol><h3>Repository</h3><p>This is the link to the complete source code, along with the dataset that i used.</p><p><a href="https://github.com/nikitaa30/Facebook-Comment-Prediction-with-Linear-Regression">nikitaa30/Facebook-Comment-Prediction-with-Linear-Regression</a></p><h3>Sources to get started with Linear Regression</h3><ul><li><a href="https://www.kaggle.com/divan0/multiple-linear-regression">Multiple Linear Regression</a></li><li><a href="https://www.kaggle.com/sudhirnl7/linear-regression-tutorial">Linear Regression Tutorial</a></li></ul><h3>Conclusion</h3><p>In this post, we continued our discussion of linear regression and its the mathematical foundations. Applying linear regression (in our case, multiple regression) manually and writing all the functions give us better insight into the techniques underpinnings.</p><p>Despite all its limitations, it’s a widely-used algorithm. This post can help anyone to apply regression from scratch. All feedback is welcome.</p><p><strong><em>Please help me grow and improve!:)</em></strong></p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64fb734d758b" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/understanding-the-mathematics-behind-linear-regression-part-ii-64fb734d758b">Understanding the Mathematics behind Linear Regression Part II</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spam Filtering using Bag-of-Words]]></title>
            <link>https://heartbeat.comet.ml/spam-filtering-using-bag-of-words-1c5484ff07f1?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c5484ff07f1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spam-filter]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[heartbeat]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Tue, 12 May 2020 13:47:12 GMT</pubDate>
            <atom:updated>2021-09-30T15:16:20.052Z</atom:updated>
            <content:encoded><![CDATA[<h3>Spam Filtering Using Bag-of-Words</h3><h4>Guide to building your own Spam filter in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VN_iumSQ1jKm_Gq1F5PXAQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@minarikd?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Daniel Minárik</a> on <a href="https://unsplash.com/s/photos/word-pile?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post, we’re going to employ one simple natural language processing (NLP) algorithm known as <em>bag-of-words</em> to classify messages as ham or spam. Using bag of words and feature engineering related to NLP, we’ll get hands-on experience on a small dataset for SMS classification.</p><p><em>So, what are we waiting for?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*74cMLAx3N6nHwdX0MvKXTg.jpeg" /><figcaption>Source: <a href="https://adage.com/article/digitalnext/holiday-email-practices-avoiding-spam-filter/306437">https://adage.com/article/digitalnext/holiday-email-practices-avoiding-spam-filter/306437</a></figcaption></figure><h4>The Problem: Spam Messages</h4><p>Spam emails or messages belong to the broad category of unsolicited messages received by a user. Spam occupies unwanted space and bandwidth, amplifies the threat of viruses like trojans, and in general exploits a user’s connection to social networks.</p><p>Spam can also be used in Denial of Service (DOS) or Distributed Denial of Service (DDOS) attacks. Various techniques are employed to filter out spam messages, usually centered on content-based filtering. This is because specific keywords, links, or websites are repeatedly sent in bulk to users, characterizing them as spam.</p><h4>The Solution: Text Classification</h4><p>Comparatively speaking, languages are harder for algorithms to interpret and analyze than numeric data. This is true for a few reasons:</p><ol><li>Sentences are not of fixed lengths, but most algorithms require a standard input vector size. Thus, padding is required, corresponding to the largest sentence in the corpora.</li><li>ML algorithms cannot understand words as input: hence, each word needs to be represented by some numeric value.</li></ol><h3>Bag-of-Words Model</h3><p>A bag-of-words model allows us to extract features from textual data. As we know, an algorithm doesn’t understand language. Thus, we need to use a numeric representation for the words in the corpus. This numeric representation can later be fed to any algorithm for further analysis.</p><p>It’s called “bag-of-words” because the order of the words or the structure of the sentence is lost in this model. Only the occurrence or presence of a word matters.</p><p>We can think of the model in such a way — we have a big bag, empty at the start, and a vocabulary or a corpus. We pick up words one by one and put them in the bag, adding to the frequency of their occurrence, and then select the most common words as features for passing through our algorithm of choice.</p><p>Thus, it promotes the view that similar documents consist of similar kinds of words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*WN18F5oVHKzf_DXcCpSFiQ.png" /><figcaption>Source: <a href="https://dudeperf3ct.github.io/lstm/gru/nlp/2019/01/28/Force-of-LSTM-and-GRU/">https://dudeperf3ct.github.io/lstm/gru/nlp/2019/01/28/Force-of-LSTM-and-GRU/</a></figcaption></figure><h3>Dataset</h3><p>The dataset that we’re going to use in this article is an SMS spam collection dataset. It contains over 5500 messages in English, with each message in a column, with the corresponding column next to it specifying whether the text is ham or spam.</p><p>You can find the dataset <a href="https://github.com/nikitaa30/Spam-Filtering-techniques/tree/master/dataset%20for%20bag%20of%20words">here</a>. The complete source code can be found in <a href="https://github.com/nikitaa30/Spam-Filtering-techniques/blob/master/bag%20of%20words.py">this</a> repository.</p><h4>Importing the dataset</h4><p>To import the dataset into a Pandas dataframe, we use the couple of lines written below:</p><pre>import pandas as pd<br>dataset = pd.read_csv(&#39;spam.csv&#39;, encoding=&#39;ISO-8859-1&#39;);</pre><p>Here’s a glimpse of the dataset we are working on. We, later convert the labels into dummy variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*0CAJJpxnTGK7v9z2hsD0NA.png" /></figure><h3>Pre-requisites</h3><ol><li><a href="https://heartbeat.comet.ml/nlp-chronicles-intro-to-nlp-with-nltk-b2c369fbb9a7">NLTK</a> for NLP-related tasks</li><li>NumPy and <a href="https://heartbeat.comet.ml/tips-and-tricks-for-data-analysis-with-pandas-dc0ae909e6be">Pandas</a> for mathematical operations</li><li>Scikit-learn for tokenization and tf-idf model</li><li><a href="https://heartbeat.comet.ml/introduction-to-matplotlib-data-visualization-in-python-d9143287ae39">Matplotlib</a> and <a href="https://heartbeat.comet.ml/seaborn-heatmaps-13-ways-to-customize-correlation-matrix-visualizations-f1c49c816f07">Seaborn</a> for visualisation tasks</li></ol><h3>Data Pre-processing</h3><h4>Removing Stopwords</h4><p>Stopwords refer to the words in the statement which add no specific meaning to it. They often involve prepositions, helping verbs, and articles (i.e. in, the, an, is). Since these add no value to our model, we need to eradicate them.</p><pre>nltk.download(&#39;stopwords&#39;)<br>from nltk.corpus import stopwords<br>stopwords.words(&#39;english&#39;)</pre><h4>Removing non-alphabetic characters</h4><p>Since we use only words for text classification, we need to get rid of punctuation and numbers. For this, we use string matching or regex in Python. The below regex only preserves alphabetic words, discarding the rest.</p><pre>text = re.sub(&#39;[^A-Za-z]&#39;, &#39; &#39;, text)</pre><h4>Changing all to lower case</h4><p>The model cannot distinguish between lowercase and uppercase, treating ‘Text’ and ‘text’ as different words. We certainly don’t want that; thus, we change the case of all words to lowercase for simplicity.</p><pre>text = text.lower()</pre><h4>Spelling Corrections</h4><p>Sometimes, people write abbreviations or misspell words by mistake. To correct these instances, we use the autocorrect package and its spell corrector.</p><pre>from autocorrect import spell<br>text.append(spell(word))</pre><h4>Stemming and Lemmatization</h4><p>Words like act, actor, and acting all are for of the same root word (act).same Stemming and lemmatization are techniques used to truncate words in order to to get the stem or the base word.</p><p>The difference between these is that after stemming, the stem may not be an actual word, whereas lemmatization always produces a real world, which results in better interpretation of the corpora by humans.</p><p>For example, studies could be stemmed as studi (not a word), but will be lemmatized as study (an existing word).</p><p>Here’s is a comparison of the dataset after and before stemming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1KFpmWTPvGo1Azzn78Pyiw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/169/1*KtfgBjoT9YbHycthEcTfTw.png" /></figure><p>You can notice the highlighted words are among a few to be stemmed. Also, stemmed words are not actual words most of the time.</p><h4>Visualizing Spam Keywords</h4><p>Data visualization is a handy way of better understanding the text data involved in our dataset. For example, we can make a wordcloud, which represents most common words in a space, with the size of each word proportional to the frequency of its occurrence. A few other visualization techniques are discussed <a href="https://machinelearningmastery.com/data-visualization-methods-in-python/">here</a>.</p><pre>from wordcloud import WordCloud<br>import matplotlib.pyplot as plt<br>spam_wc = WordCloud(width = 600,height = 512).generate(spam_words)<br>plt.figure(figsize = (12, 8), facecolor = &#39;k&#39;)<br>plt.imshow(spam_wc)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*5Ymqkx0YiLcqfFtvwUP70Q.png" /><figcaption>Wordcloud of spam words from our dataset.</figcaption></figure><h3>Feature Engineering</h3><p>Now we need to perform manipulation on the cleaned, pre-processed dataset to transform it into a form more suitable for applying a machine learning algorithm.</p><h4>Tokenisation</h4><p>For the bag of words implementation, we use CountVectorizer from scikit-learn,<strong> </strong>which counts the frequency of each word present in our pre-processed dataset, and takes the n most common words as features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*GIb1j98BvG5oLjMqWMC3NA.png" /><figcaption>Source: <a href="https://towardsdatascience.com/natural-language-processing-count-vectorization-with-scikit-learn-e7804269bb5e">https://towardsdatascience.com/natural-language-processing-count-vectorization-with-scikit-learn-e7804269bb5e</a></figcaption></figure><p>CountVectorizer returns a matrix, where the rows contain the count of messages containing the word, and columns are the top selected features.</p><pre>from sklearn.feature_extraction.text import CountVectorizer<br>data = CountVectorizer(max_features=2000)<br>X = data.fit_transform(dataset).toarray()<br></pre><p>Since the countvectoriser contains 2000 features, they are hard to depict here. Thus, <strong>for an example below,we take the first 25 words of our dataset, tokenise them and select 10 of the most frequently used ones.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*34ryEQ10eX_Be19ktXp1ZQ.png" /></figure><p>The matrix that represents the frequency of each of these features in our messages(dataset) is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*7Baa5XqFyLhlVBqcrfMIAg.png" /><figcaption>z is a sparse matrix consisting of mostly zeros.</figcaption></figure><h3>Developing the Model</h3><p>Now that our dataset is ready with its attributes, we pass it through any algorithm of our choice. Here, after splitting the dataset into training and test sets, I’ve used a simple <a href="https://heartbeat.comet.ml/understanding-the-mathematics-behind-naive-bayes-ab6ee85f50d0">Naive Bayes</a> classifier for demonstration. You can use any algorithm of your choice depending on the dataset.</p><pre>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(X, y)</pre><pre>from sklearn.naive_bayes import GaussianNB<br>classifier = GaussianNB()<br>classifier.fit(X_train, y_train)<br>y_pred = classifier.predict(X_test)</pre><h3>Results</h3><p>Let’s see how our simple model works on a test set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*FrixIVa3qa_UliVVVDJBXg.png" /></figure><p>Looks like our model crossed the finish line with a decent accuracy of ~80%. Not something to boast about, but still pretty decent given the simplicity of our model and its drawbacks, which are discussed in the next section. Thus, we can say that our model differentiates between ham and spam with a good confidence level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vkFAPzI2X1UQQTaVcrV7NA.png" /></figure><p>Here is an image of the confusion matrix depicting the true positives and false positives in the first row and false negatives and true negatives in the next row respectively.</p><h3>Drawbacks of the Bag-of-Words Model</h3><p>The bag-of-words model assumes that the words are independent. Thus, it doesn’t take into account any relationship between words. Hence, the meaning of sentences is lost.</p><p>Also, the structure of the sentence has no importance in the eyes of our model Two sentences like “These clams are good” and “Are these clams good?” mean the same to the of bag-of-words model, though one is a claims and one is a question. Additionally, for a large vocabulary, bag-of-words result in a very high-dimensional vector.</p><h3>Improvements to the above model</h3><p>A few ways to improve the accuracy of the above model include:</p><ol><li>Using custom-made stopwords, as per the dataset’s requirements(what language or lingo your dataset use), you can add other words according to the language of the corpus. Some of the text or links are specific to spam mails, or some lingo is added to pass generic spam filters. This can be avoided by analyzing the dataset well and knowing about the structure and content of spam messages specifically, tailoring stopwords to our needs.</li><li>Instead of using uni-grams (individual words), using bi-grams and tri-grams can be beneficial to better understand a message’s meaning. For example, take a message — “gift card worth millions”. Here, instead of using uni-grams, which would give us ‘gift’, ‘card’, ‘worth’, ‘millions’, we can use bi-grams that give us ‘gift card’ or ‘worth millions’ together as one feature. As you can see, this could clearly indicate a spam message, whereas ‘gift’, ‘card’, ‘worth’, and ‘millions’ could individually be a part of any day-to-day conversation.</li><li>Apply a vector space model like cosine similarity between messages, and use tf-idf vectorization to better understand the relative weight of a word to that document’s importance. You can read more about this here in <a href="https://heartbeat.fritz.ai/recommender-systems-with-python-part-i-content-based-filtering-5df4940bd831">this</a> article.</li></ol><h4>Repository</h4><p>Here’s the link to my repository, where you can find the complete source code for this tutorial. Also, I will keep on adding code on spam filtering using other techniques soon, so stay connected.</p><p><a href="https://github.com/nikitaa30/Spam-Filtering-techniques">nikitaa30/Spam-Filtering-techniques</a></p><h3>Sources to get started with NLP</h3><ul><li><a href="https://www.kaggle.com/abhishek/approaching-almost-any-nlp-problem-on-kaggle">Approaching (Almost) Any NLP Problem on Kaggle</a></li><li><a href="https://www.kaggle.com/philculliton/nlp-getting-started-tutorial">NLP Getting Started Tutorial</a></li></ul><h3>Conclusion</h3><p>In this post, we implemented a spam text classifier using a bag-of-words model. We learned how to work efficiently with text data and develop a reliable model using a few of NLP concepts.</p><p>There is a lot more to NLP, and spam filtering in general is a mature field, with various machine learning and deep learning techniques commonly used to improve model results. In future posts, I’ll try to approach spam filtering with different techniques. All feedback is welcome. Please help me improve!</p><p><em>Until next time!</em>😁</p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c5484ff07f1" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/spam-filtering-using-bag-of-words-1c5484ff07f1">Spam Filtering using Bag-of-Words</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Mathematics Behind Linear Regression (Part 1)]]></title>
            <link>https://heartbeat.comet.ml/understanding-the-mathematics-behind-linear-regression-part-1-4390b7367787?source=rss-2fa4208c113d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4390b7367787</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[heartbeat]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-science-for-ml]]></category>
            <dc:creator><![CDATA[Nikita Sharma]]></dc:creator>
            <pubDate>Wed, 06 May 2020 14:13:36 GMT</pubDate>
            <atom:updated>2021-09-30T15:47:57.001Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring regression and feature engineering on the facebook-comment-prediction dataset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pIPP1pYHHep9OdAbBLlpQw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@ridham?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ridham Nagralawala</a> on <a href="https://unsplash.com/s/photos/line?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this post, we’re going to learn about the most basic regressor in machine learning—<strong>linear regression</strong>. Specifically, we’re going to walk through the basics with a practical example in Python, and learn about applying feature engineering to the dataset we work with.</p><p>This is a two-part article, and this part encompasses the basics of feature engineering and regression. In part 2, we’ll discuss gradient descent (coding it from scratch), <a href="https://heartbeat.comet.ml/deep-learning-best-practices-regularization-techniques-for-better-performance-of-neural-network-94f978a4e518">regularization</a>, and other relevant concepts.</p><p><strong><em>So let’s dive right in!</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*9UhHkvTAQRVAWdOp9MsMuQ.png" /><figcaption>Source: <a href="https://uc-r.github.io/linear_regression">https://uc-r.github.io/linear_regression</a></figcaption></figure><h3>What is Regression?</h3><p>Regression is a type of <strong>supervised machine learning</strong> problem, under which the relationship between features in the data-space and the outcome is established, to predict the values for the outcome, which are continuous by nature.</p><p>The <strong>difference between classification and regression</strong> is that the outcome expected by a classification problem is categorical or discrete in nature, whereas regression provides a continuous value. The target classes in a classification problem are mere dummies—meaning that an input belonging to red, green, and blue groups might be given values of 0, 1, and 2 respectively.</p><p>But in regression, provided values are always meaningful and deducible, like price of a house based on the feature space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*TUy3lCMdOxBgQJ-HwYtYjA.png" /><figcaption>Source: <a href="https://www.javatpoint.com/regression-vs-classification-in-machine-learning">https://www.javatpoint.com/regression-vs-classification-in-machine-learning</a></figcaption></figure><h3>How does linear regression work?</h3><p>The name itself provides an explanation of the concept behind linear regression—specifically, it assumes a linear relationship between input variables (x) and output variables (y). Liner regression tends to establish a relationship between them by formulating an equation that describes the outcome (y) as a linear combination of the input variables (multiplied with the corresponding variables learned by the model from training).</p><h4>Types of Linear Regression</h4><p><strong>◼ Simple Linear Regression</strong></p><p>Simple linear regression establishes a linear relationship between one input and one output variable, along with a constant co-efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*Px6sf9bde7b0b0cyJe-BJA.png" /></figure><p>◼ <strong>Multiple Linear Regression</strong></p><p>When multiple input variables are linearly combined to get the value of outcome, it is called multiple linear regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*b47Omorcc7QSINPUbvUYrA.png" /></figure><p>In the above equations,</p><p>y = the target variable.</p><p>x = the input variable.(x1,x2,…xn in case of multiple regression)</p><p>b0 = the intercept value.</p><p>b1, b2,.. bn = Coefficients describing the linear relationship between a combination of the input variables and target variable.</p><p><strong>Effect of b0 on x-y relationship:</strong></p><p>The value of b0 defines the expected mean of target (y) when input (x) is 0. If we assign b0 to be 0, then we’re forcibly trying to pass the regression line through the origin, where both x and y are 0 and 0. This may be helpful in some cases, while reducing accuracy in others. Thus, it’s important to experiment with the values of b0, as per the needs of the dataset you’re working with.</p><p><strong>Effect of b1 on x-y relationship:</strong></p><p>If b1 is greater than zero, then the input variable has a positive impact on the target. The increase in one would lead to an increase in the value of the other. Whereas, if b1 is less than zero, the input and output variables will have an inversely proportional or negative relationship. Thus, an increase in one would lead to a decrease in the value of the other.</p><h3>Implementation in scikit-learn</h3><p>Implementing linear regression in scikit-learn is fairly easy. Just import the packages required, load the dataset into a Pandas dataframe, fit the model on the training set, and then predict and compare the test set results:</p><pre><strong>&gt;&gt;&gt; import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>&gt;&gt;&gt; from</strong> <strong>sklearn.linear_model</strong> <strong>import</strong> LinearRegression<br><strong>&gt;&gt;&gt; </strong>X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])<br><strong>&gt;&gt;&gt; </strong><em># y = 1 * x_0 + 2 * x_1 + 3</em><br><strong>&gt;&gt;&gt; </strong>y = np.dot(X, np.array([1, 2])) + 3<br><strong>&gt;&gt;&gt; </strong>reg = LinearRegression().fit(X, y)<br><strong>&gt;&gt;&gt; </strong>reg.score(X, y)<br>1.0<br><strong>&gt;&gt;&gt; </strong>reg.coef_<br>array([1., 2.])<br><strong>&gt;&gt;&gt; </strong>reg.intercept_<br>3.0000...<br><strong>&gt;&gt;&gt; </strong>reg.predict(np.array([[3, 5]]))<br>array([16.])</pre><h3>Dataset</h3><p>The dataset that we’re going to use is the “facebook-comment-volume-prediction” dataset, which involves the problem of predicting comment volume traffic or the number of comments on a given post. Several input variables are provided, along with the target variable.</p><p>The dataset(.csv) file is available in my <a href="https://github.com/nikitaa30/Facebook-Comment-Prediction-with-Linear-Regression/tree/master/dataset">GitHub repository</a>, and the description of the data fields is also provided. The dataset contains 24 features, including the target: the number of comments predicted on a post.</p><blockquote><strong>Note: There are a lot of variations of this dataset available online. I’ve taken the one with fewer features. If your dataset varies from mine, no need to worry, the tutorial can be followed on any relevant dataset.</strong></blockquote><p>You can find the complete source code in the same <a href="https://github.com/nikitaa30/Facebook-Comment-Prediction-with-Linear-Regression">repository</a>.</p><h4>Importing the dataset</h4><p>The first step is to import your training and test dataset into a Pandas dataframe. The describe function provides the general measures of min, max, count, etc. for each data field.</p><pre>#importing the training dataset<br>df=pd.read_csv(&quot;train.csv&quot;)<br>#importing the testing dataset<br>test_df = pd.read_csv(&quot;test.csv&quot;)</pre><pre>#description of the dataset in terms of basic measures<br>test_df.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3iEzY9Q0InbRPM3o7Gajrg.png" /></figure><p>This is what my dataset description looks like.</p><h3>Pre-requisites</h3><ol><li>Python 3.3+</li><li>sklearn</li><li>Jupyter Notebook</li><li><a href="https://heartbeat.comet.ml/introduction-to-matplotlib-data-visualization-in-python-d9143287ae39">Matplotlib</a> and <a href="https://heartbeat.comet.ml/seaborn-heatmaps-13-ways-to-customize-correlation-matrix-visualizations-f1c49c816f07">Seaborn</a> for data visualization</li><li>NumPy and <a href="https://heartbeat.comet.ml/tips-and-tricks-for-data-analysis-with-pandas-dc0ae909e6be">Pandas</a> for mathematical and matrix operations</li></ol><h3>Feature Engineering</h3><p>We’re going to take a look first at feature engineering for this dataset. It involves manipulating the data fields in ways that can help fit our model better and improve it’s accuracy.</p><p>Trying various feature engineering tricks for various combinations of fields can result in better accuracies. There are several packages available in Python to perform the same functions in different manners—or, you could code them manually to learn how feature engineering works from the ground up. Whatever suits your needs better!</p><h4>Checking for null values</h4><p>To start, we need to check to see if all the values in our dataset are valid—i.e., there should be no null values.</p><pre>#to check if any values of any of the fields are null.<br>df.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*avK455FxCff_UK-IjjzTZA.png" /></figure><p>As you can see, there are no null values in the dataset, so we can move ahead. If you find null values, you can either replace them with the mean of the feature values or with the neighborhood mean or standard deviation. These can be other default values as well, depending on the problem at hand.</p><h4>Getting acquainted with data types</h4><p>All the data fields should be in a consistent form for any analysis. Hence, for regression analysis, all data fields should be continuous. If we come across incompatible data types, we need to either convert them or drop them if they don’t contribute much to the output.</p><p>For example:</p><ol><li>If the feature is binary (no or yes) convert into dummy variables like 0 and 1, respectively.</li><li>If the feature is categorical, like color or days of the week, assign them respective dummy values.</li><li>If the data is ordinal (first, second, third…), convert it to continuous by using the following formula: (r-1)/(m-1), where r is the rank value of the field and m is the number of assigned ranks.</li></ol><pre>#finding out the data types of all fields<br>train_df.dtypes</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*Z2yRpLcNKnggPsk8b7sV9w.png" /></figure><p>As you can see above, the two features, post_day (day of the week the post was published) and basetime_day (time of day) are two object-type values. I’ll simply drop these features, as they don’t seem to be contributing a great deal to my output, which is the number of comments on the post.</p><h4>Data Visualisation</h4><p>Data visualization is an immensely powerful tool in determining attribute relationships. Large data entries and files can become cumbersome to analyze or find patterns in. Visualizing these can help us gain insight into the overall impact of one feature on another.</p><p><strong>Data Visualization for outlier detection</strong></p><p>We can plot a box-plot to better understand the range of values an attribute covers, and to drop extreme values. It also depicts the region where most data is concentrated. Below is a box-plot of our output target variable.</p><pre>#making a boxplot for outlier detection<br>matplotlib.pyplot.boxplot(df.target)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*aLbgvhSmiTb_g0qMr2XeHQ.png" /></figure><p><strong>Data visualization for interdependencies and correlation</strong></p><p>Correlation among attributes is an important parameter to know, since variables that are highly correlated to each other or the target variable often add bias to linear models, providing false predictions.</p><pre>#finding out the correlation between features<br>corr1 = df.corr()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*nZZMsK125Q1BIk9oTU2VoA.png" /></figure><p>This beautiful depiction easily captures the dependencies of all attributes. As we can see in the picture, no attribute is highly correlated to the target, but a few are related to each other, and we need to eradicate a few of them to get a final list of attributes that are contributing the most to the output.</p><p><strong>Data Visualization for Convergence</strong></p><p>Pie charts are also helpful for understanding the region of convergence for most values of an attribute, like the chart shown below for our target:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/1*NvenZbsjyiZ2qJED9y3U5Q.png" /></figure><p>Most of the posts in our dataset have zero comments, and very few of them have large numbers of comments.</p><h4>Feature Scaling</h4><p>Feature scaling refers to normalizing data to converge within a certain range. The process modifies data values to better fit a given model. Two of the most widely used scaling methods are:</p><p><strong>Log transform</strong></p><p>Log transformation is used when the values of an attribute span a very large interval. Log transformation makes data more easily comparable and interpretable, and also help avoid overflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*whf8fCL_OFyX0bB-WcgUNg.png" /><figcaption>A,L,K are input variables.</figcaption></figure><p>We won’t apply log transform here, since most of the values lie in a fairly small range. For a detailed look at log transformations, refer to <a href="https://kenbenoit.net/assets/courses/ME104/logmodels2.pdf">this</a> pdf.</p><p><strong>Min-max scaling/normalization</strong></p><p>Min-max normalization scales the data between 0 and 1. The formula used is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*ii46F2WDo9mvFvdxzUvGbQ.png" /></figure><p>It helps bringing the data closer to the origin and optimize it, so that there are no huge values which can at once drastically change the value of the cost function.</p><h3>Sources to get started on feature engineering</h3><ul><li><a href="https://github.com/nikitaa30/Image-Compression-using-ML-and-Neural-Networks">nikitaa30/Image-Compression-using-ML-and-Neural-Networks</a></li><li><a href="https://www.kaggle.com/willkoehrsen/introduction-to-manual-feature-engineering">Introduction to Manual Feature Engineering</a></li><li><a href="https://www.kaggle.com/artgor/brute-force-feature-engineering">Brute force feature engineering</a></li></ul><h3>Conclusion</h3><p>In this post, we briefly discussed the nature of regressors, focusing on linear regression in particular.</p><p>We continued by building some fundamentals for feature engineering, an important step in analyzing data through machine learning techniques. The <a href="http://add link later">next</a> post will proceed with an implementation of linear regression using gradient descent, various cost functions, regularization, and several evaluation metrics.</p><p><strong><em>So, stay tuned:)</em></strong></p><p><em>Editor’s Note: </em><a href="https://heartbeat.comet.ml/"><em>Heartbeat</em></a><em> is a contributor-driven online publication and community dedicated to providing premier educational resources for data science, machine learning, and deep learning practitioners. We’re committed to supporting and inspiring developers and engineers from all walks of life.</em></p><p><em>Editorially independent, Heartbeat is sponsored and published by </em><a href="http://comet.ml/?utm_campaign=heartbeat-statement&amp;utm_source=blog&amp;utm_medium=medium"><em>Comet</em></a><em>, an MLOps platform that enables data scientists &amp; ML teams to track, compare, explain, &amp; optimize their experiments. We pay our contributors, and we don’t sell ads.</em></p><p><em>If you’d like to contribute, head on over to our</em><a href="https://heartbeat.fritz.ai/call-for-contributors-october-2018-update-fee7f5b80f3e"><em> call for contributors</em></a><em>. You can also sign up to receive our weekly newsletters (</em><a href="https://www.deeplearningweekly.com/"><em>Deep Learning Weekly</em></a><em> and the </em><a href="https://info.comet.ml/newsletter-signup/"><em>Comet Newsletter</em></a><em>), join us on</em><a href="https://join.slack.com/t/fritz-ai-community/shared_invite/enQtNTY5NDM2MTQwMTgwLWU4ZDEwNTAxYWE2YjIxZDllMTcxMWE4MGFhNDk5Y2QwNTcxYzEyNWZmZWEwMzE4NTFkOWY2NTM0OGQwYjM5Y2U"><em> </em></a><a href="https://join.slack.com/t/cometml/shared_invite/zt-49v4zxxz-qHcTeyrMEzqZc5lQb9hgvw"><em>Slack</em></a><em>, and follow Comet on </em><a href="https://twitter.com/Cometml"><em>Twitter</em></a><em> and </em><a href="https://www.linkedin.com/company/comet-ml/"><em>LinkedIn</em></a><em> for resources, events, and much more that will help you build better ML models, faster.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4390b7367787" width="1" height="1" alt=""><hr><p><a href="https://heartbeat.comet.ml/understanding-the-mathematics-behind-linear-regression-part-1-4390b7367787">Understanding the Mathematics Behind Linear Regression (Part 1)</a> was originally published in <a href="https://heartbeat.comet.ml/">Heartbeat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>