<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by vikashraj luhaniwal on Medium]]></title>
        <description><![CDATA[Stories by vikashraj luhaniwal on Medium]]></description>
        <link>https://medium.com/@vikashrajluhaniwal?source=rss-3c5638fcb6c5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*j1TNtJlOetclYhTOJnywJQ.jpeg</url>
            <title>Stories by vikashraj luhaniwal on Medium</title>
            <link>https://medium.com/@vikashrajluhaniwal?source=rss-3c5638fcb6c5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:15:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vikashrajluhaniwal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Visual Similarity based Recommendation System using Python]]></title>
            <link>https://medium.com/geekculture/building-a-visual-similarity-based-recommendation-system-using-python-872a5bea568e?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/872a5bea568e</guid>
            <category><![CDATA[image-feature-extraction]]></category>
            <category><![CDATA[recommendations]]></category>
            <category><![CDATA[e-commerce-business]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Thu, 15 Apr 2021 11:34:22 GMT</pubDate>
            <atom:updated>2021-04-28T12:33:20.601Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a Visual Similarity-based Recommendation System Using Python</h3><h4>A Comprehensive guide to perform Image-based Recommendation from scratch</h4><p>In today’s competitive world of technology, it is very crucial for a growing <em>e-commerce platform</em> to engage its customers and maintain a consistent brand experience. Instead of allowing the users to perform search after search in order to get their desired items, <strong><em>recommending</em></strong> such relevant items is more impressive and provides a better sense of satisfaction.</p><p><strong><em>Product recommendations</em></strong> can address such challenges very effectively by analyzing the customer’s previous purchasing behavior and current platform usage.</p><p><em>Product recommendations</em> can help in:</p><ul><li>Converting the shoppers to customers</li><li>Engaging the customers</li><li>Boosting sales and revenue</li><li>Delivering the most relevant content</li><li>Maintaining the brand experience</li></ul><p>Broadly speaking there are two kinds of recommendation approaches:</p><ol><li><strong>Content-based recommendations</strong></li><li><strong>Collaborative filtering</strong></li></ol><p>As the name suggests, the <em>content-based</em> method recommends based on the <em>additional content (metadata)</em> about the customers or products. For products, this <em>content</em> may be product title, description, images, category/subcategory, specification, etc.</p><p>So, this approach <em>recommends</em> the products by finding the most similar products to a given product based on the <em>content</em>.</p><blockquote><em>In this post, we will implement a </em>content-based<em> recommendation system by utilising the </em>product images<em>. Basically, the goal is to recommend </em>product images <em>that are very similar to a recently bought/checked </em>product image<em>.</em></blockquote><p>Therefore, this <strong><em>image-based</em></strong> recommendation will be helpful in <em>recommending</em> the most similar products to the customers based on their recent shopping behaviour/platform usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*5wuv75PVNAFJK97tBcZfPw.png" /></figure><p>Let’s start implementing this using the <a href="https://www.kaggle.com/vikashrajluhaniwal/fashion-images">Fashion Product Images Dataset</a>. The dataset contains 2906 <em>product images</em> across four different <em>gender</em> categories (men, women, boys, and girls). It also contains various product features like title, category, subcategory, colour, gender, type, usage, etc.</p><blockquote><em>Additional NOTE : In order to access the entire python code follow the kaggle kernel </em><a href="https://www.kaggle.com/vikashrajluhaniwal/building-visual-similarity-based-recommendation"><em>here</em></a><em>(</em><a href="https://www.kaggle.com/vikashrajluhaniwal/building-visual-similarity-based-recommendation">https://www.kaggle.com/vikashrajluhaniwal/building-visual-similarity-based-recommendation</a><em>) .</em></blockquote><h4>1. Basic Data Analysis</h4><p>First few records from the dataset are as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N0uR2zXf8igI2E3DPtaCVQ.png" /><figcaption>First 5 records from the data</figcaption></figure><p><strong>a.</strong> <strong>Basic statistics — Number of products, subcategories &amp; gender</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54bde6f1710577d61873c97520443fa2/href">https://medium.com/media/54bde6f1710577d61873c97520443fa2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*FxswHyOXoIuIotk68ovLfg.png" /></figure><p>As mentioned earlier, the dataset contains 2906 products of 9 different <em>subcategories</em> across 4 different <em>gender</em> types.</p><p><strong>b. Frequency of each gender</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*z2x29p8XS6yVStOWlEy-wg.jpeg" /></figure><p>From the output, we can observe that most of the products belong toMen category, then Women, and so on.</p><p><strong>c. Distribution of products gender-wise</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f39e70cb551b5fc0e0c2eb2694b3d755/href">https://medium.com/media/f39e70cb551b5fc0e0c2eb2694b3d755/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*SlRFGBHJEt-KI0PDEV659Q.png" /></figure><p>From the bar chart also, we can observe that Men have the highest number of products. Similarly, the dataset is almost balanced.</p><h4>2. Data Preparation</h4><p>Since cross-category <em>recommendations</em> are not preferred, for example, <em>recommending</em> girls’ products to a bachelor. So let’s subset the data <em>gender-wise</em> into 4 different dataframes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4634f49683544aae984e9d3c853a36b3/href">https://medium.com/media/4634f49683544aae984e9d3c853a36b3/href</a></iframe><h4>3. Feature Extraction using ResNet</h4><p>Generally, the product image contains a unique pattern along with its colour, shape, and edges.</p><p>Images with the same kind of such features are supposed to be similar. Therefore, extracting such features from the images will be very helpful in order to <em>recommend</em> the most similar products.</p><p><strong>How to extract features from the images?</strong></p><p>Computer vision techniques can be used to extract features from the images. Here, since we have limitations on data size, compute resources, and time, so let’s use the standard <em>pre-trained</em> models like <strong>ResNet</strong> to extract the features. Such <em>pre-trained </em>models are already fine-tuned and trained on a huge dataset (like <strong>ImageNet</strong>). This process is also known as <em>transfer learning</em>.</p><p><strong>ResNet</strong></p><p>ResNet is an abbreviated form of <em>Residual Networks</em>, first proposed by Kaiming He in 2015. Currently, it is perceived as a classical neural network for many computer vision tasks. In 2015, during the <strong>ImageNet</strong> Challenge, this model out-performed previous models like <em>GoogleNet</em>, <em>VGGNet</em>, and <em>AlexNet</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*LHjPgojb9DBsuR9_pHmaig.png" /><figcaption>ResNet Architecture</figcaption></figure><p>The architecture allows us to train an extremely deep and wide network with 152 layers successfully. In our implementation, we will use <em>ResNet50 </em>(a smaller version of <em>ResNet152</em>) to extract the features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc5c64bd4906b6e7d917b8344552abdd/href">https://medium.com/media/bc5c64bd4906b6e7d917b8344552abdd/href</a></iframe><p><em>extract_features() </em>function extracts the features from the given images. As per the <em>ResNet</em> standard first, we resize the image to 224 x 224 and normalize them using <strong>ImageDataGenerator</strong> available in Keras. Finally, each image is represented as a <em>100352-dimensional</em> feature vector.</p><p>To avoid run-time feature extraction after deployment, the extracted features are persisted in NumPy arrays. We maintain two arrays here for product Ids and extracted features respectively.</p><p>Similarly, this same <em>feature extraction</em> process is repeated for other product images gender-wise.</p><h4>4. Computing the Euclidean distance and recommending similar products</h4><p><em>Distance</em> is the most preferred measure to assess similarity among items/records. Minimum the distance, the higher the similarity, whereas, the maximum the distance, the lower the similarity.</p><p>There are various types of distances as per geometry like <em>Euclidean distance</em>, <em>Cosine distance</em>, <em>Manhattan distance</em>, etc. We will use <em>Euclidean distance</em> here to compute similarity.</p><p>Since we have already extracted the image features so the <em>Euclidean distance</em> can be easily computed using <em>the pairwise_distances() </em>function form <em>sklearn.metrics</em>.</p><p>Once this distance is computed, we can easily <em>recommend</em> the products as per the ascending order of distance. Let’s do this!</p><p><strong>a. Loading the extracted features</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ee9318688dce391a03d95689ab2872b/href">https://medium.com/media/9ee9318688dce391a03d95689ab2872b/href</a></iframe><p><strong>b. Distance computation and Recommendation</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8dbe4d1ed5c8c04392b980f3bddc5f67/href">https://medium.com/media/8dbe4d1ed5c8c04392b980f3bddc5f67/href</a></iframe><p>The above <em>get_similar_products_cnn()</em> function <em>recommends</em> 5 most similar products to the queried product based on the extracted features. The function accepts two arguments — <em>product id</em> of recently bought/checked item and the number of products to be <em>recommended</em>.</p><p>The top 5 <em>recommended</em> products against the product id <em>13683</em> are as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oIH0zmbZHmSWeoMPyBTLeA.gif" /></figure><p>Likewise, we can <em>recommend</em> products against the products from other gender types also. Let’s see the final deployment using <strong>Streamlit</strong>.</p><h4>5. Deploying the Solution</h4><p><strong><em>Streamlit</em></strong> is an interactive library/framework to build data apps &amp; web applications and deploy machine learning workloads. The most important thing is it does not require any prior knowledge of web designing and development. Python knowledge is sufficient to interact with this as it is Python-compatible.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b89852078868d02e177daa3efcc42f0c/href">https://medium.com/media/b89852078868d02e177daa3efcc42f0c/href</a></iframe><p>Here, the below built-in functions are used to make an interactive deployment:</p><ul><li>st.text_input() — takes dynamic input from the user</li><li>st.write() — writes messages/arguments to the app</li><li>st.title() — displays an image or list of images.</li></ul><p>Like earlier, here the <em>get_similar_products_cnn()</em> function <em>recommends</em> most similar products as per the arguments specified.</p><p>To execute this deployment script in the terminal type:</p><pre>streamlit run recom_deployment.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*08uZt5bV6Q3ZbAcXC7R8wg.gif" /></figure><p><em>Tip: This complete deployment code can be downloaded from </em><a href="https://drive.google.com/file/d/123XGxKvRY7sk2pnTmVOyLp9FH-iFL5oN/view?usp=sharing"><em>here</em></a><em>.</em></p><h4>End Notes</h4><p>In this post, we discussed about <em>product recommendations</em> and implemented a <em>visual similarity-based recommendation</em> system by utilising available product images using <em>ResNet</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=872a5bea568e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/building-a-visual-similarity-based-recommendation-system-using-python-872a5bea568e">Building a Visual Similarity based Recommendation System using Python</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommending news articles based on already read articles]]></title>
            <link>https://medium.com/data-science/recommending-news-articles-based-on-already-read-articles-627695221fe8?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/627695221fe8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[content-based]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Tue, 21 Jan 2020 13:50:39 GMT</pubDate>
            <atom:updated>2021-03-27T14:19:47.429Z</atom:updated>
            <content:encoded><![CDATA[<h4>Content based recommendation in Python from scratch</h4><p>Due to the easy availability of enormous items(services) on your favorite online platforms like <em>e-commerce</em>, <em>job portal</em>, <em>food delivery</em>, <em>music or video streaming</em>, it is very hard and time consuming to find out the desired item of your choice quickly. These platforms could help you by <strong>recommending</strong> items as per your interest and preference by just analyzing your past interaction or behavior with the system.</p><p>From <strong>Amazon</strong> to <strong>Linkedin</strong>, <strong>Uber eats</strong> to <strong>Spotify</strong>, <strong>Netflix</strong> to <strong>Facebook</strong>, <strong>Recommender systems</strong> are most extensively used to suggest “Similar items”, “Relevant jobs”, “preferred foods”, “Movies of interest” etc to their users.</p><p><strong><em>Recommender system</em></strong> with appropriate item suggestions helps in boosting sales, increasing revenue, retaining customers and also adds competitive advantage. There are basically two kinds of <em>recommendation</em> methods.</p><ol><li><strong>Content-based recommendation</strong></li><li><strong>Collaborative filtering</strong></li></ol><p>The<strong> <em>content-based recommendation</em> </strong>is based on similarity among users/items obtained through their <strong><em>attributes</em></strong>. It uses the additional information(metadata) about the <strong><em>users</em></strong> or <strong><em>items</em></strong> i.e. it relies on what kind of <strong><em>content</em></strong> is already available. This metadata could be the user’s<strong> <em>demographic information</em></strong> like <em>age</em>, <em>gender</em>, <em>job</em>, <em>location</em>, <em>skillsets,</em> etc. Similarly, for <strong><em>items</em>,</strong> it can be <em>item name</em>, <em>specifications</em>, <em>category</em>, <em>registration date,</em> etc.</p><p>So the core idea is to recommend items by finding similar items/users to the concerned <strong><em>item/user</em></strong> based on their <strong><em>attributes</em></strong>.</p><p>In this blog, we are going to discuss <em>Content-based recommendation</em> using <strong>News category</strong> dataset. The goal is to recommend <em>news articles</em> that are similar to the already read article by using attributes like article <em>headline</em>, <em>category</em>, <em>author</em> and <em>publishing date</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*E33jvgZcZX2Ib8HP2SUClQ.png" /></figure><p>So let’s get started using <a href="https://www.kaggle.com/rmisra/news-category-dataset"><strong><em>News Category Dataset</em></strong></a><strong><em> </em></strong>available on Kaggle. The dataset contains around 200k observations of 6 different features like <em>news headline</em>, <em>category</em>, <em>authors</em>, <em>short description</em>, <em>publishing date,</em> etc.</p><blockquote>Additional NOTE : In order to access the entire python code follow the kaggle kernel <a href="https://www.kaggle.com/vikashrajluhaniwal/recommending-news-articles-based-on-read-articles">here</a>( <a href="https://www.kaggle.com/vikashrajluhaniwal/recommending-news-articles-based-on-read-articles">https://www.kaggle.com/vikashrajluhaniwal/recommending-news-articles-based-on-read-articles</a>).</blockquote><h4>1. Data preprocessing</h4><ul><li><strong><em>Fetching only the articles from the year 2018</em></strong> — Since the dataset size is quite large so processing through the entire dataset may consume too much time. To refrain from this, we are only considering the latest articles from the year 2018.</li><li><strong><em>Removing all the short headline articles </em></strong><em>—</em><strong><em> </em></strong>After stopword removal from the headline, articles with very short headlines may become blank headline articles. So let’s remove all the articles with fewer words(&lt;5) in the headline.</li><li><strong><em>Checking and removing all the duplicates</em></strong><em> — </em>Since some articles are exactly the same in headlines, so let’s remove all such articles having duplicate headline appearance.</li></ul><h4><strong>2. Basic Data Exploration</strong></h4><p><strong>a. Basic statistics — Number of articles, authors, categories</strong></p><p>After data preprocessing there are 8485 news articles by 892 authors across 26 different categories.</p><p><strong>b. Distribution of articles category-wise</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F42%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F42.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3dfa371ebcce26aba0af8631186f523e/href">https://medium.com/media/3dfa371ebcce26aba0af8631186f523e/href</a></iframe><p>From the bar chart, we can observe that politics category has the <em>highest</em> number of articles then entertainment and so on.</p><p><strong>c. Number of articles per month</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F44%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F44.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/4101009a492e140092511a2001dfc865/href">https://medium.com/media/4101009a492e140092511a2001dfc865/href</a></iframe><p>From the bar chart, we can observe that January month has the <em>highest</em> number of articles then March and so on.</p><p><strong>d. PDF for the length of headlines</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F47%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F47.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/2270f2812cc9149fb1c192d5287be9a8/href">https://medium.com/media/2270f2812cc9149fb1c192d5287be9a8/href</a></iframe><p>The <em>probability distribution</em> function of the <em>headline</em> length is almost similar to a <em>Gaussian distribution</em>, where most of the <em>headlines</em> are 58 to 80 words long in length.</p><h4>3. Text Preprocessing</h4><ul><li><strong><em>Stopwords removal </em></strong><em>— </em>Stop words are not much helpful in analysis and also their inclusion consumes much time during processing so let’s remove these.</li><li><strong><em>Lemmatization — </em></strong>Let’s find the base form(lemma) of words to consider different inflections of a word same as the lemma.</li></ul><h3>4. Headline based similarity on new articles</h3><p>Generally, we assess <em>similarity</em> based on <em>distance</em>. If the <em>distance</em> is minimum then high <em>similarity</em> and if it is maximum then low <em>similarity</em>. To calculate the <em>distance</em>, we need to represent the headline as a <em>d- dimensional</em> vector. Then we can find out the <em>similarity</em> based on the <em>distance</em> between vectors.</p><p>There are multiple methods to represent a <em>text</em> as a <em>d-dimensional</em> vector like <em>Bag of words</em>, <em>TF-IDF method</em>, <em>Word2Vec embedding,</em> etc. Each method has its own advantages and disadvantages.</p><p>Let’s see the feature representation of the headline through all the methods one by one.</p><h4><strong>a. Using Bag of Words method</strong></h4><p>A <em>Bag of Words(BoW)</em> method represents the occurrence of words within a <em>document.</em> Here, each headline can be considered as a <em>document</em> and set of all headlines form a <em>corpus.</em></p><p>Using <em>BoW</em> approach, each <em>document </em>is represented by a <strong><em>d-dimensional </em></strong>vector, where <em>d</em><strong> </strong>is the total number of<strong> </strong><em>unique words</em> in the corpus. The set of such unique words forms the v<em>ocabulary</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec3dbe2bc7b871a3775ed38c3286a84d/href">https://medium.com/media/ec3dbe2bc7b871a3775ed38c3286a84d/href</a></iframe><p>The above function recommends 10 <em>similar</em> articles to the <em>queried(read)</em> article based on the headline. It accepts two arguments — index of already read article and the total number of articles to be recommended.</p><p>Based on the <em>Euclidean distance</em> it finds out 10 nearest <em>neighbors </em>and recommends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*DYGga3m-2EGmws7PFR0miw.png" /></figure><p><strong>Disadvantages</strong></p><ol><li>It gives very low <em>importance</em> to less <em>frequently</em> observed words in the <em>corpus</em>. Few words from the queried article like employer, flip, fire appear less <em>frequently</em> in the entire <em>corpus </em>so the <em>BoW</em> method does not <em>recommend </em>any article whose <em>headline </em>contains these words. Since trump is commonly observed word in the <em>corpus </em>so it is recommending the articles with headlines containing trump.</li><li><em>BoW </em>method doesn’t preserve the order of words.</li></ol><p>To overcome the first disadvantage we use the <strong><em>TF-IDF</em></strong> method for feature representation.</p><h4><strong>b. Using TF-IDF method</strong></h4><p><em>TF-IDF</em> method is a weighted measure that gives more importance to less frequent words in a <em>corpus</em>. It assigns a weight to each <em>term(word)</em> in a document based on <em>Term frequency(TF)</em> and <em>inverse document frequency(IDF).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*T-K3qKOMmMNoXB3m4mf0Cw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*6rDFP44AqQCmD6ma1NDnww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*cq6F2MYVhM5HH1GRvz_wXQ.png" /></figure><p>So if a word occurs more number of times in a document but fewer times in all other documents then its <em>TF-IDF </em>value will be high.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1fcc6e154cf1da8c01acb3b410a2268/href">https://medium.com/media/c1fcc6e154cf1da8c01acb3b410a2268/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*zcgeyG3WWe6CxkPKKfwJlw.png" /></figure><p>Compared to <em>BoW </em>method, here <em>TF-IDF </em>method recommends the articles with <em>headlines </em>containing words like employer, fire, flip in the top 5 recommendations and these words occur less <em>frequently</em> in the <em>corpus</em>.</p><p><strong>Disadvantages</strong></p><ul><li><em>Bow </em>and <em>TF-IDF</em> method do not capture <em>semantic</em> and <em>syntactic</em> similarity of a given word with other words but this can be captured using <strong><em>Word embedding</em></strong>. For example, there is a good association between words like trump and white house, office and employee, tiger and leopard, USA and Washington D.C etc. Such kind of <em>semantic</em> similarity can be captured using <em>word embedding</em> techniques.<br><em>Word embedding</em> techniques like <em>Word2Vec</em>, <em>GloVe</em> and <em>fastText</em> leverage <em>semantic </em>similarity between <em>words</em>.</li></ul><h4><strong>c. Using Word2Vec embedding</strong></h4><p><em>Word2Vec </em>is one of the techniques for <em>semantic</em> similarity which was invented by <em>Google</em> in 2013. For a given <em>corpus</em>, during training, it observes the patterns and represents each word by a d<em>-dimensional</em> vector. To get better results we need a fairly large corpus.</p><p>Since our corpus size is small so let’s use Google’s pre-trained model on <em>google news </em>articles. This standard model contains vector representation for billions of words obtained by training on millions of new articles. Here, each word is represented by a 300-dimensional <em>dense </em>vector.</p><p>Since the model gives vector representation for each <em>word</em> but we calculate the distance between <em>headlines </em>so we need to obtain vector representation for each <em>headline</em>. One way to obtain this is by first adding vector representation of all the words available in the <em>headline</em> and then calculating the average. It is also known as the <em>average Word2Vec</em> model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/087f35a56ca1c81cc9ad0afcf42b5c7f/href">https://medium.com/media/087f35a56ca1c81cc9ad0afcf42b5c7f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*D3DVF1eYMvGUHxEsb6H39A.png" /></figure><p>Here, <em>Word2Vec</em> based representation recommends the <em>headlines </em>containing the word white house which is associated with the word trump in the queried <em>article</em>. Similarly, it recommends the headlines with words like official, insist which have semantic similarity to the words employer, sue in the queried <em>headline</em>.</p><blockquote>NOTE : So far we were recommending using only one feature i.e. <em>headline</em> but in order to make a <em>robust</em> recommender system, we need to consider <em>multiple</em> features at a time. Based on the business interest and rules, we can decide weight for each feature.</blockquote><p>Let’s see different models with combinations of different <em>features </em>for article similarity.</p><h4>d. Weighted similarity based on headline and category</h4><p>Let’s first see articles similarity based on <em>headline</em> and <em>category</em>. We are using <em>one-hot encoding</em> to get feature representation for the <em>category</em>.</p><p>Sometimes as per the business requirements, we may need to give more preference to the articles from the same <em>category</em>. In such cases, we can assign more weight to the <em>category</em> while recommending. Higher the weight, more the significance. Similarly, less weight leads to less significance to a particular feature.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/791bdb73d00ff2018d149a0d4579b011/href">https://medium.com/media/791bdb73d00ff2018d149a0d4579b011/href</a></iframe><p>The above function takes two extra arguments <em>w1</em> and <em>w2</em> for weights corresponding to <em>headline</em> and <em>category</em>. It is always a good practice to pass the <em>weights</em> in a range scaled from <em>0 to 1</em>, where a value close to 1 indicates high weight whereas close to 0 indicates less weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*11c7HsyGR5Qo7EbWmKx_KQ.png" /></figure><p>Here, we can observe that the recommended articles are from the same <em>category</em> as the queried article <em>category</em>. This is due to the passing of high value to <em>w2</em>.</p><h4>e. Weighted similarity based on the headline, category, and author</h4><p>Now let’s see calculate articles similarity based on <em>author</em> along with <em>headline</em> and <em>category</em>. Again, we are encoding the <em>author</em> through <em>one-hot encoding</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e97e7be1738675ce0bc7f66249dd0a15/href">https://medium.com/media/e97e7be1738675ce0bc7f66249dd0a15/href</a></iframe><p>The above function takes one extra weight argument <em>w3</em> for the <em>author</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*mtGYMlcfM9wEoQZIfXvflQ.png" /></figure><p>In the output, we can observe that the recommended articles are from the same <em>author</em> as the queried article <em>author</em> due to high weightage to <em>w3</em>.</p><h4>f. Weighted similarity based on the headline, category, author and publishing day</h4><p>Now let’s see calculate articles similarity based on the publishing <em>weekday</em> <em>author</em> along with the <em>headline</em>, <em>category,</em> and <em>author</em>. Again, we are encoding this new feature through <em>one-hot encoding</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ab01a05dc1420154bc5f1bcb1bcebc8/href">https://medium.com/media/8ab01a05dc1420154bc5f1bcb1bcebc8/href</a></iframe><p>The above function takes one extra weight argument <em>w4</em> for <em>day and month</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*AqU9gRAlEsoFJwwOeCgt9Q.png" /></figure><p>In the output, we can observe that the recommended articles are from the same <em>day and month</em> as the queried article due to high weightage to <em>w4</em>.</p><h4>End Notes</h4><p>In this article, we discussed different types of <em>word embedding</em> techniques with different combinations of <em>features </em>for <em>content based recommendation</em> and the common issues associated with each of the techniques.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=627695221fe8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/recommending-news-articles-based-on-already-read-articles-627695221fe8">Recommending news articles based on already read articles</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Probability distribution is must in DS/ML —]]></title>
            <link>https://medium.com/data-science/why-probability-distribution-is-must-in-ds-ml-4edf2057ecfe?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4edf2057ecfe</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tests-of-normality]]></category>
            <category><![CDATA[normal-distribution]]></category>
            <category><![CDATA[qq-plot]]></category>
            <category><![CDATA[binomial-distribution]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Thu, 02 Jan 2020 17:27:23 GMT</pubDate>
            <atom:updated>2021-03-27T14:20:17.017Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Probability distribution is must in DS/ML?</h3><h4>A comprehensive guide</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*0opArocPFkLCruled1_Rdw.png" /></figure><p>As the name suggests <em>probability distribution</em> is the distribution of total <em>probability</em> across all the possible <em>outcomes</em> of a <em>random variable</em>.<br>For example assume a bank provides four different kinds of debit cards(Classic, Silver, Gold, Platinum) to its customers. Each of these debit cards has specific values for different parameters like maximum transaction amount, reward points, annual service cost, the maximum purchase amount, etc.</p><p>The bank provides debit cards based on customer features like age, educational status, Profession, salary, size of family, place of residence, etc.</p><p>Now assume a software professional opens account in the bank then what kind of debit card should be provided to him. After analyzing all the customer features the bank comes with a <em>probability</em> table as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*lxf9yZPT2wJSG5f_VzitkA.png" /><figcaption>Distribution of probabilities across debit card types</figcaption></figure><p>The above table represents the <em>probability distribution</em> of debit cards where total <em>probability</em>(1.0) is distributed across all the four types of debit cards with their corresponding <em>probability</em> values.</p><p><em>Probability distribution</em> is essential in -</p><ul><li><strong>Data Analysis</strong></li><li><strong>Decision making</strong></li></ul><p>This blog emphasizes the need for <em>probability distribution </em>in the above two contexts,<em> </em>types of probability distributions and different types of tests for <em>normality</em> check.</p><p>Let us discuss it with <a href="https://www.kaggle.com/vikashrajluhaniwal/australian-athletes-data-set"><strong><em>Australian athletes&#39; data set</em></strong></a> available on kaggle. This sample data contains various physical attributes of the <em>athletes</em>. It contains 202 observations of 13 different features like height, weight, body mass index, sex, red blood count, white blood count, etc.</p><p>Data Source: <a href="https://www.kaggle.com/vikashrajluhaniwal/australian-athletes-data-set">https://www.kaggle.com/vikashrajluhaniwal/australian-athletes-data-set</a></p><p>Now, let’s try to build a simple classification model to classify <em>Sex</em> of an athlete using one variable at a time. For this purpose, <em>probability density function(PDF)</em> is very helpful to assess the importance of a continuous variable.</p><blockquote>Additional NOTE : This article contains only the final visualization output in interactive mode for different plots. In order to access the python code follow the kaggle kernel <a href="https://www.kaggle.com/vikashrajluhaniwal/5-must-know-probability-distributions">here</a>( <a href="https://www.kaggle.com/vikashrajluhaniwal/5-must-know-probability-distributions">https://www.kaggle.com/vikashrajluhaniwal/5-must-know-probability-distributions</a>).</blockquote><p><strong>a. Analyzing rcc</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F1%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F1.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a03314f8f832d69050ef6cf3387630e/href">https://medium.com/media/1a03314f8f832d69050ef6cf3387630e/href</a></iframe><p>Since the pdf’s of <em>rcc </em>across both the gender categories are slightly overlapping each other so it is not possible to correctly classify all the points based on some condition. A simple model can be built by considering the intersection point of both the pdf’s</p><pre>if(rcc&lt;=4.68):<br> sex = “female”<br>else:<br> sex = “male&quot;</pre><p>From the pdf curve we can observe that up to 4.68(value at intersection) more number of points belong to the female category than man, similarly, after 4.68 there is more number of male points. But this model leads to misclassification for the points belonging to the female category after 4.68 and similarly for the points belonging to the male category before 4.68.</p><p>Similarly, we can consider other variables to build a simple classification model like this. To find out the most significant variable, univariate analysis using pdf can be performed.</p><h3>Uni-variate analysis using pdf</h3><p><strong>Rules of thumb to determine the most significant features using pdf:-</strong></p><ul><li>Higher the separation among pdf curves for different classes of target variable better the classification</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F4%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F4.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/4db32f8a91c4a27523ba6ccc4bfc219f/href">https://medium.com/media/4db32f8a91c4a27523ba6ccc4bfc219f/href</a></iframe><ul><li>Overlapping PDF curves lead to the worst classification</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F6%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F6.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1f3d011aff52e329c00a009f4954b09/href">https://medium.com/media/d1f3d011aff52e329c00a009f4954b09/href</a></iframe><p>Now let’s perform univariate analysis on a subset of features using PDF.</p><p><strong>b. Analyzing wcc</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F8%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F8.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c367222b6fc0f372402794846f200a5/href">https://medium.com/media/7c367222b6fc0f372402794846f200a5/href</a></iframe><p>Since the pdf curves of <em>wcc </em>for both the gender categories are overlapping each other so it is very hard to come up with a condition for simple classification.</p><p><strong>c. Analyzing bmi</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F10%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F10.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/a54c6709f5feb66af2bee6f57f4329a8/href">https://medium.com/media/a54c6709f5feb66af2bee6f57f4329a8/href</a></iframe><p>Since the pdf curves of <em>bmi</em> for both the gender categories are overlapping each other so the simple classification model will have a high level of misclassification.</p><p><strong>d. Analyzing pcBfat</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F12%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F12.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa93243e4ead8b1ed71f00bd95dd6b9b/href">https://medium.com/media/fa93243e4ead8b1ed71f00bd95dd6b9b/href</a></iframe><p>A simple model(as shown below) can be built using <em>pcBfat </em>but again it leads to some misclassification.</p><pre>if(pcBfat&lt;=11.80):<br>    sex = &quot;male&quot;<br>else:<br>    sex = &quot;female&quot;</pre><p><strong>e. Analyzing lbm</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F14%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F14.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3f8d4f27231549c8157696661e6c22b/href">https://medium.com/media/e3f8d4f27231549c8157696661e6c22b/href</a></iframe><p>A simple model(as shown below) can be built using <em>lbm</em> but again it misclassifies the male points below 64.10 as female and the female points above 64.10 as male.</p><pre>if(lbm&lt;=64.10):<br>    sex = &quot;female&quot;<br>else:<br>    sex = &quot;male&quot;</pre><h3>Uni-variate analysis using CDF</h3><p>All the above simple models were leading to some sort of misclassification but using PDF we are unable to compute the level of misclassification. The magnitude of misclassification error can be obtained using CDF curve.</p><p><strong>a. Analyzing rcc</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F16%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F16.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3594e3fcffceca5fce53f96154a74943/href">https://medium.com/media/3594e3fcffceca5fce53f96154a74943/href</a></iframe><p>From the CDF curves we can observe that up to 4.68(condition value from the built model) 12.74% of the male are misclassified as female, similarly, 19% of the female are misclassified as male above 4.68.</p><p>So the total misclassification error = 12.74 + 19 = 31.74%</p><p><strong>b. Analyzing pcBfat</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F18%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F18.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd84d3c63d94ad29847745d453cebb83/href">https://medium.com/media/fd84d3c63d94ad29847745d453cebb83/href</a></iframe><p>From the CDF curves we can observe that up to 11.80(condition value from the built model) 18% of the female are misclassified as male, similarly, 14.70% of the male are misclassified as females above 11.80.</p><p>So the total misclassification error = 18 + 14.70 = 32.70%</p><p><strong>c. Analyzing lbm</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F20%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F20.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/e6484db2d06a6fe920635392f6c138b3/href">https://medium.com/media/e6484db2d06a6fe920635392f6c138b3/href</a></iframe><p>From the CDF curves, we can observe that up to 64.10(condition value from the built model) 12.74% of the male are misclassified as female, similarly, 7% of the female are misclassified as males above 64.10.</p><p>So the total misclassification error = 12.74 + 7 = 19.74%</p><p><strong>Conclusion</strong></p><p>The best predictor based on misclassification error among rcc, pcBfat, lbm is lbm.</p><h3>Types of Probability distribution</h3><p>Based on the type of a random variable(discrete or continuous) there are two types of Probability distributions — Discrete and Continuous. In this blog, we are going to discuss the following probability distributions.</p><ol><li>Discrete probability distribution</li></ol><ul><li>Discrete uniform</li><li>Binomial distribution</li></ul><p>2. Continuous probability distribution</p><ul><li>Continuous uniform</li><li>Normal distribution</li><li>Log-normal distribution</li></ul><h3>1. Normal distribution</h3><p><em>Normal(Gaussian)</em> distribution is a bell-shaped curve, its distribution pattern is observed in most of the natural phenomena such as height, weight, marks, etc. It has two parameters — mean(<strong>µ</strong>) and standard deviation(<strong>σ</strong>).</p><p>PDF of a <em>random variable</em> which follows <em>normal distribution</em> is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*AB2BrRetVnPTLKJ5-tGLqA.png" /></figure><p><strong>Properties of Normal Distribution</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F22%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F22.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d573a0f3adb13d2a0a1de7477c73e2e/href">https://medium.com/media/8d573a0f3adb13d2a0a1de7477c73e2e/href</a></iframe><ul><li>Mean = median = mode</li><li>Symmetric in nature</li><li>The total area under the curve = 1</li><li>As we move away from the mean, the PDF value decreases</li><li>As the variance increases the distribution spread also increases and the curve becomes more wider</li><li>68–95–99.7 Empirical rule<br><strong>-</strong> 68.2% of the data lies within one standard deviation away from the mean<br><strong>-</strong> 95% of the data lies within two standard deviations away from the mean<br><strong>-</strong> 99.7% of the data lies within three standard deviations away from the mean</li></ul><p>If we know in advance that a variable follows normal distribution then we can easily tell many properties of the variable without looking at the actual data.</p><p><strong>Normality check of height, weight column through visualization and skewness</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F24%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F24.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f7eb9a48bb9ddddff34275b8ffe478c/href">https://medium.com/media/4f7eb9a48bb9ddddff34275b8ffe478c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F26%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F26.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce7e38ae826fa7d4385007cbbd1eee54/href">https://medium.com/media/ce7e38ae826fa7d4385007cbbd1eee54/href</a></iframe><p>From the curves, we can easily observe that height and weight are almost normally distributed but there exists a small amount of asymmetricity which can be measured through <strong><em>skewness</em></strong><em>.</em></p><p><strong>skewness: </strong>it is a statistical parameter to measure asymmetricity about the mean in a distribution of the random variable. This parameter value can be positive, negative or undefined. A negative value indicates data is <em>left-skewed</em> whereas a positive value indicates data is <em>right-skewed</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*X6387bKEN7TBlrqTfYrNkw.png" /></figure><p>Here, height is slightly skewed to the left whereas weight is slight to the right.</p><p>Now let’s try to answer the below questions.</p><ul><li>What % of athletes have height &lt;=165 cm?</li><li>What % of athlete has a height between 165 and 185 cm?</li><li>What % of athlete has height &gt;185 cm?</li></ul><p>For a random variable with finite mean and standard deviation, these above questions can be easily answered using <strong><em>Chebyshev’s inequality</em></strong>.</p><p>Assume for a moment that height strongly follows <em>normal distribution</em> then its distribution would have looked like below -</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F28%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F28.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/c872fec3be8f946ebf63978214baacd7/href">https://medium.com/media/c872fec3be8f946ebf63978214baacd7/href</a></iframe><p>If the height feature had perfectly followed <em>normal distribution</em> then above questions would have easily answered using CDF of <em>normal distribution</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*g0gI0odpD-2ffffozAnbtg.png" /></figure><h4><strong>Normality test</strong></h4><p>The normality tests are used to determine whether the data is <em>normally</em> distributed or not OR whether the sample data comes from a <em>normally</em> distributed population or not. <br>There are various kind of graphical and numeric tests to determine this.<br>1. Graphical tests</p><ul><li>Histogram/density plot</li><li>Q-Q plot</li></ul><p>2. Numeric tests</p><ul><li>Shapiro-Wilk test</li><li>Kolmogorov-Smironv test</li></ul><p><strong>a. QQ Plot</strong></p><p>It is a graphical method for comparing two probability distributions by plotting their <strong><em>quantiles</em></strong> against each other. For <em>normality</em> test one distribution is w.r.t. the given sample that we want to test and another distribution is the <em>standard normal</em> distribution.<br>There are built-in methods available in statsmodels and scipy package to plot the <em>Q-Q plot</em>. We can also plot it manually.</p><p><strong>Steps for plotting Q-Q plot manually</strong></p><p>Assume X is a random variable representing the given sample and Y<strong>~</strong>(0,1) is a random variable that follows the <em>standard normal</em> distribution with <em>mean(</em><strong><em>µ</em></strong><em>) </em>equals to 0 and <em>standard deviation(</em><strong><em>σ</em></strong><em>)</em> equals to 1.</p><ul><li>Compute all the percentiles of X. Say x’₁, x’₂, x’₃………..x’₁₀₀. These percentiles are also known as sample quantiles.</li><li>Compute all the percentiles of Y. Say y’₁, y’₂, y’₃………..y’₁₀₀. These percentiles are also known as theoretical quantiles.</li><li>Plot each percentile of X against the same percentile of Y. i.e. 2d points are formed as (x’₁,y’₁), (x’₂,y’₂), (x’₃,y’₃)…………..(x’₁₀₀,y’₁₀₀).</li><li>If all the points lie on a straight line(reference line y=x) then X follows a <em>normal </em>distribution.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F30%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F30.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea220dc519c9c06b0ec02d267924f9f0/href">https://medium.com/media/ea220dc519c9c06b0ec02d267924f9f0/href</a></iframe><p>From the plot can we assume <em>height</em> feature follows a <em>normal</em> distribution?</p><p>Here, most of the points fall about the reference line so our assumption seems to fairly safe.</p><p><strong>b. Shapiro-Wilk test</strong></p><p>It’s a numeric test to check whether a sample is <em>normally</em> distributed or not. It is a <em>hypothesis</em> based test where the <em>null</em> and <em>alternate</em> hypothesis is defined as below -</p><p><strong>H₀(Null Hypothesis): </strong>Sample is <em>normally</em> distributed</p><p><strong>H₁(Alternate Hypothesis):</strong> Sample is not <em>normally </em>distributed</p><p>This, if the <strong><em>p-value</em></strong> obtained for the W statistic is less than the <em>significance level</em>(<strong>α</strong>) then the <em>null hypothesis</em> is rejected, on the other hand, if the <strong><em>p-value</em></strong> is greater than <strong>α </strong>then we failed to reject the <em>null</em> hypothesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*IYU3hDySyiUtFzHifiu2dw.png" /></figure><p>Here for <strong>α</strong> = 0.05, obtained <strong><em>p-value</em></strong>(0.2120) &gt; <strong>α</strong> , so we failed to reject <em>null</em> hypothesis i.e <em>height</em> came from a <em>normally</em> distributed population.</p><p><strong>c. Kolmogorov–Smirnov(K-S) test</strong></p><p>K-S test provides a way to —</p><ul><li>check whether a sample is drawn from a reference probability distribution or not(<em>one-sample K–S test</em>)</li><li>check whether two samples are drawn from the same distribution or not(<em>two-sample K–S test</em>)</li></ul><p>It is a hypothesis based test where null and alternate hypothesis for one-sample <em>K–S test</em> is defined as below -</p><p><strong>H₀(Null Hypothesis): </strong>Sample follows the <em>reference</em> distribution</p><p><strong>H₁(Alternate Hypothesis):</strong> Sample does not follow the <em>reference </em>distribution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*IVrJX7A-eLPc_yt8PVVPGw.png" /></figure><p>Here for <strong>α</strong> = 0.05, obtained p-value(0.7958) &gt; <strong>α</strong> , so we failed to reject <em>null </em>hypothesis i.e <em>height</em> follows <em>normal</em> distribution.</p><h4><strong>Chebyshev inequality</strong></h4><p>By following 68–95–99.7 Empirical rule of a <em>normally</em> distributed dataset, we know that what % of data lies within <strong><em>k</em></strong> <em>standard deviation</em> from the <em>mean</em> but what if the data does not follow <em>normal</em> distribution? OR how to know what fraction of data lies within <strong><em>k </em></strong><em>standard deviation</em> from the <em>mean</em> for any random distribution?</p><p>To answer such questions pertaining to the dispersion of data for a random distribution <strong><em>Chebyshev inequality</em></strong> is <strong><em>used.</em></strong></p><p><strong><em>Chebyshev inequality</em></strong> states that no more than <strong><em>1/k²</em></strong> fraction of data falls more than <strong><em>k </em></strong><em>standard deviations</em> away from the <em>mean</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*H9Ftd2AkHRuVwe9osOrOeg.png" /></figure><p>In simplified terms</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*_ughcXZ02-oiRtWp2ZwjVw.png" /></figure><p>In other words, at least <strong><em>1- (1/k²)</em></strong><em> </em>fraction of data falls within <strong><em>k </em></strong><em>standard deviations</em> from the <em>mean</em> for a sample with finite <em>mean</em> and finite <em>standard deviation</em>.</p><p>Let’s explore the inequality more with a few values of <strong>k</strong>.</p><ul><li>For <strong><em>k</em></strong> = 2, <strong><em>1- (1/k²)</em></strong> = 0.75 i.e. at least 75% of the data falls within <em>two</em> standard deviations of the <em>mean</em> for any <em>random</em> distribution.</li><li>For <strong><em>k</em></strong> = 3, <strong><em>1- (1/k²)</em></strong> = 0.89 i.e. at least 89% of the data falls within <em>three</em> standard deviations of the <em>mean</em> for any <em>random</em> distribution.</li></ul><p>Let’s try to answer the following questions using <em>Chebyshev inequality</em>.</p><ul><li>What % of athletes have a height between 160.68 and 199.52 cm?</li><li>What % of athletes have a height between 150.97 and 209.23 cm?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*kn8Jf7791YIGM7Ya9Chdog.png" /></figure><h3>2. Log-Normal distribution</h3><p>A random variable X is said to be <em>log-normally</em> distributed if the natural logarithm of X is <em>normally</em> distributed. In other words, X <strong>~</strong>LogNormal(<strong>µ,σ</strong>) if log(X) is <em>normally</em> distributed.</p><p>PDF of a <em>log-normally</em> distributed random variable is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*UnmRsRGSD8a4Z5uOzbBk3Q.png" /></figure><p>Let’s plot the distribution of <em>ferr </em>feature.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F32%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F32.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0bb00361a54ab93a99c445db35a5828/href">https://medium.com/media/d0bb00361a54ab93a99c445db35a5828/href</a></iframe><p>We can observe that <em>ferr</em> feature follows some sort of <em>log-normal</em> distribution where the right portion has long tall than the left one.</p><p><strong>Comparing the distribution of ferr against log-normal distribution using QQ plot</strong></p><p><em>QQ plot</em> can be used to compare two <em>probability</em> distributions by plotting their quantiles against each other.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F34%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F34.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/b994a90831017c870bf1c6133bcbbbf3/href">https://medium.com/media/b994a90831017c870bf1c6133bcbbbf3/href</a></iframe><p>From the above <em>QQ plot,</em> we can observe that most of the points are not closely residing on the reference line so <em>ferr</em> feature does not strictly follow a <em>log-normal</em> distribution.</p><h3>3. Binomial distribution</h3><p>The binomial distribution is a discrete probability distribution for obtaining exactly <strong><em>k </em></strong>successes out of <strong><em>n </em></strong><em>Bernoulli trails</em>.</p><p>Characteristics of <em>Bernoulli trails</em> -</p><ul><li>Each trail has only two possible outcomes — success and failure.</li><li>The total number of trails is fixed.</li><li>The probability of success and failure remains the same throughout all the trails.</li><li>The trails are independent of each other.</li></ul><p>The binomial distribution is a way of calculating the probability of <strong><em>k </em></strong>successes from <strong><em>n </em></strong><em>Bernoulli trails</em>.</p><p>The PMF of a <em>binomial</em> random variate is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*_d0QH6PRgZPMSyKGhedBiA.png" /></figure><p>where <em>p</em> = probability of success and <em>(1-p)</em> = probability of failure</p><p><em>k</em> = number of successes and <em>(n-k)</em> = number of failures</p><p>From the underlying dataset, we can observe that only 12.37%(25/202) athletes play Basketball, Now if we choose a <em>random</em> sample of 50 athletes then</p><ul><li>What is the probability that exactly two athletes play basketball?</li><li>What is the probability that at most 10 athletes play basketball?</li><li>What is the probability that at least 20 athletes play basketball?</li></ul><p>Since all the above questions hold a varying number of successes(2,10,20) from a fixed number of trails(50) with <strong><em>p</em></strong> = 0.1237 so <em>binomial</em> distribution can be used to answer these questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*xCFJW45CFuUz2LU-_hpNUQ.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F36%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F36.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9add6972f7ef570e66089647b589c7b/href">https://medium.com/media/a9add6972f7ef570e66089647b589c7b/href</a></iframe><h3>4. Uniform distributions</h3><p><strong>4.a Discrete uniform distribution</strong></p><p>A <em>discrete uniform</em> distribution is a <em>symmetric</em> distribution with the following properties.</p><ul><li>It has a fixed number of outcomes.</li><li>All the outcomes are equally likely to occur.</li></ul><p>If a random variable <em>X</em> follows <em>discrete uniform</em> distribution and it has <em>k</em> discrete values say x₁, x₂, x₃………..x₁₀₀, then PMF of <em>X </em>is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/1*8LbDptmWIkFLFVbCmamN3g.png" /></figure><p>From the given dataset, we can observe that <em>sex</em> feature has two possible values <em>male</em> and <em>female</em>. There is an almost equal number of <em>male(100)</em> and <em>female(102)</em> athletes so if we assume that <em>sex</em> feature strictly follows uniform distribution then</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*aTin4_oOT_pCOao0IpHQGg.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F38%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F38.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc08ae2a14dd57ac7d43684abd157e03/href">https://medium.com/media/bc08ae2a14dd57ac7d43684abd157e03/href</a></iframe><p>So there is 50% chance that a random athlete will be <em>man</em>, similarly, there is 50% chance for a <em>random</em> athlete to be <em>female</em>.</p><p><strong>4.b Continuous uniform distribution</strong></p><p>If a continuous uniformly distributed random variable <em>X</em> is defined in <em>a</em> and <em>b</em> then PDF of <em>X</em> is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*iTZ14825GgBP8MTC9ZeG9A.png" /></figure><p>Let’s consider a subset of data having <em>wcc</em> values between 4.40 and 5.40 for one set and between 7.70 and 9.90 for another set. <em>The distribution</em> of this subset of data is as shown below where <em>probability</em> is same across all three bins of <em>continuous</em> range.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F40%2F&amp;src=https%3A%2F%2Fplot.ly%2F%7Evikasrajluhaniwal%2F40.embed%3Fautosize%3Dtrue&amp;type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=plot" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a57d52060b6e002650aaf8b81ea9f77/href">https://medium.com/media/3a57d52060b6e002650aaf8b81ea9f77/href</a></iframe><h3>End Notes</h3><p>In this journey so far, we discussed different kinds of probability distributions by giving special importance to the need for probability distribution in DS/ML context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4edf2057ecfe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-probability-distribution-is-must-in-ds-ml-4edf2057ecfe">Why Probability distribution is must in DS/ML —</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature selection using Wrapper methods in Python]]></title>
            <link>https://medium.com/data-science/feature-selection-using-wrapper-methods-in-python-f0d352b346f?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0d352b346f</guid>
            <category><![CDATA[forward-selection]]></category>
            <category><![CDATA[backward-elimination]]></category>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[wrapper-methods]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Fri, 04 Oct 2019 06:39:44 GMT</pubDate>
            <atom:updated>2020-10-24T10:12:41.835Z</atom:updated>
            <content:encoded><![CDATA[<p>Dear Reader,</p><p>The same article has been relocated to below link.</p><p><a href="https://www.analyticsvidhya.com/blog/2020/10/a-comprehensive-guide-to-feature-selection-using-wrapper-methods-in-python/">A comprehensive guide to Feature Selection using Wrapper methods in Python</a></p><p>Request you to follow above link. Apologise for the inconvenience caused.</p><p>Happy reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0d352b346f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/feature-selection-using-wrapper-methods-in-python-f0d352b346f">Feature selection using Wrapper methods in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing different types of activation functions in neural networks — which one to prefer?]]></title>
            <link>https://medium.com/data-science/analyzing-different-types-of-activation-functions-in-neural-networks-which-one-to-prefer-e11649256209?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e11649256209</guid>
            <category><![CDATA[leaky-relu]]></category>
            <category><![CDATA[relu]]></category>
            <category><![CDATA[tanh]]></category>
            <category><![CDATA[activation-functions]]></category>
            <category><![CDATA[sigmoid]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Tue, 07 May 2019 13:16:25 GMT</pubDate>
            <atom:updated>2021-03-27T14:29:51.478Z</atom:updated>
            <content:encoded><![CDATA[<h3>Analyzing different types of activation functions in neural networks — which one to prefer?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YI211tVqoRB414cjUcF7WQ.gif" /></figure><p>While building a <em>neural network</em>, one of the mandatory choices we need to make is which <em>activation function</em> to use. In fact, it is an unavoidable choice because <em>activation functions</em> are the foundations for a <em>neural network</em> to learn and approximate any kind of complex and continuous relationship between variables. It simply adds non-linearity to the network.</p><p>Any neuron (except input layer neurons) calculates the weighted sum of inputs, adds some bias and then applies <em>activation function</em> to it. Before talking more about <em>activation functions</em>, let’s see first why it is required.</p><blockquote><strong>Additional NOTE</strong></blockquote><blockquote>This article assumes that the reader has basic knowledge about the concept of the neural network, forward and backward propagation, weight initalization, optimization algorithms, etc. In case you are not familiar then I would recommend you to follow <strong>my other</strong> <strong>articles</strong> on these topics.</blockquote><blockquote><a href="https://towardsdatascience.com/forward-propagation-in-neural-networks-simplified-math-and-code-version-bbcfef6f9250">Forward propagation in neural networks — Simplified math and code version</a></blockquote><blockquote><a href="https://towardsdatascience.com/why-better-weight-initialization-is-important-in-neural-networks-ff9acf01026d">Why better weight initialization is important in neural networks?</a></blockquote><blockquote><a href="https://towardsdatascience.com/why-gradient-descent-isnt-enough-a-comprehensive-introduction-to-optimization-algorithms-in-59670fd5c096">Why Gradient descent isn’t enough: A comprehensive introduction to optimization algorithms in neural networks</a></blockquote><h3>Need for an activation function</h3><p>Refer to the below neural network with two hidden layers, input, and output layer. This network is suitable for a ternary class classification problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*xcBdSYRndl6dhouE1y0KHg.png" /></figure><p>Consider a case where no activation function is used in this network, then from the hidden layer 1 the calculated weighted sum of inputs will be directly passed to hidden layer 2 and it calculates a weighted sum of inputs and pass to the output layer and it calculates a weighted sum of inputs to produce the output. The output can be presented as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*9es-pAjxSJe3tN61B6p64A.png" /></figure><p>So the output is simply a linear transformation of weights and inputs and it is not adding any non-linearity to the network. Therefore, this network is similar to a <strong><em>linear regression model</em></strong> which can only address the linear relationship between variables i.e. a model with limited power and not suitable for complex problems like image classifications, object detections, language translations, etc.</p><p>So a neural network without an activation function</p><ol><li>can only represent a linear relationship between variables.</li><li>does not hold <em>Universal approximation theorem.</em></li></ol><h3>Popular activation functions</h3><p>There are many activation functions available. Since activation function is just a mathematical function so you can also come up with your own activation function. Commonly used activation functions are</p><ol><li>Logistic</li><li>tanh ( hyperbolic tangent)</li><li>ReLu (Rectified linear units)</li><li>Leaky ReLu</li></ol><h3>Logistic activation function</h3><p>It is a “S” shaped curve with equation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*2MoOSKaUQyj0_9Q-lnVkEA.png" /></figure><p>It ranges from 0 to 1. It is also referred as <em>Sigmoid activation</em> function. The weighted sum of inputs is applied to it as an input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*qRS650xg0-JrXJPUD_E32w.png" /></figure><p>For a large positive input, it results in a large positive output which tends to fire and for large negative input, it results in a large negative output which tends not to fire.</p><p>The derivative of an activation function helps in calculation during backpropagation. During backpropagation, the derivative of loss w.r.t parameter is calculated. The derivative of the logistic function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*IoBAUPDmZRUupH2afTEkjw.png" /></figure><p>Issues with the logistic activation function :</p><h4>1. Saturation problem</h4><p>A neuron is said to be saturated if it reaches to its peak value either maximum or minimum.</p><p>Saturation when <strong>f(x) = 0 or 1 </strong>then<strong> f’(x) = f(x)(1- f(x)) = 0</strong></p><p><strong>Why do we care about Saturation?</strong></p><p>Refer to the below neural network where let us assume weight w211 needs to be updated during backpropagation using gradient descent update rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*vPOALlwmwETibKpWVHhFkg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*PMbFUt0BGULx1pM3zS5avQ.png" /></figure><p>Now if h21 is found to be 1 then its derivative will be 0. So there is no any update in weight w211, this problem is known as <strong>vanishing gradient problem. </strong>The gradient of weight vanishes or goes down to zero.</p><p>So using logistic activation function, the saturated neuron may cause the gradient to vanish and therefore the network refuses to learn or keep learning at a very small rate.</p><h4>2. Not a zero-centered function</h4><p>A function having an equal mass on both the sides of zero line (x-axis) is known as a zero-centered function. Or in other words, in a zero-centered function, the output can be either negative or positive.</p><p>In the case of logistic activation function, the output is always positive and the output is always accumulated only towards one side (positive side) so it is not a zero-centered function.</p><p><strong>Why do we care about zero-centered functions?</strong></p><p>Let us assume weight w311 and w312 needs to be updated during backpropagation using gradient descent update rule.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*z4Fgv0nINF6zRDQOa7N56Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*MrcVmASsCo_PyUEW4fcwOg.png" /></figure><p><strong>h21</strong> and <strong>h22</strong> will be always positive due to the logistic activation function. The value of the gradient of w311and w312 can be positive or negative depending on the value of the common part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*pJj0pTQT1mbTLRkmQWE0ow.png" /></figure><p>So the gradient of all the weights connected to the same neuron is either positive or negative. Hence during update rule, these weights are only allowed to move in certain directions, not in all the possible directions. It makes the optimization harder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*_NwKFxAl3L4PMBKiIj-Jxg.png" /></figure><p>It is similar to a situation where you are only allowed to move left and forward, not allowed to move right and backward then it is very hard to reach the desired destination.</p><p>Having understood this, let us point down the <strong>issues</strong> associated with logistic activation function</p><ol><li>Saturated logistic neuron causes the gradient to <strong><em>vanish</em></strong>.</li><li>It is not a <strong><em>zero-centered</em></strong> function,</li><li>Because of e^x, it is highly<strong> <em>compute-intensive </em></strong>which makes convergence slower.</li></ol><h3>2. Tanh (Hyperbolic tangent)</h3><p>It is similar to <em>logistic activation</em> function with a mathematical equation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*ibDdAN-lHnSafuCG1EjP6g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*IrLb4Z_Mp-cbyCa6bBgsKg.png" /></figure><p>The output ranges from -1 to 1 and having an equal mass on both the sides of zero-axis so it is <strong><em>zero centered</em></strong> function. So <em>tanh </em>overcomes the non-zero centric issue of the <em>logistic</em> activation function. Hence optimization becomes comparatively easier than <em>logistic</em> and it is always preferred over <em>logistic</em>.</p><p>But Still, a <em>tanh</em> activated neuron may lead to <em>saturation </em>and cause <em>vanishing gradient</em> problem<em>.</em></p><p>The derivative of <em>tanh</em> activation function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*ZyQv9ma0lFipjC3vRwSRcw.png" /></figure><p><strong>Issues</strong> with <em>tanh</em> activation function:</p><ol><li>Saturated <em>tanh</em> neuron causes the gradient to <strong><em>vanish</em></strong>.</li><li>Because of e^x, it is highly<strong> <em>compute-intensive.</em></strong></li></ol><h3>3. ReLu (Rectified linear units)</h3><p>It is the most commonly used function because of its simplicity. It is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*jyyzxadG8Sbqcgfv08ttAw.png" /></figure><p>If the input is a positive number the function returns the number itself and if the input is a negative number then the function returns 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*E9Az5dBreEwvI5JmlG1dTA.png" /></figure><p>The derivative of <em>ReLu</em> activation function is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*vDV1QZKWD3MoS96Ht2GknA.png" /></figure><p><strong>Advantages of ReLu activation function</strong></p><ol><li>Easy to <strong><em>compute</em></strong>.</li><li>Does not <strong><em>saturate </em></strong>for the positive value of the weighted sum of inputs.</li></ol><p>Because of its simplicity,<em> ReLu</em> is used as a standard activation function in CNN.</p><p>But still, <em>ReLu</em> is not a zero-centered function.</p><p><strong>Issues with ReLu activation function</strong></p><p><em>ReLu</em> is defined as <em>max(0, w1x1 + w2x2 + …+b)</em></p><p>Now Consider a case <em>b(bias</em>) takes on (or initialized to) a large negative value then the <em>weighted sum of inputs</em> is close to 0 and the <em>neuron</em> is not activated. That means the <em>ReLu</em> activation neuron <strong><em>dies</em></strong> now. Like this, <strong><em>up to 50%</em></strong> of <em>ReLu</em> activated neurons may <strong>die</strong> during the training phase.</p><p>To <strong>overcome</strong> this problem, two solutions can be proposed</p><ol><li>Initialize the <em>bias(b) </em>to a large positive value.</li><li>Use another variant of ReLu known as <strong>Leaky ReLu</strong>.</li></ol><h3><strong>4. Leaky ReLu</strong></h3><p>It was proposed to fix the <strong><em>dying neurons </em></strong>problem of <em>ReLu</em>. It introduces a small slope to keep the update alive for the neurons where the <em>weighted sum of inputs</em> is <strong><em>negative</em></strong>. It is defined as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*sy8LauNPCdU6ycPFuqhKag.png" /></figure><p>If the input is a <em>positive</em> number the function returns the number itself and if the input is a <em>negative </em>number then it returns a <em>negative</em> value scaled by 0.01(or any other small value).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*2U9o7ma_pp4rkalyMs2oXg.png" /><figcaption>LeakyReLu curve</figcaption></figure><p>The derivative of <em>LeakyReLu</em> is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*nyX9El4tLjP2XdtaYsyxJQ.png" /></figure><p><strong>Advantages of LeakyReLu</strong></p><ol><li>No <strong><em>saturation</em></strong> problem in both positive and negative region</li><li>The neurons <strong><em>do not die</em></strong> because <strong>0.01x</strong> ensures that at least a small gradient will flow through. Although the change in weight will be small but after a few iterations it may come out from its original value.</li><li>Easy to <strong><em>compute</em></strong>.</li><li>Close to<strong><em> zero-centered</em></strong> functions.</li></ol><h3>Softmax activation function</h3><p>For a <em>classification</em> problem, the output needs to be a probability distribution containing different probability values for different classes. For a binary classification problem, the <em>logistic</em> activation function works well but not for a <em>multiclass</em> classification problem. So <strong><em>Softmax</em></strong> is used for <em>multiclass </em>classification problem.</p><p>The <em>softmax</em> activation function is again a type of s<em>igmoid</em> function. As the name suggests, it is a <em>“soft”</em> flavor of the <em>max</em> function where instead of selecting only one maximum value, it assigns the maximal element largest portion of the distribution, and other smaller elements getting some part of the distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*aRQ7yg0Fxd9-V-Tu_CLTjA.png" /><figcaption>Softmax equation</figcaption></figure><p><em>Softmax</em> is generally preferred in the output layer where we are trying to get probabilities for different classes in the output. It is applied to the obtained weighted sum of inputs in the output layer.</p><p><strong>Need for the Softmax activation function</strong></p><p>Consider a case where for a ternary classification problem we have preactivation outputs as [2.4, -0.6, 1.2] in the output layer. Preactivation output can be any real value.</p><p>Now if we directly calculate the probability for each class by taking one vale and dividing it by sum of all the values. Then we get probabilities [0.57, -0.14, 0.12]. Since probability can never be negative so here the predicted probability for class 2 is not accepted. Therefore we use the <em>softmax </em>activation function in the output layer for multiclass classification problem.</p><p>If the input value is negative then also <em>softmax</em> returns positive value because e^x always gives positive values.</p><h3>End Notes: Now which one to prefer?</h3><p>Having understood all the activation functions, now the interesting question is which one to use?</p><p>Well, there is no hard and fast rule for scenario-based activation function selection. The decision completely relies on the properties of the problem i.e the relationship you are trying to approximate possesses some properties and based on these properties you can try with different activation functions and select the one which helps in faster convergence and faster learning process or may be according to different evaluation parameters of your choice.</p><ul><li>As a rule of thumb, you can start with <strong>ReLu</strong> as a general approximator and switch to other functions if <strong>ReLu</strong> doesn&#39;t provide better results.</li><li>For <strong>CNN</strong>, <strong>ReLu</strong> is treated as a standard activation function but if it suffers from <strong>dead neurons</strong> then switch to <strong>LeakyReLu</strong>.</li><li>Always remember <strong>ReLu</strong> should be only used in <strong><em>hidden</em></strong> layers.</li><li>For <strong>classification</strong>, <strong>Sigmoid functions(Logistic, tanh, Softmax) </strong>and their combinations work well. But at the same time, it may suffer from <strong>vanishing gradient problem</strong>.</li><li>For <strong>RNN,</strong> the <strong>tanh</strong> activation function is preferred as a standard activation function.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e11649256209" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analyzing-different-types-of-activation-functions-in-neural-networks-which-one-to-prefer-e11649256209">Analyzing different types of activation functions in neural networks — which one to prefer?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why better weight initialization is important in neural networks?]]></title>
            <link>https://medium.com/data-science/why-better-weight-initialization-is-important-in-neural-networks-ff9acf01026d?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff9acf01026d</guid>
            <category><![CDATA[xavier-initialization]]></category>
            <category><![CDATA[random-initialization]]></category>
            <category><![CDATA[zero-initialization]]></category>
            <category><![CDATA[weight-initialization]]></category>
            <category><![CDATA[he-initialization]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Tue, 07 May 2019 13:15:50 GMT</pubDate>
            <atom:updated>2021-03-27T14:30:00.727Z</atom:updated>
            <content:encoded><![CDATA[<p>At the beginning of my deep learning journey, I always underrated weight initialization. I believed weights should be initialized to random values without knowing answers to the questions like why random initialization? why not any other approach? How much powerful or meaningful weight initialization should be? etc.</p><p>These series of questions inspired me to write a blog where I will cover different weight initialization techniques, pros and cons for each, need for individual techniques, etc. in a simpler way.</p><blockquote><strong>Additional NOTE</strong></blockquote><blockquote>This article assumes that the reader has basic knowledge about the concept of the neural network, forward and backward propagation, activation functions, optimization algorithms, etc. In case you are not familiar then I would recommend you to follow <strong>my other</strong> <strong>articles</strong> on these topics.</blockquote><blockquote><a href="https://towardsdatascience.com/forward-propagation-in-neural-networks-simplified-math-and-code-version-bbcfef6f9250">Forward propagation in neural networks — Simplified math and code version</a></blockquote><blockquote><a href="https://towardsdatascience.com/analyzing-different-types-of-activation-functions-in-neural-networks-which-one-to-prefer-e11649256209">Analyzing different types of activation functions in neural networks — which one to prefer?</a></blockquote><blockquote><a href="https://towardsdatascience.com/why-gradient-descent-isnt-enough-a-comprehensive-introduction-to-optimization-algorithms-in-59670fd5c096">Why Gradient descent isn’t enough: A comprehensive introduction to optimization algorithms in neural networks</a></blockquote><p>Before proceeding further first let us see the parameter notations used throughout the article.</p><h4><strong>Terminology used</strong></h4><p>Consider an <em>L</em> <em>layer</em> network with <em>L-2</em>(excluding input and output layers) <em>hidden layers</em>. The <em>parameters</em> for an arbitrary layer <em>l</em> is represented as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*8gaXoS1YksVvRU8Ii8isaw.png" /></figure><p>To get a neural network with better and optimal results, weight initialization is the first step which comes into consideration. A network with improper weight initialization can make the entire learning process tedious and time-consuming. Therefore, to achieve better optimization, faster convergence, and feasible learning process weight initialization is very crucial. Now let us start with some question w.r.t. weight initialization.</p><h3>Why not simply initialize all the weights to zero?</h3><p>Consider a scenario where all the <em>weights</em> are initialized to <strong>0</strong>. Refer to the below network designed for <em>multi-class classification</em> problem. The dataset is generated using make_blobs()function from sklearn.datasets. The dataset has four different classes of two different features. The network has one <em>hidden</em> and<em> input</em> layer with two neurons,<em> an output </em>layer with four neurons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*kOPmEdmhjhhkn9AlfE3Jig.png" /></figure><p>where W1 is a 2x2 and W2 is a 4x2 matrix. B1 and B2 are column vectors with size 2 and 4 respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*5GMeqCRraYxKbI3lBCTP-w.png" /></figure><p>During <em>preactivation</em> at hidden layer</p><p>a11 = w1x1 + w2x2 + b1</p><p>a12 = w3x1 + w4x2 + b2</p><p>a11 = a12 (assume b1 = b2) then h11 = h12</p><p>Similarly a21 = a22 = a23 =a24 then h21 = h22 = h23 =h24</p><p>During <em>back propagation</em> of w1 and w3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*8ilmX0cYkaFG8IUS_DniyQ.png" /></figure><p>Note - <em>here the gradients of w1 and w3 shown are calculated by following single path only, in fact, these are computed by considering all the derivatives across multiple possible paths.</em></p><p>If the gradients are equal then weights are going to be updated by the same amount. The weights attached to the same neuron, continue to remain the same throughout the training. It makes the hidden units symmetric and this problem is known as the <strong><em>symmetry problem</em></strong>.</p><p>Hence to break this <strong>symmetry</strong> the weights connected to the same neuron should not be initialized to the same value.</p><p><strong>Key points</strong></p><ul><li>Never initialize all the weights to <strong>zero</strong>.</li><li>Never initialize all the weights to the <strong>same </strong>value.</li></ul><h3>Can we initialize weights to large values?</h3><p>If the weights are initialized to large values and if sigmoid activation function(logistic, tanh) is used then <em>saturation problem</em> can occur which leads to <em>vanishing gradient problem. </em>As a result of this the gradient changes slowly and the learning becomes tedious. Similarly <em>exploding gradient problem</em> can occur.</p><p><strong>Key Points</strong></p><ul><li>Never initialize weights to <strong><em>large</em></strong> values.</li><li>It is better to <strong><em>normalize</em></strong><em>/</em><strong><em>standardize the</em></strong> inputs so that they lie on a small common scale.</li></ul><h3>What works in practice?</h3><p>Having known the issues with zero initialization and high-value initialization, what can be other feasible initializations?</p><p>Well, we can initialize weights randomly following a particular kind of distribution(uniform, normal, truncated normal, etc).</p><p>For example, I can use np.random.randn(size_l, size_l-1) in Python for random initialization of weights following a standard normal distribution with mean 0 and standard deviation 1.</p><p><strong>But for the deeper and wider networks with a large number of neurons in hidden layers and a large number of hidden layers, the random initialization may cause problems.</strong></p><p>For example, if there are <strong><em>n</em></strong> neurons in a hidden layer and if <strong><em>n </em></strong>is a very large number then during preactivation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*EDdz_ODzFQF5Yr5HCXIafA.png" /></figure><p>In a21 all these weighted sums will blow up because of the large number of neurons. Since a21 value is high then again saturation problem can occur which leads to vanishing gradient problem. Similarly <em>exploding gradient problem</em> can occur.</p><p><strong>How to overcome this?</strong></p><p>To deal with these <em>gradient</em> problems, we can initialize random weights in a suitable distribution fashion. Instead of using a standard normal distribution, use a normal distribution with k/n variance where k is selected based on the activation function.</p><p><em>Xavier Glorot</em> and <em>He et al.</em> are the first contributors to this concept of better random weights initialization.</p><h3>Xavier initialization for tanh and logistic activation function</h3><p><em>Xavier</em> proposed a better random weight initialization approach which also includes the size of the network (number of input and output neurons) while initializing weights.</p><p>According to this approach, the weights should be inversely proportional to the square root of the number of neurons in the previous layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*ZLJiB88aR0zjgbt7kc5RVQ.png" /></figure><p>where weights can be initialized in Python as np.random.randn(sizes[i-1],sizes[i])*np.sqrt(1/sizes[i-1])</p><p>Following this approach, the weighted sum of inputs at a hidden layer will not take a large value and reduces the chances of the <em>vanishing</em>/<em>exploding gradient problem.</em></p><h3>He initialization for ReLu and leaky ReLu activation function</h3><p><strong><em>He </em></strong>initialization is similar to <strong><em>Xavier </em></strong>initialization where the number of neurons in the previous layer is given importance. But the factor is multiplied by <strong><em>two</em></strong> instead of one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/181/1*r1xULrvfItQ_Pt7RAH226A.png" /></figure><p>where weights can be initialized in Python as np.random.randn(sizes[i-1],sizes[i])*np.sqrt(2/sizes[i-1])</p><p><strong><em>Dying neuron</em></strong> is a common problem with <em>ReLu activation</em> function. With ReLu activation function, <strong><em>up to 50%</em></strong> of the neurons may <strong><em>die</em></strong> during training. To deal with this the factor is multiplied by <strong>two</strong> in <em>He initialization</em>.</p><h4>End Notes</h4><p>In this journey so far we have discussed various <strong><em>weight initialization</em></strong> methods and the issues associated with each of them. It is recommended to use <strong><em>Xavier initialization</em></strong> for <strong><em>sigmoid based </em></strong><em>activation functions</em> and<strong><em> He initialization</em></strong> for <strong><em>ReLu and Leaky ReLu</em></strong> <em>activation functions</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff9acf01026d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-better-weight-initialization-is-important-in-neural-networks-ff9acf01026d">Why better weight initialization is important in neural networks?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Gradient descent isn’t enough: A comprehensive introduction to optimization algorithms in…]]></title>
            <link>https://medium.com/data-science/why-gradient-descent-isnt-enough-a-comprehensive-introduction-to-optimization-algorithms-in-59670fd5c096?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/59670fd5c096</guid>
            <category><![CDATA[adam]]></category>
            <category><![CDATA[optimization-algorithms]]></category>
            <category><![CDATA[adagrad]]></category>
            <category><![CDATA[stochastic-gradient]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Tue, 07 May 2019 13:14:51 GMT</pubDate>
            <atom:updated>2023-05-19T05:42:05.636Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Gradient descent isn’t enough: A comprehensive introduction to optimization algorithms in neural networks</h3><p>The goal of neural networks is to minimize the <strong><em>loss, </em></strong>for producing better and accurate results<strong><em>. </em></strong>In order to minimize the loss, we need to update the internal learning parameters(especially <strong><em>weights</em></strong> and <strong><em>biases</em></strong>). These parameters are updated based on some <em>update rule/function</em>. Generally, we think about <strong><em>Gradient descent</em></strong> as an update rule. Now two types of questions arise w.r.t parameters update.</p><ul><li><strong>How much/what data should be used for an update?</strong></li><li><strong>What update rule should be used?</strong></li></ul><p>This post revolves around these two questions and answers in the simplest way in the context of better optimization. In this post, I will present an intuitive vision of optimization algorithms, their different types, and variants.</p><blockquote><strong>Additional NOTE</strong></blockquote><blockquote>This article assumes that the reader has basic knowledge about the concept of the neural network, forward and backward propagation, weight initalization, activation functions, etc. In case you are not familiar then I would recommend you to follow <strong>my other</strong> <strong>articles</strong> on these topics.</blockquote><blockquote><a href="https://towardsdatascience.com/forward-propagation-in-neural-networks-simplified-math-and-code-version-bbcfef6f9250">Forward propagation in neural networks — Simplified math and code version</a></blockquote><blockquote><a href="https://towardsdatascience.com/why-better-weight-initialization-is-important-in-neural-networks-ff9acf01026d">Why better weight initialization is important in neural networks?</a></blockquote><blockquote><a href="https://towardsdatascience.com/analyzing-different-types-of-activation-functions-in-neural-networks-which-one-to-prefer-e11649256209">Analyzing different types of activation functions in neural networks — which one to prefer?</a></blockquote><h3>Optimization algorithms</h3><p><strong>Optimization algorithm</strong> tries to minimize the <strong>loss(cost)</strong> by following some update rule. The loss is a <em>mathematical</em> function denoting the difference between the <em>predicted value</em> and <em>actual value</em>. Loss is dependent on the predicted value which is derived with the help of <strong><em>learning parameters(weights </em></strong>and<strong><em> biases) </em></strong>and <em>inputs</em>. Therefore learning <em>parameters</em> are very important for better training and producing accurate results. To find out the optimal value of these parameters we need to continuously <em>update</em> them. There should be some update rule for this purpose. So we use various <strong><em>optimization algorithms </em></strong>to follow some update rule and each <em>optimization algorithm</em> has a different approach to calculate, update and find out the optimal value of model parameters.</p><h3>Types of optimization algorithms</h3><p>Based on our first question <strong>“How much data should be used for an update” </strong>optimization algorithms can be classified as <strong><em>Gradient Descent</em></strong>, <strong><em>Mini batch Gradient Descent</em></strong>, and <strong><em>Stochastic Gradient Descent.</em></strong></p><p>In fact, the basic algorithm is <em>Gradient Descent. Mini-batch Gradient descent </em>and <em>Stochastic Gradient Descent</em><strong><em> </em></strong>are two different strategies based on the amount of the data taken. These two are also known as the variants of <em>Gradient Descent.</em></p><h3><em>Gradient Descent</em></h3><p><strong><em>Gradient descent</em></strong> is most commonly used and popular<strong><em> iterative </em></strong>machine learning algorithm. It is also the foundation for other optimization algorithms. <em>Gradient descent</em> has the following <em>update rule</em> for <em>weight</em> parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*faTF4_-a7qMAzw3y27wzOw.png" /></figure><p>Since during <em>backpropagation</em> for updating the parameters, the <em>derivative of loss</em> w.r.t. a parameter is calculated. This <em>derivative</em> can be dependent on more than one variable so for its calculation <strong><em>multiplication chain rule</em></strong> is used. For this purpose, a <strong><em>Gradient</em></strong> is required. A <strong><em>gradient</em></strong> is a vector indicating the direction of increase.</p><p><strong>For gradient calculation, we need to calculate <em>derivatives of loss</em> w.r.t the <em>parameters</em> and update the <em>parameters</em> in the opposite direction of the <em>gradient.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*jYQYuHpHdkZqNFQKJSuDTw.png" /></figure><p>The above ideal convex curve image displays the <em>weight update</em> in the opposite direction of the <em>gradient</em>. As we can notice for too large and small values of <em>weights</em> the <em>loss</em> is maximum and our goal is to <em>minimize</em> the <em>loss</em> so the <em>weights </em>are updated. If the <em>gradient</em> is negative then <strong><em>descent</em></strong>(dive) towards the positive side and if the <em>gradient</em> is positive then descent towards the negative side until the minimal value of <em>gradient</em> is found.</p><p>Algorithm for <strong><em>Gradient descent</em></strong> using a single neuron with <em>sigmoid</em> activation function in Python</p><pre>def sigmoid(w,b,x):<br>    return 1.0 / (1.0 + np.exp(-w*x + b))</pre><pre>def grad_w(w,b,x,y):<br>    fx = sigmoid(w,b,x)<br>    return (fx — y) * fx * (1-fx) * x</pre><pre>def grad_b(w,b,x,y):<br>    fx = sigmoid(w,b,x)<br>    return (fx — y) * fx * (1-fx)</pre><pre>def do_gradient_descent():<br>    w,b,eta = -3, 4, 1.0<br>    max_epochs = 1000<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        for x,y in zip(X,Y):<br>            dw += grad_w(w,b,x,y)<br>            db += grad_b(w,b,x,y)<br>        w = w — eta * dw<br>        b = b — eta * db</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*nuoD-tQylhMj-SF8WJfUjg.gif" /></figure><p>The above <em>animation</em> represents how the algorithm converges after 1000 <em>epochs</em>. The error surface used in this <em>animation</em> is as per the input. This error surface is animated in 2D space. For 2D, a <em>contour map</em> is used where the contours represent the third dimension i.e. <strong><em>error</em></strong>. The red regions represent the high values of error, more the intensity of the red region more the error. Similarly, the blue regions represent the low values of error, less the intensity of the blue region less the error.</p><p><strong><em>Standard Gradient descent</em></strong> updates the <em>parameters</em> only after each epoch i.e. after calculating the <em>derivatives</em> for all the observations it updates the <em>parameters</em>. This phenomenon may lead to the following <strong><em>caveats</em></strong>.</p><ul><li>It can be very slow for very large datasets because only one-time <em>update</em> for each <em>epoch</em> so large number of <strong><em>epochs</em></strong> is required to have a substantial number of <em>updates</em>.</li><li>For large datasets, the vectorization of data doesn’t fit into <strong><em>memory</em></strong>.</li><li>For <em>non-convex</em> surfaces, it may only find the <strong><em>local minimums.</em></strong></li></ul><p>Now let see how <strong><em>different variations of gradient descent</em></strong> can address these challenges.</p><h3>Stochastic gradient descent</h3><p><strong><em>Stochastic gradient descent</em></strong> updates the <em>parameters</em> for <em>each observation </em>which leads to more number of updates<em>. </em>So it is a faster approach which helps in quicker decision making.</p><p>Algorithm for <em>Stochastic</em> <em>Gradient descent</em> using a single neuron with <em>sigmoid</em> activation function in Python</p><pre>def do_stochastic_gradient_descent():<br>    w,b,eta = 2.1, 4, 1.0<br>    max_epochs = 1000<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        for x,y in zip(X,Y):<br>            dw += grad_w(w,b,x,y)<br>            db += grad_b(w,b,x,y)<br>            w = w — eta * dw<br>            b = b — eta * db<br>            dw,db = 0,0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*eqbCPD7Yx_nRchmP6YtjYA.gif" /></figure><p><strong><em>Quicker updates</em></strong> in different directions can be noticed in this animation. Here, lots of oscillations take place which causes the <em>updates</em> with <strong><em>higher variance</em></strong> i.e. <strong><em>noisy</em></strong> <strong><em>updates</em></strong>. These noisy updates help in finding <strong><em>new </em></strong>and <strong><em>better local minima</em></strong>.</p><p><strong>Disadvantages of SGD</strong></p><ul><li>Because of the <strong><em>greedy approach</em></strong>, it only <strong><em>approximates (stochastics)</em></strong> the gradient.</li><li>Due to <strong><em>frequent fluctuations</em></strong>, it will keep <strong><em>overshooting</em></strong> near to the desired <strong><em>exact minima</em></strong>.</li></ul><p>Now let see how another <strong><em>variant of gradient descent</em></strong> can address these challenges.</p><h3>Mini Batch Gradient Descent</h3><p>Another variant of <strong><em>GD </em></strong>to address the problems of <strong><em>SGD, </em></strong>it lies in between<strong><em> GD </em></strong>and<strong><em> SGD.</em></strong> <strong><em>Mini-batch Gradient descent</em></strong> updates the parameters for a finite number of observations. These observations together are referred to a <strong>batch</strong> with some fixed size. <strong>Batch size</strong> is chosen as a multiple of 64 e.g. 64, 128, 256, etc. Many more updates take place in one epoch through <em>Mini-batch GD</em>.</p><p>Algorithm for <em>Mini-batch</em> <em>Gradient descent</em> using a single neuron with <em>sigmoid</em> activation function in Python</p><pre>def do_mini_batch_gradient_descent():<br>    w,b,eta = 2.1, 4, 1.0<br>    max_epochs = 1000<br>    mini_batch_size = 3<br>    num_of_points_seen = 0<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        for x,y in zip(X,Y):<br>            dw += grad_w(w,b,x,y)<br>            db += grad_b(w,b,x,y)<br>            num_of_points_seen += 1<br>        if num_of_points_seen % mini_batch_size == 0:<br>            w = w — eta * dw<br>            b = b — eta * db</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*b9hLP6yfXtlZzfAOAAe7-A.gif" /></figure><p>As we can see there are fewer oscillations in <strong><em>Mini-batch</em></strong> in contrast to<strong> <em>SGD</em></strong>.</p><p><strong>Basic notations</strong></p><p><strong>1 epoch</strong> = one pass over the entire data</p><p><strong>1 step</strong> = one update for parameters</p><p><strong>N </strong>= number of data points</p><p><strong>B </strong>= Mini-batch size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*XKVDn-pHsGpX8Jh1wMZloQ.png" /></figure><p><strong>Advantages of Mini-batch GD</strong></p><ul><li>Updates are less noisy compared to SGD which leads to better convergence.</li><li>A high number of updates in a single epoch compared to GD so less number of epochs are required for large datasets.</li><li>Fits very well to the processor memory which makes computing faster.</li></ul><h3>Better optimization w.r.t. Gradient Descent</h3><p>The <strong><em>error surface</em></strong> contains <em>more sloppy</em> as well <em>less sloppy</em> areas. During <em>back propagation</em>, there will be more update in parameters for the regions with <em>more slope</em> whereas less update in parameters for the regions with a <em>gentle slope</em>. More change in parameters leads to more change in loss, similarly less change in parameters leads to less change in the loss.</p><p>If the parameter initialization lands in a <em>gentle slope</em> area then it requires a large number of <em>epochs</em> to navigate through these areas. It happens so because the <em>gradient </em>will be very small in <em>gentle slope</em> regions. So it moves with <strong><em>small baby steps</em></strong> in <em>gentle</em> regions.</p><p>Consider a case with initialization in a flat surface as shown below where<strong><em> GD </em></strong>is used and the<strong><em> error</em></strong> is not reducing when the <strong><em>gradient</em></strong> is in the <strong><em>flat surface</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ABZ3ZzHd63FdAWqlJLNErw.gif" /></figure><p>Even after a large number of <em>epochs</em> for e.g. 10000 the algorithm is <strong><em>not converging</em></strong>.</p><p>Due to this issue, the <em>convergence</em> is not achieved so easily and the learning takes too much time.</p><p>To overcome this problem <strong><em>Momentum based gradient descent</em></strong> is used.</p><h3><strong><em>Momentum-based gradient descent</em></strong></h3><p>Consider a case where in order to reach to your desired destination you are continuously being asked to follow the same direction and once you become confident that you are following the right direction then you start taking <strong><em>bigger steps</em></strong> and you keep getting <strong><em>momentum</em></strong> in that same direction.</p><p>Similar to this if the <strong><em>gradient</em></strong> is in a <strong><em>flat surface</em></strong> for long term then rather than taking constant steps it should take <strong><em>bigger steps</em></strong> and keep the <strong><em>momentum</em></strong> continue. This approach is known as <strong><em>momentum based gradient descent</em></strong>.</p><p><strong>Momentum-based gradient descent update rule</strong> for weight parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*w3A9ZNZowdjjUDzYpsqtGQ.png" /></figure><p><strong><em>Gamma parameter(</em>γ)</strong> is the momentum term which indicates how much acceleration you want. Here along with the <strong><em>current gradient (</em>η∇w(t)), </strong>the movement is also done according to history <strong>(γV(t−1)) </strong>so the <strong><em>update</em></strong> becomes larger which leads to faster movement and faster <strong><em>convergence</em></strong>.</p><p><strong>v(t) </strong>is <strong><em>exponentially decaying weighted sum</em></strong>, as <strong>t</strong> increases <strong>γV(t−1) </strong>becomes smaller and smaller i.e. this equation holds the farther updates by a small magnitude and recent updates by a large magnitude.</p><p><strong>Momentum-based gradient descent </strong>in Python for sigmoid neuron</p><pre>def do_momentum_based_gradient_descent():<br>    w,b,eta,max_epochs = -4, 0, 1.0, 1000<br>    v_w, v_b = 0, 0<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        for x,y in zip(X,Y):<br>            dw += grad_w(w,b,x,y)<br>            db += grad_b(w,b,x,y)<br>        v_w = gamma * v_w + eta * dw<br>        v_b = gamma * v_b + eta * db<br>        w = w — v_w<br>        b = b — v_b</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*MxPOjaJDWZeE2zNUZxpdNQ.gif" /></figure><p>This algorithm <strong><em>adds momentum</em></strong> in the direction of <strong><em>consistent gradients</em></strong> and <strong><em>cancels the momentum </em></strong>if the <strong><em>gradients</em></strong> are in <strong><em>different directions</em></strong>.</p><p><strong>Issues with momentum based Gradient descent</strong></p><p>In the <strong><em>valley</em></strong> that leads to exact <strong><em>desired minima</em></strong>, there are a large number of <em>oscillations </em>using <em>momentum-based GD</em>. Because it <strong><em>overshoots</em></strong> the minima with <strong><em>larger steps</em></strong> and takes a <strong><em>U-turn</em></strong> but again overshoots so this process repeats. Which means moving with larger steps is not always good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*nBI9KeSoepPGNNf25ANaJg.gif" /></figure><ul><li><strong><em>Momentum-based GD</em></strong> <em>oscillates</em> for a <strong><em>large number of times</em></strong> in and out of the <strong><em>minima</em></strong>.</li></ul><p>To overcome this issue <strong><em>Nesterov accelerated Gradient Descent</em></strong> is used.</p><h3>Nesterov accelerated Gradient Descent</h3><p>In <em>momentum based GD </em>as the <em>gradient</em> heads to the valley(minima region), it makes a lot of <strong><em>U-turns(oscillations) </em></strong>before it<strong><em> </em></strong><em>converges. </em>This problem was initially identified and responded by a researcher named Yurii Nesterov.</p><p>He suggested, make the <em>movement</em> first by <strong><em>history amount(previous momentum)</em></strong> then calculate the <strong><em>temporary gradient</em></strong> at this point and then <em>update</em> the <em>parameters</em>. In other words, before making an <em>update</em> directly first it looks ahead by moving with the <em>previous momentum</em> then it finds what the <em>gradient</em> should be.</p><p>This <em>looking ahead</em> helps <strong><em>NAG </em></strong>in finishing its job(finding the minima) quicker than<strong><em> momentum-based GD. </em></strong>Hence the <strong><em>oscillations </em></strong>are <strong><em>less</em></strong> compared to <strong><em>momentum based GD </em></strong>and also there are fewer chances of missing the <strong><em>minima.</em></strong></p><p><strong>NAG update rule </strong>for <em>weight</em> parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*aWTX8EOu10OkJzRzaR2P2A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*SV3x2pD4lOGvaWmyMiSgPQ.png" /></figure><p><strong>NAG</strong> algorithm in Python for sigmoid neuron</p><pre>def do_nag_gradient_descent():<br>    w,b,eta,max_epochs = 2.1, 4, 1.0, 1000<br>    v_w, v_b, gamma = 0, 0, 0.9<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        #compute the look ahead value<br>        w = w — gamma * v_w<br>        b = b — gamma * v_b<br> <br>       for x,y in zip(X,Y):<br>           #compute the derivatives using look ahead value<br>           dw += grad_w(w,b,x,y)<br>           db += grad_b(w,b,x,y)<br>       #Now move further in the opposite direction of that gradient<br>       w = w — eta * dw<br>       b = b — eta * db<br> <br>      #Now update the previous momentum<br>      v_w = gamma * v_w + eta * dw<br>      v_b = gamma * v_b + eta * db</pre><p>Here v_w and v_b refer to <strong>v(t) </strong>and<strong> v(b) </strong>respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*5G4Adw4fy70IzVOuCjfu5w.gif" /></figure><h3>Concept of Adaptive learning rate</h3><p>As per the update rule</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*jPYlnZJbprDsunEQX8D0vg.png" /></figure><p>The <em>update</em> is directly proportional to the <strong><em>gradient</em></strong>(<strong>∇w). </strong>Smaller the <em>gradient</em> smaller the <em>update</em><strong> </strong>and the <em>gradient </em>is directly proportional to the <strong><em>input</em></strong>. Therefore the <em>update</em> is dependent on the <em>input</em> also.</p><p><strong>Need for an adaptive learning rate</strong></p><p>For the real-time datasets, most of the features are <strong><em>sparse</em></strong> i.e. having zero values. Due to this for most of the cases, the corresponding <em>gradient</em> is zero and therefore the parameters <em>update</em> is also zero. To resonate this problem, these update should be boosted i.e. a <strong><em>high learning rate</em></strong> for <em>sparse</em><strong><em> </em></strong>features. Therefore the <em>learning rate</em> should be<strong><em> adaptive</em></strong> for fairly <em>sparse</em><strong><em> </em></strong>data.</p><p>In other words, if we are dealing with <strong><em>sparse features</em></strong> then <em>learning rate </em>should be <em>high</em> whereas for<em> </em><strong><em>dense features</em></strong><em> learning rate</em> should be <em>low</em>.</p><p><strong><em>Adagrad, RMSProp, Adam</em></strong> algorithms are based on the concept of <em>adaptive learning rate</em>.</p><h3>Adagrad</h3><p>It adopts the <em>learning rate(</em><strong>η)</strong><em> </em>based on the <strong><em>sparsity</em></strong> of features. So the parameters with <strong>small <em>updates(sparse features) </em></strong>have high <em>learning rate</em> whereas the parameters with <strong>large <em>updates(dense features) </em></strong>have low <em>learning rate</em>.<em> </em>Therefore <strong>adagrad</strong> uses a different <em>learning rate</em> for each <em>parameter.</em></p><p><strong>Adagrad</strong> update rule for <em>weight</em> parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*iXwh6i5NLsyigdIwMQg10Q.png" /></figure><p><strong>v(t) </strong>accumulates the running sum of square of the gradients. Square of <strong>∇w(t) </strong>neglects the sign of gradients. <strong>v(t) </strong>indicates accumulated gradient up to time <strong>t</strong>. <strong><em>Epsilon</em></strong> in the denominator avoids the chances of <strong><em>divide by zero error</em></strong>.</p><p>So if <strong>v(t) </strong>is<strong> low</strong> (due to less <em>update</em> up to time<strong> t</strong>) for a <em>parameter</em> then the effective <em>learning rate</em> will be <strong>high</strong> and if <strong>v(t) </strong>is<strong> </strong>high for a <em>parameter </em>then effective <em>learning rate</em> will be <strong>less</strong>.</p><p><strong>Adagrad</strong> algorithm in Python for sigmoid neuron</p><pre>def do_adagrad():<br>    w,b,eta,max_epochs = -6, 4, 0.5, 200<br>    v_w, v_b = 0, 0<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        for x,y in zip(X,Y):<br>            dw += grad_w(w,b,x,y)<br>            db += grad_b(w,b,x,y)<br>        v_w += dw**2<br>        v_b += db**2<br>        self.w -= (eta / np.sqrt(v_w) + eps) * dw<br>        self.b -= (eta / np.sqrt(v_b) + eps) * db</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*b2AOmFFikbWDbr7ExMBgYQ.gif" /></figure><p><strong>Disadvantage with Adagrad</strong></p><ul><li>The learning rate decays very aggressively</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Ewo2vS0XfcbPYD3-UwalBw.gif" /></figure><p>For <em>parameters</em>(especially <em>bias</em>) corresponding to <em>dense features</em>, after a few <em>updates</em>, the <em>learning rate</em> <strong><em>decays</em></strong> rapidly as the denominator grows rapidly due to the accumulation of <em>squared gradients</em>. So after a finite number of <em>updates</em>, the algorithm refuses to learn and <em>converges slowly</em> even if we run it for a large number of <em>epochs</em>. The <em>gradient</em> reaches to a bad minimum (close to desired <em>minima)</em> but not at exact <em>minima</em>. So <strong>adagrad</strong> results in decaying and decreasing learning rate for <em>bias</em> parameters.</p><h3>RMSProp</h3><p><strong><em>RMSProp</em></strong> overcomes the <strong><em>decaying learning rate</em></strong> problem of <strong><em>adagrad </em></strong>and prevents the rapid growth<strong><em> </em></strong>in <strong>v(t)<em>.</em></strong></p><p>Instead of accumulating <em>squared gradients</em> from the beginning, it accumulates the <em>previous gradients</em> in some portion(weight) which prevents rapid growth of <strong>v(t) </strong>and due to this the algorithm keeps learning and tries to <em>converge</em>.</p><p><strong>RMSProp</strong> update rule for <em>weight</em> parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*_2iZQroJGtJBNKmXxCk2Lw.png" /></figure><p>Here <strong>v(t)</strong> is <strong><em>exponentially decaying weighted sum </em></strong>of all the previous squared gradients. The <strong><em>beta</em></strong> parameter value is set to a similar value as the momentum term. The running average <strong>v(t)</strong> up to time t is dependent on <strong><em>weighted previous average gradients</em></strong> and <strong><em>current gradient</em></strong>. <strong>v(t)</strong> maintains (<strong>∇w(t))² </strong>for a fixed window time.</p><p><strong>Adagrad</strong> algorithm in Python for sigmoid neuron</p><pre>def do_RMSProp():<br>    w,b,eta,max_epochs = -6, 4, 0.1, 200<br>    v_w, v_b = 0, 0<br>    for i in range(max_epochs):<br>        dw,db = 0,0<br>        for x,y in zip(X,Y):<br>            dw += grad_w(w,b,x,y)<br>            db += grad_b(w,b,x,y)<br>        v_w = beta * v_w + (1 — beta) * dw**2<br>        v_b = beta * v_b + (1 — beta) * db**2<br>        self.w -= (eta / np.sqrt(v_w) + eps) * dw<br>        self.b -= (eta / np.sqrt(v_b) + eps) * db</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*g7Vi1mYHxa1Ai3bwZovDxw.gif" /></figure><p><strong>Issues with RMSProp</strong></p><ul><li>A large number of oscillations with high learning rate or large gradient</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Kmmt5FsBdZKsvWZeWC8Mnw.gif" /></figure><p>So far in <strong><em>Adagrad, RMSProp </em></strong>we were calculating different <strong><em>learning rates</em></strong> for different parameters, can we have different <strong><em>momentums</em></strong> for different parameters. <strong><em>Adam</em></strong> algorithm introduces the concept of <strong><em>adaptive momentum</em></strong> along with <strong><em>adaptive learning rate</em></strong>.</p><h3>Adam</h3><p><strong>Adaptive Moment Estimation (Adam) </strong>computes<strong> </strong>the<strong> </strong><em>exponentially decaying weighted sum of previous gradients m</em><strong><em>(t) </em></strong>along with an <strong><em>adaptive learning rate. Adam </em></strong>is a combined form of <strong><em>Momentum-based GD </em></strong>and<strong><em> RMSProp.</em></strong></p><p>In <strong><em>Momentum-based GD, </em></strong><em>previous gradients(history)</em> are used to compute the current gradient whereas, in RMSProp <em>previous gradients(history)</em> are used to adjust the<em> learning rate</em> based on the features<em>. Therefore </em><strong><em>Adam </em></strong>deals with <em>adaptive learning rate </em>and<em> adaptive momentum </em>where <strong><em>RMSProp</em></strong> ensures <strong>v(t) </strong>does not grow rapidly to avoid the chances of decaying learning rate and <strong>m(t) </strong>from <strong><em>Momentum-based GD </em></strong>ensures it calculates the<strong><em> </em></strong><em>exponentially decaying weighted sum of previous gradients, </em>not the <em>current gradient.</em></p><p><strong><em>Adam</em></strong> update rule for weight parameter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*rVzUkpsTiIJl8fRfyjRH5Q.png" /></figure><p>Here <strong><em>m(t)</em></strong> and <strong><em>v(t) </em></strong>are values of the <strong><em>mean</em></strong> obtained from the first moment.</p><p><strong>Adam </strong>uses<strong> bias corrected values (<em>uncentered variance)</em></strong> of gradients for <em>update rule </em>and these values are obtained through the second moment<strong><em>.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*zN5DtJoo605GZS0QS-1U3Q.png" /></figure><p>The final<em> update rule </em>is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*ikZwULDSmQCEfGqJGnITYA.png" /></figure><p><strong>Adam</strong> algorithm in Python for sigmoid neuron</p><pre>def do_Adam():<br>    w,b,eta,max_epochs = -6, 4, 0.5, 200<br>    v_w, v_b = 0, 0<br>    m_w, m_b = 0, 0<br>    num_updates = 0<br>    for i in range(epochs):<br>        dw, db = 0, 0<br>        for x, y in zip(X, Y):<br>            dw = self.grad_w(x, y)<br>            db = self.grad_b(x, y)<br>        num_updates += 1<br>        m_w = beta1 * m_w + (1-beta1) * dw<br>        m_b = beta1 * m_b + (1-beta1) * db<br>        v_w = beta2 * v_w + (1-beta2) * dw**2<br>        v_b = beta2 * v_b + (1-beta2) * db**2<br>        #m_w_c, m_b_c, v_w_c and v_b_c for bias correction   <br>        m_w_c = m_w / (1 — np.power(beta1, num_updates))<br>        m_b_c = m_b / (1 — np.power(beta1, num_updates))<br>        v_w_c = v_w / (1 — np.power(beta2, num_updates))<br>        v_b_c = v_b / (1 — np.power(beta2, num_updates))<br>        self.w -= (eta / np.sqrt(v_w_c) + eps) * m_w_c<br>        self.b -= (eta / np.sqrt(v_b_c) + eps) * m_b_c</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*df0uRBl949xJGv7jp8pGdA.gif" /></figure><p>So in <strong><em>Adam</em></strong> unlike <strong><em>RMSProp</em></strong> fewer <em>oscillations</em> and it moves more deterministically in the right direction which leads to <strong><em>faster</em></strong> <strong><em>convergence</em></strong> and <strong><em>better optimization</em></strong>.</p><h4>End Notes</h4><p>In this article, I have discussed the different types of optimization algorithms and the common issues one might encounter while using each of them. Generally, <strong>Adam </strong>with<strong> mini-batch</strong> is preferred for the training of deep neural networks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59670fd5c096" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-gradient-descent-isnt-enough-a-comprehensive-introduction-to-optimization-algorithms-in-59670fd5c096">Why Gradient descent isn’t enough: A comprehensive introduction to optimization algorithms in…</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forward propagation in neural networks — Simplified math and code version]]></title>
            <link>https://medium.com/data-science/forward-propagation-in-neural-networks-simplified-math-and-code-version-bbcfef6f9250?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbcfef6f9250</guid>
            <category><![CDATA[activation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[feed-forward]]></category>
            <category><![CDATA[forward-propagation]]></category>
            <category><![CDATA[weighted-sum-of-inputs]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Tue, 07 May 2019 04:25:58 GMT</pubDate>
            <atom:updated>2023-05-05T05:39:56.316Z</atom:updated>
            <content:encoded><![CDATA[<h3>Forward propagation in neural networks — Simplified math and code version</h3><p>As we all know from the last one-decade <strong><em>deep learning</em></strong> has become one of the most widely accepted emerging technology. This is due to its representational power of functions.</p><p>According to <strong><em>Universal approximation theorem</em></strong>, a well-guided and engineered <em>deep neural network</em> can approximate any arbitrary complex and continuous relationship among the variables. In fact, there are several other reasons behind the success of <em>deep learning</em>. I am not going to discuss these possible reasons here.</p><p>The goal of this post is to explain <strong><em>forward propagation</em></strong>(one of the core process during learning phase) in a simpler way.</p><p>A learning algorithm/model finds out the parameters (weights and biases) with the help of <em>forward propagation</em> and <em>backpropagation.</em></p><h3>Forward propagation</h3><p>aAs the name suggests, the input data is fed in the forward direction through the network. Each hidden layer accepts the input data, processes it as per the activation function and passes to the successive layer.</p><h4>Why Feed-forward network?</h4><p>In order to generate some output, the input data should be fed in the forward direction only. The data should not flow in reverse direction during output generation otherwise it would form a cycle and the output could never be generated. Such network configurations are known as <strong><em>feed-forward network</em></strong>. The <em>feed-forward network</em> helps in <em>forward propagation</em>.</p><p>At each neuron in a hidden or output layer, the processing happens in two steps:</p><ol><li><strong>Preactivation:</strong> it is a <em>weighted sum of inputs</em> i.e. the <em>linear transformation of weights</em> w.r.t to inputs available. Based on this <em>aggregated sum</em> and <em>activation function</em> the neuron makes a decision whether to pass this information further or not.</li><li><strong>Activation:</strong> the calculated weighted sum of inputs is passed to the activation function. An activation function is a mathematical function which adds non-linearity to the network. There are four commonly used and popular activation functions — sigmoid, hyperbolic tangent(tanh), ReLU and Softmax.</li></ol><p>Now let us understand <em>forward propagation</em> with the help of an example. Consider a <em>non-linearly</em> separable data in the form of two moons of data points following a swirl pattern. This generated data has two different classes.</p><p>The data can be generated using make_moons() function of sklearn.datasets module. The total number of samples to be generated and noise about the moon’s shape can be adjusted using the function parameters.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>import matplotlib.colors<br>from sklearn.datasets import make_moons</pre><pre>np.random.seed(0)</pre><pre>data, labels = make_moons(n_samples=200,noise = 0.04,random_state=0)<br>print(data.shape, labels.shape)</pre><pre>color_map = matplotlib.colors.LinearSegmentedColormap.from_list(&quot;&quot;, [&quot;red&quot;,&quot;yellow&quot;])<br>plt.scatter(data[:,0], data[:,1], c=labels, cmap=my_cmap)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*F50x2COgQ8LySWV0LtdFLw.png" /><figcaption>dataset visualization</figcaption></figure><p>Here, 200 samples are used to generate the data and it has two classes shown in red and green color.</p><p>Now, let us see the neural network structure to predict the class for this <em>binary classification</em> problem. Here, I am going to use one <em>hidden layer</em> with two neurons, an <em>output layer</em> with a single neuron and <em>sigmoid activation</em> function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uiHGvuZLAQkQJHoIwL84kQ.jpeg" /></figure><p>During <em>forward propagation </em>at each node of hidden and output layer <em>preactivation</em> and <em>activation</em> takes place. For example at the first node of the hidden layer, <strong>a1</strong>(<em>preactivation</em>) is calculated first and then <strong>h1</strong>(<em>activation</em>) is calculated.</p><p><strong>a1</strong> is a weighted sum of inputs. Here, the weights are randomly generated.</p><p><strong>a1 </strong>= w1*x1 + w2*x2 + b1 = 1.76* 0.88 + 0.40*(-0.49) + 0 = 1.37 approx and<strong> h1 </strong>is the value of activation function applied on a1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*WrkgXLQSjHpzmR_H3xsnCQ.png" /></figure><p>Similarly</p><p><strong>a2</strong> = w3*x1 + w4*x2 + b2 = 0.97 *0.88 + 2.24 *(- 0.49)+ 0 = -2.29 approx and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*46xma79g8Gdew_LbT6x2aw.png" /></figure><p>For any layer after the first hidden layer, the input is output from the previous layer.</p><p><strong>a3 </strong>= w5*h1 + w6*h2 + b3 = 1.86*0.8 + (-0.97)*0.44 + 0 = 1.1 approx</p><p>and</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/1*lCVQROFldjILndKg-pKHxw.png" /></figure><p>So there are 74% chances the first observation will belong to class 1. Like this for all the other observations predicted output can be calculated.</p><p>The below image represents the transformation of data from the input layer to the output layer for the first observation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ts5LSdtkfSsMYS7M0X84Tw.gif" /><figcaption>Data transformation from the input layer to the output layer</figcaption></figure><p>Now let us see the implementation of the above neural network in Jupyter notebook. In fact, while building deep neural networks frameworks like Tensorflow, Keras, PyTorch, etc. are used.</p><pre>from sklearn.model_selection import train_test_split</pre><pre>#Splitting the data into training and testing data<br>X_train, X_val, Y_train, Y_val = train_test_split(data, labels, stratify=labels, random_state=0)<br>print(X_train.shape, X_val.shape)<br></pre><p>Here, 150 observations are used for training purpose and 50 for testing purpose as per the default split ratio of 75:25.</p><p>Now let us define a class for <em>forward propagation</em> where weights are randomly initialized.</p><pre>class FeedForwardNetwork:<br> <br> def __init__(self):<br>     np.random.seed(0)<br>     self.w1 = np.random.randn()<br>     self.w2 = np.random.randn()<br>     self.w3 = np.random.randn()<br>     self.w4 = np.random.randn()<br>     self.w5 = np.random.randn()<br>     self.w6 = np.random.randn()<br>     self.b1 = 0<br>     self.b2 = 0<br>     self.b3 = 0<br> <br> def sigmoid(self, x):<br>     return 1.0/(1.0 + np.exp(-x))<br> <br> def forward_pass(self, x):<br>     self.x1, self.x2 = x<br>     self.a1 = self.w1*self.x1 + self.w2*self.x2 + self.b1<br>     self.h1 = self.sigmoid(self.a1)<br>     self.a2 = self.w3*self.x1 + self.w4*self.x2 + self.b2<br>     self.h2 = self.sigmoid(self.a2)<br>     self.a3 = self.w5*self.h1 + self.w6*self.h2 + self.b3<br>     self.h3 = self.sigmoid(self.a3)<br>     forward_matrix = np.array([[0,0,0,0,self.h3,0,0,0], <br>                      [0,0,(self.w5*self.h1),        (self.w6*self.h2),self.b3,self.a3,0,0],<br>                      [0,0,0,self.h1,0,0,0,self.h2],<br>                      [(self.w1*self.x1), (self.w2*self.x2),         self.b1, self.a1,(self.w3*self.x1),(self.w4*self.x2), self.b2,  self.a2]])<br>     forward_matrices.append(forward_matrix)<br>     return self.h3</pre><p>Here, the forward_pass()function calculates the output value for the given input observation. forward_matrix is a 2d array to store the values of a1, h1, a2, h2, a3, h3, etc for each observation. The reason to use it is just to visualize the transformation of these values using a GIF image. Entries offorward_matrix are as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*v_s725vl3FTGyCLmORU3HA.png" /><figcaption>forward_matrix</figcaption></figure><pre>forward_matrices = []<br>ffn = FeedForwardNetwork()<br>for x in X_train:<br>   ffn.forward_pass(x)</pre><p>forward_matrices is a list offorward_matrix for all the observations.</p><pre>import seaborn as sns<br>import imageio<br>from IPython.display import HTML</pre><pre>def plot_heat_map(observation):<br>    fig = plt.figure(figsize=(10, 1))<br>    sns.heatmap(forward_matrices[observation], annot=True,     cmap=my_cmap, vmin=-3, vmax=3)<br>    plt.title(“Observation “+str(observation))</pre><pre>    fig.canvas.draw()<br>    image = np.frombuffer(fig.canvas.tostring_rgb(), dtype=’uint8&#39;)<br>    image = image.reshape(fig.canvas.get_width_height()[::-1] +             (3,))</pre><pre>    return image</pre><pre>imageio.mimsave(‘./forwardpropagation_viz.gif’, [plot_heat_map(i) for i in range(0,len(forward_matrices),len(forward_matrices)//15)], fps=1)</pre><p>plot_heat_map() function creates a heat map to visualize the values offorward_matrix for each observation. These heatmaps are stored in forwardpropagation_viz.gifimage. Here, total 15 different heat maps are created for 15 different observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Y9Mt8EJnPaowXQfcleWSGg.gif" /><figcaption>forward propagation for 15 different observations</figcaption></figure><h4>Code Optimization</h4><p>Instead of using different variables like w1, w2…w6, a1, a2, h1, h2, etc. separately, a vectorized matrix can be used for weights, preactivation(a) and activation(h) respectively. Vectorization enables much more efficient and faster execution of code. It also has easy to follow and learn syntax.</p><pre>class FeedForwardNetwork_Vectorised:<br> <br> def __init__(self):<br>    np.random.seed(0)<br>    self.W1 = np.random.randn(2,2)<br>    self.W2 = np.random.randn(2,1)<br>    self.B1 = np.zeros((1,2))<br>    self.B2 = np.zeros((1,1))<br> <br> def sigmoid(self, X):<br>    return 1.0/(1.0 + np.exp(-X))<br> <br> <br> def forward_pass(self,X):<br>    self.A1 = np.matmul(X,self.W1) + self.B1 <br>    self.H1 = self.sigmoid(self.A1) <br>    self.A2 = np.matmul(self.H1, self.W2) + self.B2<br>    self.H2 = self.sigmoid(self.A2) <br>    return self.H2</pre><pre>ffn_v = FeedForwardNetwork_Vectorised()<br>ffn_v.forward_pass(X_train)</pre><h4><strong>Conclusion</strong></h4><p>This is about <strong><em>forward propagation</em></strong> from my side and I hope I was able to explain the intuition and steps involved in <em>forward propagation</em>. If you are interested in learning or exploring more about neural networks, refer to my other blog offerings on neural networks. The links are as below</p><p><a href="https://www.analyticsvidhya.com/blog/2020/10/a-comprehensive-guide-to-feature-selection-using-wrapper-methods-in-python/">https://www.analyticsvidhya.com/blog/2020/10/a-comprehensive-guide-to-feature-selection-using-wrapper-methods-in-python/</a></p><p><a href="https://towardsdatascience.com/why-better-weight-initialization-is-important-in-neural-networks-ff9acf01026d">Why better weight initialization is important in neural networks?</a></p><p><a href="https://towardsdatascience.com/analyzing-different-types-of-activation-functions-in-neural-networks-which-one-to-prefer-e11649256209">Analyzing different types of activation functions in neural networks — which one to prefer?</a></p><p><a href="https://towardsdatascience.com/why-gradient-descent-isnt-enough-a-comprehensive-introduction-to-optimization-algorithms-in-59670fd5c096">Why Gradient descent isn’t enough: A comprehensive introduction to optimization algorithms in neural networks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbcfef6f9250" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/forward-propagation-in-neural-networks-simplified-math-and-code-version-bbcfef6f9250">Forward propagation in neural networks — Simplified math and code version</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dealing with Apply functions in R]]></title>
            <link>https://medium.com/data-science/dealing-with-apply-functions-in-r-ea99d3f49a71?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea99d3f49a71</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[lapply]]></category>
            <category><![CDATA[apply]]></category>
            <category><![CDATA[apply-function]]></category>
            <category><![CDATA[sapply]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Wed, 27 Mar 2019 08:12:40 GMT</pubDate>
            <atom:updated>2021-03-27T14:26:46.801Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*IBBn38qq8fXa2K63aYztdQ.jpeg" /><figcaption>Apply functions in R</figcaption></figure><p>Iterative control structures (loops like for, while, repeat, etc.) allow repetition of instructions for several numbers of times. However, at large scale data processing usage of these loops can consume more time and space. R language has a more efficient and quick approach to perform iterations with the help of <strong>Apply</strong> functions.</p><p>In this post, I am going to discuss the efficiency of <strong>apply</strong> functions over loops from a visual perspective and then further members of <strong>apply </strong>family.</p><p>Before proceeding further with <strong>apply</strong> functions let us first see how code execution takes less time for iterations using <strong>apply</strong> functions compared to basic loops.</p><p>Consider the <strong>FARS(Fatality Analysis Recording System)</strong> dataset available in gamclass package of R. It contains 151158 observations of 17 different features. The dataset includes every accident in which there was at least one fatality and the data is limited to vehicles where the front seat passenger seat was occupied.</p><p>Now let us assume we want to calculate the mean of age column. This can be done using traditional loops and also using apply functions.</p><p>Method 1: Using for loop</p><pre>library(&quot;gamclass&quot;)<br>data(FARS)<br>mean_age &lt;- NULL<br>total &lt;- NULL<br>for(i in 1:length(FARS$age)){<br>  total &lt;- sum(total, FARS$age[i])<br> }<br>mean_age &lt;- total/length(FARS$age)<br>mean_age</pre><p>Method 2: Using apply() function</p><pre>apply(FARS[3],2, mean)</pre><p>Now let us compare both the approaches through visual mode with the help of Profvis package.</p><p>Profvis<em> </em>is a code-profiling tool, which provides an interactive graphical interface for visualizing the memory and time consumption of instructions throughout the execution.</p><p>To make use of profvis, enclose the instructions in profvis(), it opens an interactive profile visualizer in a new tab inside R studio.</p><pre>#for method 1<br>profvis({<br>mean_age &lt;- NULL<br>total &lt;- NULL<br>for(i in 1:length(FARS$age)){<br>  total &lt;- sum(total, FARS$age[i])<br> }<br>mean_age &lt;- total/length(FARS$age)<br>mean_age<br>})</pre><p>Output using method 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*SYc1TRYVdWF12frADChfdw.png" /></figure><p>Under Flame Graph tab we can inspect the time taken (in ms) by the instructions.</p><pre>#for method 2<br>profvis({<br>  apply(FARS[3],2, mean)<br>})</pre><p>Output using method 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*M73olXQpqBR0ugOIzH99FA.png" /></figure><p>Here, one can easily notice that the time taken using method 1 is almost 1990 ms (1960 +30) whereas for method 2 it is only 20 ms. So this is the actual power of apply() functions in terms of time consumption.</p><h4>Benefits of apply functions over traditional loops</h4><ol><li>Much more efficient and faster in execution.</li><li>Easy to follow syntax (rather than writing a block of instructions only one line of code using apply functions)</li></ol><h4>Apply family in R</h4><p>Apply family contains various flavored functions which are applicable to different data structures like list, matrix, array, data frame etc. The members of the apply family are apply(), lapply(), sapply(), tapply(), mapply() etc. These functions are substitutes/alternatives to loops.</p><p>Each of the apply functions requires a minimum of two arguments: an object and another function. The function can be any inbuilt (like mean, sum, max etc.) or user-defined function.</p><h3>Explore the members</h3><h4>1. apply() function</h4><p>The syntax of apply() is as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*MU9DKWA6hmKpg1-VD3cS4A.png" /></figure><p>where X is an input data object, MARGIN indicates how the function is applicable whether row-wise or column-wise, margin = 1 indicates row-wise and margin = 2 indicates column-wise, FUN points to an inbuilt or user-defined function.</p><p>The output object type depends on the input object and the function specified. apply() can return a vector, list, matrix or array for different input objects as mentioned in the below table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*V3juasHOzh-ew3POY8lqjw.png" /></figure><pre>#---------- apply() function ---------- <br>#case 1. matrix as an input argument<br>m1 &lt;- matrix(1:9, nrow =3)<br>m1<br>result &lt;- apply(m1,1,mean)   #mean of elements for each row<br>result<br>class(result)                #class is a vector<br>result &lt;- apply(m1,2,sum)    #sum of elements for each column<br>result<br>class(result)                #class is a vector<br>result &lt;- apply(m1,1,cumsum) #cumulative sum of elements for each row<br>result                       #by default column-wise order<br>class(result)                #class is a matrix<br>matrix(apply(m1,1,cumsum), nrow = 3, byrow = T) #for row-wise order <br>#user defined function <br>check&lt;-function(x){<br>  return(x[x&gt;5])<br>}<br>result &lt;- apply(m1,1,check)  #user defined function as an argument<br>result<br>class(result)                #class is a list</pre><pre>#case 2. data frame as an input<br>ratings &lt;- c(4.2, 4.4, 3.4, 3.9, 5, 4.1, 3.2, 3.9, 4.6, 4.8, 5, 4, 4.5, 3.9, 4.7, 3.6)<br>employee.mat &lt;- matrix(ratings,byrow=TRUE,nrow=4,dimnames = list(c(&quot;Quarter1&quot;,&quot;Quarter2&quot;,&quot;Quarter3&quot;,&quot;Quarter4&quot;),c(&quot;Hari&quot;,&quot;Shri&quot;,&quot;John&quot;,&quot;Albert&quot;)))<br>employee &lt;- as.data.frame(employee.mat)<br>employee<br>result &lt;- apply(employee,2,sum)     #sum of elements for each column<br>result<br>class(result)                       #class is a vector<br>result &lt;- apply(employee,1,cumsum)  #cumulative sum of elements for each row<br>result                              #by default column-wise order<br>class(result)                       #class is a matrix<br>#user defined function <br>check&lt;-function(x){<br>  return(x[x&gt;4.2])<br>}<br>result &lt;- apply(employee,2,check)   #user defined function as an argument<br>result<br>class(result)                       #class is a list</pre><h4>2. lapply() function</h4><p>lapply() always returns a list, ‘l’ in lapply() refers to ‘list’. lapply() deals with list and data frames in the input. MARGIN argument is not required here, the specified function is applicable only through columns. Refer to the below table for input objects and the corresponding output objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*Pr-1xzu92CLbjge4DHK_OA.png" /></figure><pre>#---------- lapply() function ---------- <br>#case 1. vector as an input argument<br>result &lt;- lapply(ratings,mean)<br>result<br>class(result)                       #class is a list</pre><pre>#case 2. list as an input argument<br>list1&lt;-list(maths=c(64,45,89,67),english=c(79,84,62,80),physics=c(68,72,69,80),chemistry = c(99,91,84,89))<br>list1<br>result &lt;- lapply(list1,mean)<br>result<br>class(result)                       #class is a list<br>#user defined function<br>check&lt;-function(x){<br>  return(x[x&gt;75])<br>}<br>result &lt;- lapply(list1,check)       #user defined function as an argument<br>result<br>class(result)                       #class is a list</pre><pre>#case 3. dataframe as an input argument<br>result &lt;- lapply(employee,sum)     #sum of elements for each column<br>result<br>class(result)                       #class is a list<br>result &lt;- lapply(employee,cumsum)  #cumulative sum of elements for each row<br>result                              <br>class(result)                       #class is a list<br>#user defined function <br>check&lt;-function(x){<br>  return(x[x&gt;4.2])<br>}<br>result &lt;- lapply(employee,check)   #user defined function as an argument<br>result<br>class(result)                      #class is a list</pre><p><strong>apply() vs. </strong><strong>lapply()</strong></p><ul><li>lapply() always returns a list whereas apply() can return a vector, list, matrix or array.</li><li>No scope of MARGIN in lapply().</li></ul><h4>3. sapply() function</h4><p>sapply() is a simplified form of lapply(). It has one additional argument <strong>simplify </strong>with default value as true<strong>, </strong>if simplify = F then sapply() returns a list similar to lapply(), otherwise, it returns the simplest output form possible.</p><p>Refer to the below table for input objects and the corresponding output objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*ALlwjH0drVjMJ5CnQdDiUw.png" /></figure><pre>#---------- sapply() function ---------- <br>#case 1. vector as an input argument<br>result &lt;- sapply(ratings,mean)<br>result<br>class(result)                       #class is a vector<br>result &lt;- sapply(ratings,mean, simplify = FALSE)<br>result <br>class(result)                       #class is a list<br>result &lt;- sapply(ratings,range)<br>result<br>class(result)                       #class is a matrix</pre><pre>#case 2. list as an input argument<br>result &lt;- sapply(list1,mean)<br>result<br>class(result)                       #class is a vector<br>result &lt;- sapply(list1,range)<br>result<br>class(result)                       #class is a matrix<br>#user defined function<br>check&lt;-function(x){<br>  return(x[x&gt;75])<br>}<br>result &lt;- sapply(list1,check)       #user defined function as an argument<br>result<br>class(result)                       #class is a list</pre><pre>#case 3. dataframe as an input argument<br>result &lt;- sapply(employee,mean)<br>result<br>class(result)                       #class is a vector<br>result &lt;- sapply(employee,range)<br>result<br>class(result)                       #class is a matrix<br>#user defined function<br>check&lt;-function(x){<br>  return(x[x&gt;4])<br>}<br>result &lt;- sapply(employee,check)    #user defined function as an argument<br>result<br>class(result)                       #class is a list</pre><h4>4. tapply() function</h4><p>tapply() is helpful while dealing with categorical variables, it applies a function to numeric data distributed across various categories. The simplest form of tapply() can be understood as</p><p>tapply(column 1, column 2, FUN)</p><p>where column 1 is the numeric column on which function is applied, column 2 is a factor object and FUN is for the function to be performed.</p><pre>#---------- tapply() function ---------- <br>salary &lt;- c(21000,29000,32000,34000,45000)<br>designation&lt;-c(&quot;Programmer&quot;,&quot;Senior Programmer&quot;,&quot;Senior Programmer&quot;,&quot;Senior Programmer&quot;,&quot;Manager&quot;)<br>gender &lt;- c(&quot;M&quot;,&quot;F&quot;,&quot;F&quot;,&quot;M&quot;,&quot;M&quot;)<br>result &lt;- tapply(salary,designation,mean)<br>result<br>class(result)                       #class is an array<br>result &lt;- tapply(salary,list(designation,gender),mean)<br>result<br>class(result)                       #class is a matrix</pre><h4>5. by() function</h4><p>by() does a similar job to tapply() i.e. it applies an operation to numeric vector values distributed across various categories. by() is a wrapper function of tapply().</p><pre>#---------- by() function ---------- <br>result &lt;- by(salary,designation,mean)<br>result<br>class(result)                       #class is of &quot;by&quot; type<br>result[2]                           #accessing as a vector element<br>as.list(result)                     #converting into a list<br>result &lt;- by(salary,list(designation,gender),mean)<br>result<br>class(result)                       #class is of &quot;by&quot; type<br>library(&quot;gamclass&quot;)<br>data(&quot;FARS&quot;)<br>by(FARS[2:4], FARS$airbagAvail, colMeans)</pre><h4>6. mapply() function</h4><p>The ‘m’ in mapply() refers to ‘multivariate’. It applies the specified functions to the arguments one by one. Note that here function is specified as the first argument whereas in other apply functions as the third argument.</p><pre>#---------- mapply() function ---------- <br>result &lt;- mapply(rep, 1:4, 4:1)<br>result<br>class(result)                       #class is a list<br>result &lt;- mapply(rep, 1:4, 4:4)<br>class(result)                       #class is a matrix</pre><p><strong>Conclusion</strong></p><p>I believe I have covered all the most useful and popular apply functions with all possible combinations of input objects. If you think something is missing or more inputs are required. Let me know in the comments and I’ll add it in!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea99d3f49a71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dealing-with-apply-functions-in-r-ea99d3f49a71">Dealing with Apply functions in R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simulating(Replicating) R regression plot in Python using sklearn]]></title>
            <link>https://medium.com/data-science/simulating-replicating-r-regression-plot-in-python-using-sklearn-4ee48a15b67?source=rss-3c5638fcb6c5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ee48a15b67</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[vikashraj luhaniwal]]></dc:creator>
            <pubDate>Fri, 22 Feb 2019 18:57:48 GMT</pubDate>
            <atom:updated>2021-03-27T14:25:48.989Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<p>When it comes to data science and machine learning workloads, R and Python are the most popular and powerful languages. Python is often treated as a general-purpose language with an easy-to-understand syntax, whereas R for statistical analysis with an availability of around 12000 packages.</p><p>There are dozens of articles available comparing Python and R from a subjective point of view. I am not going to favour here one language over the other.</p><p>In this post, we will discuss the <strong>replication of R regression plots in Python</strong> using sklearn.</p><p>Most of the R’s functionality can be easily and directly converted in Python, but some are surprisingly hard to find equivalents without using custom functions. For e.g. <strong>plot ()</strong> function for Regression models in R does not have a direct equivalent for all the plots in Python.</p><p>Let us discuss it with <strong>faithful</strong> dataset available in R. The dataset contains 272 observation of two variables <strong><em>eruptions</em></strong> (Eruption time in minutes) and <strong><em>waiting</em></strong> (Waiting time to next eruption). This dataset reveals Waiting time between eruptions and the duration of the eruption for the Old Faithful geyser in Yellowstone National Park, Wyoming, USA.</p><p>Now let us fit a simple linear regression model in R to this dataset for predicting waiting time based on eruption time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d7034cd9bb47cffe0ad38a37b333d76/href">https://medium.com/media/5d7034cd9bb47cffe0ad38a37b333d76/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aBXHf2tjXHqtaHoGU8WF2w.png" /></figure><p>Each of the above plots has its own significance for validating the assumptions of linearity. We are not going to deep dive here for the same. Let’s focus on Python code for fitting the same linear regression model.</p><p>Import all the necessary libraries and load the required data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e671f54e14ed3d89588aef6b5c96f634/href">https://medium.com/media/e671f54e14ed3d89588aef6b5c96f634/href</a></iframe><p>Now let’s fit a linear regression model on <strong><em>faithful</em></strong> dataset using sklearn.linear_model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7dfe3e3aea3a696fa1fee66d40a3428/href">https://medium.com/media/b7dfe3e3aea3a696fa1fee66d40a3428/href</a></iframe><p>Now let us focus on all the regression plots one by one using sklearn.</p><ol><li><strong>Residual plot</strong></li></ol><p>It’s the first plot generated by <strong>plot()</strong> function in R and also sometimes known as residual vs fitted plot. It is useful in validating the assumption of linearity, by drawing a scatter plot between fitted values and residuals.</p><p>If the plot depicts any specific or regular pattern then it is assumed the relation between the target variable and predictors is non-linear in nature i.e. non-linearity exists. And no pattern in the curve is a sign of linearity among the selected features and the target variable.</p><p>This same plot in Python can be obtained using residplot() function available in Seaborn. Here, the first and second argument points to fitted(predicted) values and target variable respectively. lowess=True ensures lowess(smoothened) regression line is drawn and using line_kws argument we can customize the attributes of this line.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c21e66f0bf4a88d76b61031c5c231079/href">https://medium.com/media/c21e66f0bf4a88d76b61031c5c231079/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*sZKiD9cnJMspVW_zJ5ek1Q.png" /></figure><p><strong>2. QQ plot</strong></p><p>This plot depicts whether the residuals(errors) are normally distributed or not. If the points lie close to the normal line then residuals are assumed to be normally distributed.</p><p>In Python, this same plot can be achieved using probplot() function available in seaborn. Here, the residuals are passed as an argument to the function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/648d460016e2ffe3c897483e4621e806/href">https://medium.com/media/648d460016e2ffe3c897483e4621e806/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*SP19fI4jyCgQXCsl_ezR2A.png" /></figure><p><strong>3. Scale-Location plot</strong></p><p>Generally, it is used to guess homoscedasticity of residuals. It is a plot of square-rooted standardized residual against fitted value. If it depicts no specific pattern then the fitted regression model upholds homoscedasticity assumption.</p><p>This same plot in Python can be obtained using regplot() function available in Seaborn. Here, the first and second argument points to fitted values and square-rooted standardized residuals respectively.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dea35f3be02377122bd69cd05640be62/href">https://medium.com/media/dea35f3be02377122bd69cd05640be62/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*-Q115uDOf3HSTrV5hQuv3Q.png" /></figure><p><strong>4. Leverage plot</strong></p><p>Generally, it is used to guess the impact of outliers over the regression fit. Currently, I could not figure out how to draw the same in Python for a sklearn based fitted model. Once I would be able to figure out, I will be updating the same. Using statsmodels it is quite easy to draw using the built-in leverage plot but I am not going to discuss it over here.</p><p>If you know already how to draw this same plot for a sklearn based fitted model. Let me know in the comments and I’ll add it in!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ee48a15b67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/simulating-replicating-r-regression-plot-in-python-using-sklearn-4ee48a15b67">Simulating(Replicating) R regression plot in Python using sklearn</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>