<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Retin P Kumar on Medium]]></title>
        <description><![CDATA[Stories by Retin P Kumar on Medium]]></description>
        <link>https://medium.com/@retinpkumar?source=rss-e06b7a5f1c70------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ugIkvtpv7kMZwKjqSmdkvg.jpeg</url>
            <title>Stories by Retin P Kumar on Medium</title>
            <link>https://medium.com/@retinpkumar?source=rss-e06b7a5f1c70------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:35:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@retinpkumar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Top 10 insights from my analysis on GeeksforGeeks dataset]]></title>
            <link>https://retinpkumar.medium.com/analyzing-geeksforgeeks-5f4639307754?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f4639307754</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Mon, 28 Feb 2022 15:37:33 GMT</pubDate>
            <atom:updated>2022-02-28T16:03:08.379Z</atom:updated>
            <content:encoded><![CDATA[<p>My analysis of the articles published on GeeksforGeeks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/0*5NEancmsVdoed70z.png" /><figcaption>credit: <a href="https://www.geeksforgeeks.org/">https://www.geeksforgeeks.org/</a></figcaption></figure><p>For every Indian student in the Computer science / IT-related field, GeeksforGeeks is one of the top websites in their list of references. To some, GeeksforGeeks has been a source of education and knowledge up-gradation.</p><p>So, when I came across this <a href="https://www.kaggle.com/ashishjangra27/geeksforgeeks-articles">dataset in Kaggle</a>, that contained information regarding the articles published and related details, naturally I was curious to analyze it.</p><p>Hence, I performed a quick analysis of the dataset, and here are some of the details that I found during my analysis.</p><h3>About the dataset</h3><p>This dataset has details regarding 34574 articles posted in GeeksforGeeks from 11 Nov 2010 till 23 Feb 2022.</p><p>The dataset contains 5 features that are:</p><p>1. Title of the Article</p><p>2. Author ID</p><p>3. Last updated date</p><p>4. Link of the Article on GeeksforGeeks</p><p>5. Category (difficulty level) to which the article belongs</p><h3>Most written Topic</h3><p>The topic with the most number of articles on the GeeksforGeeks platform was related to interview preparation and work experience.</p><p>Python topped the list of articles written for programming languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*m9finFoqCvZpATlI" /><figcaption>Most written topic (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h3>What all categories do the articles belong to?</h3><p>Most of the articles published in GeeksforGeeks are of ‘medium’ difficulty (30.20%), followed by ‘easy’ (27.96%) and ‘basic’ (23.55%) levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*JCyuNEtrYuGAKmKC" /><figcaption>Article categories (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h3>Who contributed the most?</h3><p>Most articles were written by “GeeksforGeeks” itself. But, the most number of the articles contributed by an individual contributor belongs to “ManasChhabra2”, followed by “Striver” and “manjeet_04”.</p><p>The figure below shows the top 10 individual contributors in GeeksforGeeks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*0Ox6cLzucCkLdFmH" /><figcaption>Top 10 contributors (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h3>Number of articles contributed</h3><p>Most articles were contributed in the year 2021 followed by 2020 and 2019. This can be attributed to the pandemic-induced importance of online work.</p><p>The number of articles contributed in 2022 so far has been close to 2019 levels and is expected to surpass 2021 levels by the end of 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*tk9zq3MVhBVWOz5L" /><figcaption>Articles per year (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h3>Month-wise distribution of article contributions</h3><p>Most of the articles contributed in 2021 were written in the month of June followed by May and August.</p><p>The pattern indicates the recruitment period from May to November where the majority of the students are either applying or appearing for job interviews.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*IRJ5eM5oJQWw-4uh" /><figcaption>Monthly distribution of articles (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Articles contributed during June 2021 followed a cyclical pattern with an increasing trend for articles published during weekends and a decreasing trend for the ones after the days following weekends.</p><p>The majority of the articles published during June 2021 were on the 28th.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1dAvw8uQtsCSoGrZ" /><figcaption>Date wise distribution of articles on June 2021 (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h3>Topics contributed for June 2021</h3><p>Interestingly, the most number of articles contributed during June 2021 were related to ‘GATE’. But, “GATE 2021’ had already been conducted in Feb 2021.</p><p>Java topped the list of programming languages for which articles were published during the period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*S0cUUIODH0nmAhRs" /><figcaption>Topics for June 2021 (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h3>Most contributed topic in 2022</h3><p>‘Number’ and ‘array’ were the most appeared keyword for the articles published in 2022 so far.</p><p>Python, followed by Java and Javascript were the top 3 programming languages based on the articles published.</p><p>Interviews and experience were other top keywords. TCS interview topped more than ‘GATE’ for the articles published in 2022. This can be due to the upcoming TCS hiring window for April 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*w1Ki3dYV_TxUc6M8" /><figcaption>Topics for 2022 (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p><a href="https://medium.com/geekculture/i-analyzed-a-data-science-job-salaries-dataset-dca3c24081c0">I Analyzed a Data science Job Salaries Dataset</a></p><h3>Overall Contributions</h3><p>Of all the contributions made in the GeeksforGeeks platform from 11th Nov 2010 till 23 Feb 2022, the highest monthly contributions were made during the May-June period and the least were made during March.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*ZKkr6VlFHEvEr9fW" /><figcaption>Overall monthly contributions (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*PYnJY9deABPU3PMo" /><figcaption>The cumulative sum of Monthly contributions (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Most of the daily contributions happened during the last week of every month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6kh19t-Mpi4qxC5w" /><figcaption>Daily contributions (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Of all the contributions, the majority of the contributions were made during Mondays, followed by Wednesdays and Thursdays.</p><p>The least number of contributions were made during the weekends.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/0*KJx6_6bK9El2GGh4" /><figcaption>Contributions per week-day (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>With that, I conclude my analysis of the articles published in GeeksforGeeks.</p><p>Thank you for reading. I hope, you liked the article.</p><p>Follow me for more interesting and actionable articles at:</p><ul><li><a href="https://medium.com/@retinpkumar">Retin P Kumar - Medium</a></li><li><a href="https://blog.devgenius.io/implementing-k-nearest-neighbors-from-scratch-in-python-d5eaaf558d49">Implementing K Nearest Neighbors from scratch in Python</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f4639307754" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing K Nearest Neighbors from scratch in Python]]></title>
            <link>https://blog.devgenius.io/implementing-k-nearest-neighbors-from-scratch-in-python-d5eaaf558d49?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5eaaf558d49</guid>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Thu, 03 Feb 2022 07:42:17 GMT</pubDate>
            <atom:updated>2022-02-13T09:44:30.939Z</atom:updated>
            <content:encoded><![CDATA[<h4>And comparing our model performance with Scikit-learn implementation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BSSy9Hmjni2LpQzI" /><figcaption>Photo by <a href="https://unsplash.com/@derickray?utm_source=medium&amp;utm_medium=referral">Derick McKinney</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>K Nearest Neighbors(KNN) is one of the simplest supervised machine learning algorithms. The algorithm was initially developed for classification tasks but was later extended for performing regression tasks as well.</p><p>Unlike other supervised machine learning algorithms, <strong>KNN doesn’t have an underlying model and the algorithm works solely on the stored training data.</strong></p><p>KNN tries to predict the values based on the distances of each test data point from the training dataset. Hence the algorithm doesn’t make any assumptions regarding the features or the output of the data.</p><p>Due to this nature of the algorithm where it starts working and makes a prediction only after a test data is introduced, it is also known as <strong>Lazy learner</strong>.</p><p>Apart from classification and regression tasks, the KNN algorithm is also used for missing value imputation, resampling datasets, outlier detection, etc. Some of the industrial applications of KNN include building a Recommender system, handwriting recognition, image recognition, etc.</p><p>There are several approaches for finding K Nearest Neighbors like <strong>K-dimensional tree</strong> which uses a rectangular decision boundary for classifying data points, <strong>Ball-tree</strong> which uses a spherical decision boundary, etc. In this article, we will implement the brute force approach to KNN using Python from scratch.</p><h3>The Algorithm</h3><p>So, the steps for creating a KNN model is as follows:</p><ol><li>We need an optimal value for K to start with.</li><li>Calculate the distance of each data point in the test set with each point in the training set.</li><li>Sort the calculated distances along with the corresponding target values from training data in ascending order.</li><li>From these sorted values, select the K top values.</li><li>For <strong>Classification tasks</strong>, the Mode of data points corresponding to K top rows will be the predicted output and for <strong>Regression tasks</strong>, Mean/ Median will be the predicted output.</li></ol><h3>Implementation from scratch</h3><p>Let’s create the basic structure of our KNN algorithm. We will follow the Scikit-learn approach and define a class with a fit and predict method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae3135a158ae90bdbc2d3ee4e15c6c16/href">https://medium.com/media/ae3135a158ae90bdbc2d3ee4e15c6c16/href</a></iframe><p>We will initialize our class instance using the number of neighbors, problem type, and evaluation metric.</p><p>Problem type will decide whether it will be a regression or a classification task and metric will decide which distance metric should be used for calculating the distances.</p><p>For now, we will assign problem type 0 for regression and 1 for classification. And, metric 0 for Euclidean distance and 1 for Manhattan distance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ba13bf1e268866a1cecea02e2c84d02/href">https://medium.com/media/0ba13bf1e268866a1cecea02e2c84d02/href</a></iframe><p>In our fit method, we will fit the algorithm with the training data features and target.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9ab2ddaa11377c7f956dc3715a29881/href">https://medium.com/media/a9ab2ddaa11377c7f956dc3715a29881/href</a></iframe><p>Inside our predict method that takes test feature data as the argument, we have 3 additional steps to perform.</p><ul><li>First, we calculate the distances.</li><li>Then we calculate the neighbors.</li><li>Finally, we make the prediction.</li></ul><p>The code is self-explanatory.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eaf703a2c21a9903c394ec2d3a479414/href">https://medium.com/media/eaf703a2c21a9903c394ec2d3a479414/href</a></iframe><p>Putting it all together, we have our BruteForceKNN algorithm.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dec25f842ea8a3d466328e5362fb6a13/href">https://medium.com/media/dec25f842ea8a3d466328e5362fb6a13/href</a></iframe><p>Let’s now try to implement KNN using Scikit learn library and calculate the classification metrics. Then, we will implement our BruteForceKNN on the same data and compare how it performs with respect to Scikit learn implementation.</p><h3>Testing with Scikit-learn implementation</h3><p>Let’s import the necessary libraries and dependencies.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f7378c5fe8b55eaaf1dc2c39d3bd933/href">https://medium.com/media/4f7378c5fe8b55eaaf1dc2c39d3bd933/href</a></iframe><p>Now, let’s load iris data from Scikit learn datasets. Then, we will split the data and use standard scaling to scale our data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7c78995a2a75a8c460b39f709c5633b/href">https://medium.com/media/f7c78995a2a75a8c460b39f709c5633b/href</a></iframe><p>Now, we need to find the optimal value of ‘<strong>k</strong>’.</p><p>For that, we will instantiate a base KNN classifier with default settings and make predictions for values of ‘k’ ranging from 1 to 31.</p><p>Then we will calculate the train and test errors and plot the <strong>error curve</strong>. With the help of the error curve, we will find the optimal value for ‘k’.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60c05d380ef32fe889b7d4b951b27725/href">https://medium.com/media/60c05d380ef32fe889b7d4b951b27725/href</a></iframe><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*YyTQYKUWuBDuCMT5YCb41w.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><p>From the error curve, we find that k=14 suits well for our data. So, we instantiate a KNN object with k=14, fit the training data, and predict using the test data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cbecef1b44afa64d78a4bf659f065fc/href">https://medium.com/media/3cbecef1b44afa64d78a4bf659f065fc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*wUaqVfsyUv7KWUm_hdvazA.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><p>Now, let’s plot the confusion matrix and check other classification metrics.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cec6f078941603ad924e0a19959b8f29/href">https://medium.com/media/cec6f078941603ad924e0a19959b8f29/href</a></iframe><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*0pAEvYyalchEhAhswx3M9w.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*0uqvcJKRFu12EcSfg5ed4w.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><h3>Testing using our Brute Force KNN</h3><p>Now, we will instantiate our BruteForceKNN class and make predictions with the same data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f0e85079d98183b4641e15cecd61482/href">https://medium.com/media/3f0e85079d98183b4641e15cecd61482/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*tpFV9LwaW4S5cVIh5b_EVA.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><p>This is great. Our model accuracy matches with Scikit-learn implementation. Now let’s check the confusion matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/925b0f3cd21c52dcf46ba23c471dcfbe/href">https://medium.com/media/925b0f3cd21c52dcf46ba23c471dcfbe/href</a></iframe><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*_uyWskc5ARynpq516cTndQ.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*sLd-RKzYJQPv7s2uhzyogQ.png" /><figcaption>Image by <a href="https://retinpkumar.medium.com/">author</a></figcaption></figure><p>We find that our BruteForceKNN model performance matches exactly with the Scikit-learn implementation of the KNN Classifier.</p><p>With that, we have reached the end of this article.</p><p>I hope you’ve enjoyed learning and implementing the KNN algorithm from scratch in Python and comparing the model performance along with Scikit-learn API.</p><p>If you liked this article, do <a href="https://retinpkumar.medium.com/">follow me</a> for more interesting and actionable posts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5eaaf558d49" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/implementing-k-nearest-neighbors-from-scratch-in-python-d5eaaf558d49">Implementing K Nearest Neighbors from scratch in Python</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Analyzed a Data science Job Salaries Dataset]]></title>
            <link>https://medium.com/geekculture/i-analyzed-a-data-science-job-salaries-dataset-dca3c24081c0?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/dca3c24081c0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Tue, 01 Feb 2022 18:00:57 GMT</pubDate>
            <atom:updated>2022-02-02T07:25:32.879Z</atom:updated>
            <content:encoded><![CDATA[<h4>And here’s what I found.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SKW73MbqAb88jwVG" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>So, I found this interesting <a href="https://www.kaggle.com/saurabhshahane/data-science-jobs-salaries"><strong>Data science Job Salaries dataset </strong></a>on Kaggle and thought of trying my analytical skills on it.</p><p>This dataset contains details regarding the salaries of different Data science professionals all over the world based on factors like employment type, company location, etc. for the year 2020. It also comes with the estimated salaries of Data science professionals for the year 2021 as well.</p><p>I plotted the distribution of salaries for the years 2020 and 2021 and this is what I found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SljaSA2xos0Dmpq3aw-Skw.png" /><figcaption>Image by the author(<a href="https://retinpkumar.medium.com/">https://retinpkumar.medium.com/</a>)</figcaption></figure><p>For the period from 2020 to 2021, we can observe a considerable increase in the upper range of salary. Also, a substantial increase in the number of professionals bagging salaries up to 300,000 USD can be observed.</p><p>So, as a Data science aspirant from India, I thought why not give it a try and find out the answers to a few questions that popped up in my mind.</p><p>(Note that the salaries mentioned in the upcoming charts are in Indian Rupees.)</p><h3>Question 1. What type of employment is best suited for an entry-level Data science professional?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bEwDSPABDHgL7MrzeSuI-Q.png" /><figcaption>Image by the author(<a href="https://retinpkumar.medium.com/">https://retinpkumar.medium.com/</a>)</figcaption></figure><p>We find that a full-time role is the best option for an entry-level Data science professional which is obvious.</p><h3>Question 2. Which career option should an entry-level Data science professional from India opt for?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C1rptuyUN27sHECTqtZEFQ.png" /><figcaption>Image by the author(<a href="https://retinpkumar.medium.com/">https://retinpkumar.medium.com/</a>)</figcaption></figure><p>We find that, for Indian Data science aspirants, becoming a Data Scientist would be the best choice, but being a data engineer is not bad either. After all, more than money, the passion for the job matters.</p><h3>Question 3. Whether an entry-level Data science professional in India should try remote work or not?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrQ_iBaXQbh4q2_bK93k7Q.png" /><figcaption>Image by the author(<a href="https://retinpkumar.medium.com/">https://retinpkumar.medium.com/</a>)</figcaption></figure><p>Having a full-time job along with remote work is the best option for an entry-level Data science professional from India.</p><p>Leave aside the salary, it is still the best option for a fresher as the office exposure and experience gained from seniors and peers are priceless.</p><h3>Question 4. Which country pays the highest for an entry-level Data science professional?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XrkjnfmAqYcAWEHhPY1qTw.png" /><figcaption>Image by the author(<a href="https://retinpkumar.medium.com/">https://retinpkumar.medium.com/</a>)</figcaption></figure><p>We find that China with annual pay of 75 lakhs INR, followed by the US and UK would be the best choice for an entry-level data scientist considering only salary as the criteria.</p><h3>Question 5. Based on the size of the company, what type of company should an entry-level Data science professional from India opt for?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWdQ-e0SeuY70yVRKRizPA.png" /><figcaption>Image by the author(<a href="https://retinpkumar.medium.com/">https://retinpkumar.medium.com/</a>)</figcaption></figure><p>This is an interesting finding against my assumptions that a large company paid the highest for any level of Data science professional. A medium-sized company pays 20% more than a large company and hence the best choice for an entry-level Data science professional.</p><p>So, these were some of the interesting observations I found during my analysis of the dataset.</p><p>You can find the complete analysis including dataset exploration in my <a href="https://www.kaggle.com/retinpkumar/data-science-jobs-salaries-dataset-analysis"><strong>Kaggle notebook</strong></a><strong>.</strong></p><p>Feel free to go through the notebook and if you liked it, do consider giving an upvote and post your valuable feedback.</p><p>Also, if you liked this article, do follow me for interesting and actionable data science articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dca3c24081c0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/i-analyzed-a-data-science-job-salaries-dataset-dca3c24081c0">I Analyzed a Data science Job Salaries Dataset</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Extract Reddit Posts for an NLP Project]]></title>
            <link>https://medium.com/geekculture/how-to-extract-reddit-posts-for-an-nlp-project-56d121b260b4?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/56d121b260b4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Fri, 31 Dec 2021 04:32:38 GMT</pubDate>
            <atom:updated>2021-12-31T07:53:21.450Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to extract Reddit posts using python</h3><h4>And use it for your Natural Language Processing related projects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n5eR5dHvyQ1EH-tn" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It&#39;s the year-end, and we are back again with yet another data extraction article. I&#39;m glad that I started writing on medium this year with a<strong> </strong><a href="https://medium.com/geekculture/how-to-build-a-simple-web-scraper-using-beautifulsoup-2f2f0f97dc47"><strong>data scraping post</strong></a> and ending the year with another one.</p><p>And this time, we are going to <strong>extract data from Reddit posts</strong> from your required subreddit. So, without any further ado, let&#39;s begin.</p><p>Now, you should be having the following requirements to complete this hands-on session successfully.</p><ol><li><strong>Python 3.6</strong> or above installed in your system.</li><li>A <strong>Reddit account</strong></li><li>A Reddit account <strong>client id </strong>and <strong>client secret</strong></li><li>A Reddit account <strong>user agent.</strong></li></ol><p>The first two are self-explanatory. But for the client id, client secret, and user agent, follow me along.</p><h3>Getting your client ID and secret</h3><p>Once you have installed Python3.6 or above in your system and you create a Reddit account for yourself, then login into your Reddit account. Then, go to <a href="https://www.reddit.com/prefs/apps">reddit.com/prefs/apps</a></p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*Ftj6KgkJxQ-Nkmtb1PsHAA.png" /><figcaption>Image by Author(<a href="https://medium.com/@retinpkumar">https://medium.com/@retinpkumar</a>)</figcaption></figure><p>Now, as you can see in the image, click on the <strong>‘create an app’</strong> button. In creating the application, select <strong>‘script’ </strong>and then give your app, a name of your choice. You can avoid filling in the description and about URL as it&#39;s not mandatory.</p><p>Now, you have to enter <strong>“http://localhost:8080&quot;</strong> or <strong>“http://localhost: [port number of your choice]”</strong> at redirect uri and press <strong>“create app”</strong>.</p><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*E2L0uF_eu6KbDt6InkPEcw.png" /><figcaption>Image by Author(<a href="https://medium.com/@retinpkumar">https://medium.com/@retinpkumar</a>)</figcaption></figure><p>Now, that you have created your app, you can find your client id at the top left corner, under <strong>“personal use script”</strong>.</p><figure><img alt="image-3.png" src="https://cdn-images-1.medium.com/proxy/1*AO5fqYsntdgddDq0DdbATA.png" /><figcaption>Image by Author(<a href="https://medium.com/@retinpkumar">https://medium.com/@retinpkumar</a>)</figcaption></figure><p>Also, you can see your <strong>“secret” </strong>which is your client secret.</p><h3>Creating the scraper</h3><p>Now, it&#39;s time to put on our developer coat and run the terminal.</p><p>The first thing to do is to install a library called <strong>‘PRAW’ </strong>(Python Reddit API Wrapper). You can do this by running the following command in your terminal.</p><pre>pip install praw</pre><p>Once you’ve done installing PRAW, it&#39;s time to import it into your script.</p><pre>import praw</pre><p>Now, we will create our scraper instance using the Reddit class inside ‘praw’ library.</p><p>For that, let&#39;s first <strong>define the user_agent</strong> that is going to access the data for us. You can give any name you like. Here, I’ve decided to go with <strong>“praw_scraper_1.0”.</strong></p><p>Then, create a Reddit instance as shown below using your Reddit account credentials.</p><p>Note that you only have to specify your username and password if you wish to scrape more than 100 posts.</p><pre># Define user agent<br>user_agent = &quot;praw_scraper_1.0&quot;<br><br># Create an instance of reddit class<br>reddit = praw.Reddit(username=&quot;username&quot;,<br>                     password=&quot;password&quot;,<br>                     client_id=&quot;client id&quot;,<br>                     client_secret=&quot;client secret&quot;,<br>                     user_agent=user_agent<br>)</pre><p>Now, specify the subreddit from which you need to scrape the data from.</p><pre># Create sub-reddit instance<br>subreddit_name = &quot;deeplearning&quot;<br>subreddit = reddit.subreddit(subreddit_name)</pre><p>Confirm if your subreddit is correct by displaying it.</p><pre># Printing subreddit info<br>print(subreddit.display_name)</pre><p>Now, we will be using our subreddit object for extracting data</p><p>Let&#39;s import the pandas library and create a data frame.</p><pre>import pandas as pd<br><br>df = pd.DataFrame()</pre><p>If you don&#39;t have pandas installed in your system, you can use the following code to do so.</p><pre>pip install pandas</pre><p>Now, we need to know the various types of posts that you can find on Reddit. These can be classified into <strong>new, top, hot, controversial, rising, and gilded.</strong></p><p>If you don’t have a clue as to what all items you can scrape, run the following code and pass in your subreddit object. It will list out all possible attributes you could scrape using the object.</p><pre>vars(submission)</pre><p>For this post, I&#39;m going to scrape the title, id, and upvotes for 21 posts under the <strong>‘new’ </strong>category. For that, let&#39;s create 3 lists that will be storing the scraped details for each post.</p><pre>titles=[]<br>scores=[]<br>ids=[]</pre><p>Now, for accessing the new posts, we should create a generator object using our subreddit instance and loop over all the posts we wish to scrape.</p><p>While creating the object, we can pass in a parameter called <strong>“limit” </strong>that will limit the number of posts we wish to scrape.</p><p>But, there is a caveat.</p><p>Remember, when we created our scraper app, we had mentioned our user id and password to scrape as many posts as we would want to.</p><p>We can do that by setting the parameter <strong>“limit=None”.</strong></p><p>By default, if you don’t mention the limit parameter, the scraper will get you the details of 100 posts. Since we are limiting our posts to 21, we will pass <strong>“limit=21”</strong> in our object.</p><pre>for submission in subreddit.new(limit=21):<br>    titles.append(submission.title)<br>    scores.append(submission.score) #upvotes<br>    ids.append(submission.id)</pre><p>Now, we have extracted the required details into our lists, we will create a data frame and view our collected data.</p><pre>df[&#39;Title&#39;] = titles<br>df[&#39;Id&#39;] = ids<br>df[&#39;Upvotes&#39;] = scores #upvotes<br><br>print(df.shape)<br>df.head(10)</pre><p>Putting it all together, we have…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db432b818fe6c44c260f6141b0a27a41/href">https://medium.com/media/db432b818fe6c44c260f6141b0a27a41/href</a></iframe><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*qkPTMTYlBF2uZberHR5Nvw.png" /><figcaption>Image by Author(<a href="https://medium.com/@retinpkumar">https://medium.com/@retinpkumar</a>)</figcaption></figure><p>There you go. And that’s it for this hands-on post.</p><p><strong>But wait, there’s more…</strong></p><p>What if you want to extract details from the URL of a post? Let&#39;s see how it&#39;s done.</p><h3>Extracting data from a URL</h3><p>Create a url variable for the url you wish to scrape data from.</p><pre>input_url = &quot;https://www.reddit.com/r/europe/comments/r0hthg/sweden_is_taking_the_lead_to_persuade_the_rest_of/&quot;</pre><p>This is a random URL that I’ve grabbed from Reddit.</p><p>If you look at the structure of any Reddit URL, it looks like this:</p><pre>https://www.reddit.com/r/ &lt;subreddit name&gt; / comments / &lt;user id&gt; / &lt;title&gt; /</pre><p>We can use string splitting and extract the subreddit name, user id, and title from the URL.</p><p>Once you have these, you can use the subreddit name in our scraper and append the title to our final data frame.</p><p><strong>But what about the user id?</strong></p><p>We can use the user id to extract more details using the following code:</p><pre>userid_from_url = input_url.split(&quot;/&quot;)[6]<br>more_details = reddit.submission(id=userid_from_url)</pre><p>Pass the object in the <strong>vars()</strong> function and you will get the dictionary object containing the attributes and values corresponding to your user id. You can append this info to your final data frame as well.</p><p>So, that’s the end of this post. Hope you liked it.</p><p><strong><em>Wishing you and your family a wonderful, prosperous, and joyful new year, and may you scale new heights to achieve everything you wish for.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KrFYBY9u1u-Y3TBq" /><figcaption>Photo by <a href="https://unsplash.com/@sincerelymedia?utm_source=medium&amp;utm_medium=referral">Sincerely Media</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56d121b260b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-extract-reddit-posts-for-an-nlp-project-56d121b260b4">How to Extract Reddit Posts for an NLP Project</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Custom Datasets within Seconds]]></title>
            <link>https://medium.com/geekculture/create-custom-datasets-within-seconds-53abf103bf8b?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/53abf103bf8b</guid>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 07:26:25 GMT</pubDate>
            <atom:updated>2021-12-08T18:20:35.904Z</atom:updated>
            <content:encoded><![CDATA[<h3>Here’s How You Can Create Custom Datasets in a Second</h3><h4>Build the perfect dataset to test your model within seconds</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PrRRhIaPlftodPX9" /><figcaption>Photo by <a href="https://unsplash.com/@mbaumi?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a beginner, it might be a bit daunting to start working on an already existing dataset.</p><p>Or maybe you want to test a model real quick or practice building a model on the latest algorithm you’ve learned.</p><p>But you don’t want to waste your time behind data cleaning and stuff.</p><p>This article will help you build your own custom dataset within the blink of an eye and you can start practicing as soon as you want.</p><p>In this article, we will be making our own custom function that generates random datasets according to our requirements. We will create this function with the help of the Sklearn library.</p><p>We will be using functions defined for creating random datasets as defined in the <a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets">Sklearn datasets module.</a></p><p>So, let&#39;s begin.</p><h3>Creating Custom dataset</h3><p>We will start by importing the make_regression() method from the Sklearn module for creating a dataset for regression.</p><pre>from sklearn.datasets import make_regression</pre><p>Then we will create a regression object by passing the required parameters.</p><p>For now, we will pass arguments for creating a dataset with 100 samples, 4 features, and 1 target.</p><pre>reg = make_regression(n_samples=100, n_features=4, n_targets=1)<br>type(reg)</pre><pre>tuple</pre><p>Now, that our dataset is ready, but it is in the form of a tuple of arrays.</p><p>We need to convert it to a data frame for easy processing.</p><p>So, we will define our columns first.</p><pre>columns = [f&quot;F{i}&quot; for i in range(1, 7)] #defining our column names</pre><p>Now, let&#39;s generate our data frame.</p><pre># features<br>features = pd.DataFrame(reg[0], columns=columns)<br><br># target<br>target = pd.DataFrame(reg[1], columns=[&#39;Target&#39;])<br><br># checking dataframe shape<br>print(&quot;Dataset shape: &quot;, features.shape, target.shape)</pre><pre>Dataset shape:  (10000, 6) (10000, 1)</pre><p>Now, let&#39;s concatenate features and target dataframe to obtain our final data frame.</p><pre># concatenating features and target.<br>df_reg = pd.concat([features, target], axis=1)<br>df_reg.head() # looking at first 5 observations</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*pj6CtgfDEoN1Kkq_SV6z_g.png" /></figure><p>That’s it. Our custom dataframe is ready and now you can start practicing with your machine learning models real quick.</p><p><strong>But wait, what about datasets for other tasks?</strong></p><p>No worries. I have made a custom function for you all which you can copy-paste in your terminal/window/notebook wherever you like.</p><h3>Custom Function for Creating Custom Dataset</h3><p>You can make use of this function to generate your custom datasets at ease.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/446c2abd52625d527ec306cd0d1f102e/href">https://medium.com/media/446c2abd52625d527ec306cd0d1f102e/href</a></iframe><p>You should provide the required number for the problem to generate the corresponding dataset.</p><p>For example, to make a dataset for regression, type</p><pre>makedf(1).head()</pre><pre>Dataset shape:  (10000, 6) (10000, 1)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*7xiSEQWYUpjuZeZEtB8caw.png" /></figure><p>For a classification dataset,</p><pre>makedf(3).head()</pre><pre>Dataset shape:  (10000, 6) (10000, 1)</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*aOcm0cRRxq4rmwteTHnDfA.png" /></figure><p>Similarly, you can pass the argument and get your required dataset.</p><p>Hope you liked the post.</p><p>Follow me for more actionable data science and machine learning content.</p><p>You can visit my GitHub profile at: <a href="https://github.com/Retinpkumar">https://github.com/Retinpkumar</a> for accessing the code files related to my blog posts.</p><p>Thank you and have a nice day.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53abf103bf8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/create-custom-datasets-within-seconds-53abf103bf8b">Create Custom Datasets within Seconds</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Let’s Build and Deploy Your First Machine Learning App]]></title>
            <link>https://medium.com/geekculture/lets-build-and-deploy-your-first-machine-learning-app-fa350ec6b5cf?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa350ec6b5cf</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Fri, 03 Dec 2021 08:13:35 GMT</pubDate>
            <atom:updated>2022-03-16T03:51:41.269Z</atom:updated>
            <content:encoded><![CDATA[<h3>Quickly Build and Deploy a Machine Learning App</h3><h4>A step-by-step guide to building and deploy your machine learning model using Python, Streamlit, and Heroku</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O5-Bv8ylSyJ-Mof7" /><figcaption>Photo by <a href="https://unsplash.com/@santesson89?utm_source=medium&amp;utm_medium=referral">Andrea De Santis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It&#39;s a great day and you’re now ready to build and deploy your first machine learning app.</p><p>But you’re skeptical about it.</p><p>No worries, today we will together build a machine learning app from scratch and deploy it on a cloud platform.</p><p>This article will help you build a machine learning app quickly using Streamlit and deploy it on the Heroku platform. This will help boost your confidence in building more machine learning apps with your own unique and innovative ideas.</p><p>So let’s begin.</p><p>Note that this article will have 3 parts:</p><p>Part 1: Building a model from scratch</p><p>Part 2: Creating an app for the model</p><p>Part 3: Deploying the model</p><h3>Part 1: Building a model from scratch</h3><p>We will be using the classic Iris Flower dataset from Scikit learn library for creating our Iris Flower Prediction App.</p><p>On to your jupyter notebooks folks.</p><p>Let’s import some libraries first.</p><p>In case, you don’t have any of these libraries installed in your system, you can do either of the following to get them:</p><p>Create a new cell in your jupyter notebook and type</p><pre>!pip install “libraryname”</pre><p>(don’t use the double quotes while mentioning the library name)</p><p>For example, if you want to install pandas, type</p><pre>!pip install pandas</pre><p>Open your system terminal/command prompt from the project directory where you have downloaded and saved the project files and type</p><pre>pip install -r requirements.txt</pre><p>You can find the files for this article at: <a href="https://github.com/Retinpkumar/Iris-flower">https://github.com/Retinpkumar/Iris-flower</a></p><p>Once you’re done, let&#39;s proceed with importing the required libraries for creating our app.</p><pre># data manipulation<br>import numpy as np<br>import pandas as pd<br><br># data transfer<br>import joblib<br><br># raw dataset<br>from sklearn.datasets import load_iris</pre><p>Now, let&#39;s load our dataset and view the keys corresponding to it.</p><pre>iris = load_iris()<br>iris.keys()</pre><p>We find that the dataset is being stored in the form of a dictionary inside the Sklearn datasets module. Let&#39;s access a few keys to get our required data.</p><p>‘DESCR’ key gives a detailed description of the dataset. But in order to view it in a proper format, let’s split the string with respect to newline char.</p><pre>iris.DESCR.split(&quot;\n&quot;)</pre><p>Wow. That’s a lot of information.</p><p>Feel free to go through it when you find the time. This same works for all the datasets within the Sklearn package.</p><p>Now let&#39;s create our data frame of features by accessing the ‘data’ variable</p><pre># creating dataframe<br>data = pd.DataFrame(iris.data, columns=iris.feature_names)</pre><pre># printing observations<br>print(f&quot;The data has {data.shape[1]} features and {data.shape[0]} observations.&quot;)</pre><pre># checking first 5 observations<br>data.head()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*9cjzbJ0DEomhCIXqrl9jlA.png" /><figcaption>Iris dataset (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Our data column looks a bit complicated. Let’s simplify it by creating a dictionary map.</p><pre>data = data.rename(columns={<br>    &#39;sepal length (cm)&#39;:&#39;sepal_length&#39;, <br>    &#39;sepal width (cm)&#39;:&#39;sepal_width&#39;, <br>    &#39;petal length (cm)&#39;:&#39;petal_length&#39;,<br>    &#39;petal width (cm)&#39;:&#39;petal_width&#39;<br>})<br>data.head()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*K_d_F1LpAb0ofaFBHnSUkA.png" /><figcaption>Mapped columns (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Now, our data frame looks great. We can save it for further reference, but it&#39;s not required.</p><pre># save data if needed<br>data.to_csv(&#39;data.csv&#39;, index=False)</pre><p>Now that we have our input features, let’s explore our target feature.</p><pre># explore target<br>iris.target</pre><p>Seems like we have 3 different types of target classes (which is obvious).</p><p>So,</p><p>0 represents “Iris-Setosa”,</p><p>1 represents “Iris-Versicolour”, and</p><p>2 represents “Iris-Virginica”.</p><pre># add target to dataframe<br>data[&#39;Target&#39;] = iris.target<br>data.head()</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*2as6ATXSQobnwYWw1vYwCA.png" /><figcaption>Datset with target (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Let’s have a check at the target composition</p><pre># check target composition<br>data[&#39;Target&#39;].value_counts()</pre><pre>0    50<br>1    50<br>2    50<br>Name: Target, dtype: int64</pre><p>That looks nice. We have equal representation from all the classes in our target feature.</p><p>Let’s have a quick check if any value is missing from our data.</p><pre># check missing values<br>data.isnull().any()</pre><pre>sepal_length    False<br>sepal_width     False<br>petal_length    False<br>petal_width     False<br>Target          False<br>dtype: bool</pre><p>Great... We don’t have any missing values in our dataset.</p><p>We will now split our data set into target feature ‘y’ and input features ‘X’.</p><pre># split data<br>y = data[&#39;Target&#39;]<br>X = data.drop(columns=[&#39;Target&#39;], axis=1)</pre><p>Now we will split our data into training and testing datasets.</p><p>We will create a test set that will be almost 25% of the total data size.</p><pre># create train and test sets<br>from sklearn.model_selection import train_test_split<br>train_X, test_X, train_y, test_y = train_test_split(X, y, test_size=0.25, random_state=7)<br><br>print(train_X.shape, test_X.shape, train_y.shape, test_y.shape)</pre><pre>(112, 4) (38, 4) (112,) (38,)</pre><p>We will be using a Logistic regression model to make the classification prediction and hence we will be scaling our input features to make it easy for the algorithm to make predictions.</p><p>We will scale our training data first by calling the fit_transfer method so that the fit method will capture the parameters from the training data for transforming the testing data.</p><p>We can use the fit_transform method even for testing data as well, but if the testing data has only one row of input, then the fit method will fail to capture the parameters from the testing data.</p><p>Hence we use the parameters from the training data to scale the testing data.</p><p>Make sure that you save your scalar file after you have scaled your training data and use that to transform your testing data.</p><pre>from sklearn.preprocessing import StandardScaler<br>scaler = StandardScaler()<br><br>train_X_scaled = scaler.fit_transform(train_X) # scaling training data<br><br>#save scaler<br>joblib.dump(scaler, &#39;standard_scaler.sav&#39;)<br><br>test_X_scaled = scaler.transform(test_X) # scaling testing data</pre><pre>[&#39;standard_scaler.sav&#39;]</pre><p>Let’s now import a Logistic Regression model from the Sklearn library and create a model object.</p><pre>from sklearn.linear_model import LogisticRegression<br><br>model = LogisticRegression()</pre><p>We are dealing with a multi-class classification problem where our target feature has three distinct classes.</p><p>By default, the Logistic regression model in Sklearn Library is enabled to handle multi-class classification tasks with the parameter <strong>multi_class</strong> being set to <strong>“auto”</strong> which will use the <strong>“multinomial”</strong> method for performing multi-class classification by default.</p><p>Hence for now we will proceed with a base model of Logistic regression for our problem.</p><p>For further reference, you can visit the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#:~:text=%E2%80%98auto%E2%80%99%20selects%20%E2%80%98ovr%E2%80%99%20if%20the%20data%20is%20binary%2C%20or%20if%20solver%3D%E2%80%99liblinear%E2%80%99%2C%20and%20otherwise%20selects%20%E2%80%98multinomial%E2%80%99.">following link</a></p><p>Let’s now fit our model and make predictions</p><pre># fitting model<br>model.fit(train_X_scaled, train_y)</pre><pre>LogisticRegression()</pre><pre># Making predictions using test data<br>y_pred = model.predict(test_X_scaled)</pre><p>Now that we have made predictions using our test dataset, we will now calculate the accuracy metrics to see if our model is working well enough.</p><p>Since our objective is just to create a model and deploy it, we will not be proceeding with improving our accuracy or any other classification metrics.</p><pre># calculating accuracy<br>from sklearn.metrics import accuracy_score<br><br>accuracy = accuracy_score(test_y, y_pred)<br>accuracy</pre><pre>0.8947368421052632</pre><p>Our model is working pretty fine.</p><p>So we will now save our model file so that we can import this file to create our app for deployment.</p><pre># save model<br>joblib.dump(model, &#39;model.sav&#39;)</pre><pre>[&#39;model.sav&#39;]</pre><h3>Part 2: Creating an app for the model</h3><p>You know how I created and arrange the project files in the project directory do read this article on<strong> </strong><a href="https://medium.com/geekculture/how-to-organize-a-data-science-project-directory-aa4d7c9f6156"><strong>How to Organize your Data Science Project files</strong></a></p><p>In this section, we will be focusing on creating our three main app files:</p><ol><li><strong>app.py</strong></li><li><strong>model.py</strong></li><li><strong>scalar.py</strong></li></ol><p>Let’s begin with <strong>“model.py”</strong></p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*Z8IOLxrqM5TRZzHJ4L889Q.png" /><figcaption>Creating model file (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>This file will be used for making predictions using the input data from the user.</p><p>So we will create a function that takes in user input as its argument and returns a predicted value with the help of our saved model.</p><p>Now, we will create our <strong>“scalar.py”</strong></p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*OhQKMlgBvDoepqBpOIGjCg.png" /><figcaption>Creating scaler file (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>This file takes the input dataframe as its argument and returns a scaled input dataframe using our saved scalar object.</p><p>And finally, its time to create our “app.py”</p><p>We will be using Streamlit library for making our app.</p><p>Since we fed a dataframe as the input for creating the scalar object, we have to import pandas library to create a dataframe of user inputs. This dataframe has to be fed to our scalar object to generate a scaled version of user inputs for making the final prediction.</p><p>Hence we will import Streamlit and Pandas along with our model files in our <strong>“app.py”</strong></p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*PSAJQ0khOPlYPubiaBe8wQ.png" /><figcaption>Importing libraries (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>We will set our app display page settings using st.<strong>set_page_config</strong> method from the streamlit library.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*_asfj0jf82N-zQ7_uQvakw.png" /><figcaption>Setting app page configuration (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Use <strong>st.title</strong> to set a title text for the app form.</p><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*r4uELQKMnYpddymA35tDmA.png" /><figcaption>Giving app title (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Now, we will use <strong>st.form</strong> to create our user input form.</p><p>Inside the form, we will create a form header using <strong>st.header</strong> and input elements using <strong>st.number_input</strong> and <strong>st.slider</strong>.</p><p>Using input text field or slider object is completely your choice. I have implemented both just for the purpose of demonstration.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*dqR-hadnyU_5e1GLpQjDwA.png" /><figcaption>Creating app form (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>So now that we have our form ready, let&#39;s proceed to process inputs and make predictions.</p><p>This should be self-explanatory, except for the <strong>st.balloons </strong>method which are nothing but a cool graphical illustration for our successful prediction.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*M3vNkN92G3jEEQApdQfIOQ.png" /><figcaption>Complete app (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>If you’ve made it this far, pat yourself on the back.</p><p>You’ve now successfully made your first machine learning model for predicting Iris flower classes. Yay…</p><p>Now, to check if your app is working, open a terminal/command prompt in the project directory or use the CLI in your IDE and type</p><pre>streamlit run app.py</pre><p>Hurray… Now you can interact with your app in your default browser and make predictions.</p><p>Now, we will move on to part 3 where we will deploy our app on a cloud platform. For this, we will make use of GitHub and Heroku.</p><p>But before that, you would require a few other files as well.</p><h4>Starting with the <strong>“requirements.txt”</strong> file for installing libraries.</h4><p>For creating the “requirements.txt” file, open the terminal in your project directory and type</p><pre>pip install pipreqs</pre><p>to install <a href="https://pypi.org/project/pipreqs/"><strong>pipreqs</strong></a> library. After you’re done with the installation, in your terminal, type</p><pre>pipreqs --encoding=utf8  --debug  &quot;path/to/project&quot;</pre><p>Make sure to enter the actual path to your project directory in the place of <strong>“path/to/project”</strong>.</p><p>This will automatically create <strong>“requirements.txt” </strong>for your project.</p><h4>Other 3 important files</h4><p>Now, inside your directory where you have your <strong>“app.py”</strong> and <strong>“requirements.txt”</strong> files, create 3 more files named: <strong>Procfile, runtime.txt</strong>, and <strong>setup.sh</strong></p><p>Inside, <strong>“setup.sh”</strong>, type the following and save:</p><pre>mkdir -p ~/.streamlit/<br>echo &quot;\<br>[general]\n\<br>email = \&quot;your-email@domain.com\&quot;\n\<br>&quot; &gt; ~/.streamlit/credentials.toml<br>echo &quot;\<br>[server]\n\<br>headless = true\n\<br>enableCORS=false\n\<br>port = $PORT\n\<br>&quot; &gt; ~/.streamlit/config.toml</pre><p>Now, open your <strong>“Procfile”</strong>, and enter this and save:</p><pre>web: sh setup.sh &amp;&amp; streamlit run app.py</pre><p>Then open your <strong>“runtime.txt”</strong> and enter your Python version as follows:</p><pre>python-3.8.8</pre><p>If you’re unsure about your python version, open your terminal and type:</p><pre>python --version</pre><p>Copy the text and paste. As simple as that.</p><p>Now over to deploying our model on a cloud platform.</p><h3>Part 3: Deploying the Model</h3><p>I will be skipping the part where you create a GitHub repo here as I assume that you know how to create a Github repository.</p><p>So, assuming that you have created your GitHub repo and uploaded all your project files from your local git to the project repo, now visit <a href="https://dashboard.heroku.com/apps">https://dashboard.heroku.com/apps</a> and create a <strong>“New app”</strong> as follows:</p><figure><img alt="image-3.png" src="https://cdn-images-1.medium.com/proxy/1*lUvMZJFIAwkYUNC332r4BA.png" /><figcaption>Creating new app in Heroku (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Give your app a name and click <strong>Create App</strong></p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*dOuHo2iQAgSqBuRrPyUFbQ.png" /><figcaption>Giving app name (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>You will be now moved to a page that looks like this</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*9z2TzJG9oupybiXQjgGUeA.png" /><figcaption>Selecting deployment pipeline (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Click on that GitHub icon, enter your GitHub repo name in the search field and click “Search”. Then press “Connect”.</p><p>Now, you will see a few options that look like this:</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*stNOwZgs7PtqKIKyuzT0PQ.png" /><figcaption>Enabling automatic deploy (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Click <strong>“Enable Automatic Deploys”</strong> as this will make changes to your app as you modify your files related to the app in your GitHub repo.</p><p>Then, click <strong>“Deploy Branch” </strong>and wait for some time.</p><p>Meanwhile, you can enjoy watching your app getting deployed through this window.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*ZWQvi2g8PkJzrzZKxVC2fQ.png" /><figcaption>Deploying the app (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>After successful deployment, you will get to see something like this:</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*-E5_WiehGg9XPZffhK1Hlw.png" /><figcaption>App successfully deployed (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Yay…</p><p>No. Wait...</p><p>Most of the time, you will receive an error and be redirected to this page:</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*H2zhVrR4UOf46jeBvFXILA.png" /><figcaption>Deployment error (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>But don’t worry.</p><p>That’s just an <strong>error:503</strong> which means the Heroku server isn’t ready to host your app. So, just give it some time and check back later. It will be deployed for sure.</p><p>For personal reasons, I will not be deploying my app, but you can get the codes in my GitHub repo <a href="https://github.com/Retinpkumar/Iris-flower">here</a>.</p><p>And if you could successfully get it deployed without any error, then Cheers!!!…</p><p>After successful deployment, the app should now look like this:</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*8kCnt3Aavw9Kcxdk7BtOHg.png" /><figcaption>Final app (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Congrats!! Now you have successfully built a model from scratch and deployed it on a cloud platform.</p><p>And I hope you liked the article. Follow me for more actionable content and feel free to post your comments and valuable feedback</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa350ec6b5cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/lets-build-and-deploy-your-first-machine-learning-app-fa350ec6b5cf">Let’s Build and Deploy Your First Machine Learning App</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Organize a Data Science Project Directory]]></title>
            <link>https://medium.com/geekculture/how-to-organize-a-data-science-project-directory-aa4d7c9f6156?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa4d7c9f6156</guid>
            <category><![CDATA[machine-learning-tools]]></category>
            <category><![CDATA[data-science-projects]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Wed, 01 Dec 2021 04:04:04 GMT</pubDate>
            <atom:updated>2022-02-10T00:18:52.371Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Organize your Data Science Project files</h3><h4>Make them presentable and easier to navigate and search</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e84iedtfKF8WT1bb" /><figcaption>Photo by <a href="https://unsplash.com/@viktortalashuk?utm_source=medium&amp;utm_medium=referral">Viktor Talashuk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It might be hard for a beginner in Data science to structure their project files in an ordered fashion. A well-organized Data science project directory not only helps with searching and navigating through the contents, but also increases the visual appeal for the readers.</p><p>This article will help you structure your Data science project files in a meaningful and ordered manner and make them more presentable in your profile. This will help your recruiters and other interested people access your content and understand your projects easily.</p><p>Not only that, but it will also make them presentable in your GitHub repos as well.</p><p>So, without further ado let’s begin.</p><h3>Creating a project directory</h3><p>We will start by creating our main project directory. Inside this directory, we will be creating sub-directories to store our project files in an ordered fashion.</p><figure><img alt="2021-12-01%2007_55_40-TestProject.png" src="https://cdn-images-1.medium.com/proxy/1*UA00hNgF7vBhvEZ5E06YMA.png" /><figcaption>Creating project directory (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Once inside your project directory, we will start by creating our main app file. It&#39;s better to always start with your main app file so that you have an initial point of reference with which you can later modify the directory structure if needed.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*mLIOnmckt2aS3GC80-ptgg.png" /><figcaption>Creating new python file (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>You can give any name to your main app file. (usually depends on the platform where you intend to deploy). But, for the time being, we will name it “app.py”</p><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*OtDgEoPNnB7jZYU7E9OqYQ.png" /><figcaption>Creating main app file (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Once you are done with creating your main app file, let’s now move on with creating sub-directories.</p><h3>Creating sub-directories</h3><p>Now, we have to create sub-directories within our main directory that holds our main file “app.py”.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*s2IVcxleHf91Qx8ieDs9Uw.png" /><figcaption>Creating sub-directories (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h4>For storing your utilities</h4><p>Let’s create a <strong>“utils” </strong>directory to store our utilities including custom modules, unit tests, and logging modules.</p><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*a9q9bVuBZKu6O9owsTc7yQ.png" /><figcaption>Creating utils folder (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>As I mentioned, after creating the <strong>“utils”</strong> directory, now create separate directories as follows:</p><p><strong>1. Create an “application_logger” for storing your custom “app_logger.py” module.</strong></p><p>Inside application_logger, you can create your custom logger file.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*xzjBgJQHSWgyJkGIvjibFQ.png" /><figcaption>Creating an imaginary logger file (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p><strong>2. Create a “models” directory for storing your custom modules that aid in processing your data.</strong></p><p>Now back to the <strong>“utils” </strong>directory.</p><p>Inside the <strong>“models” </strong>directory, you can save your own python modules containing functions for data processing, main model, etc.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*ThBr9Iz0msiEVOCaQrzR6g.png" /><figcaption>Creating a sub-directory for models (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p><strong>3. Create a “tests” directory for storing your custom unit test module.</strong></p><p>Inside the <strong>“tests”</strong> directory, create a <strong>test_app</strong> file to store your unit tests.</p><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*53cXYHKOoakjEftnTBWuIw.png" /><figcaption>Creating a sub-directory for testing (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Now, that you have finished creating your “<strong>utils”</strong> sub-directory, your project directory should look like this.</p><figure><img alt="image-3.png" src="https://cdn-images-1.medium.com/proxy/1*nSfhI-oaXC5AKfd_bxksgg.png" /><figcaption>Creating utils completed (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><h4>For storing your saved files</h4><p>Now let’s create a directory for storing your saved files.</p><p>Create a “<strong>data”</strong> directory for storing your training and testing data.</p><p>With your training and validation data, it should now look like this.</p><figure><img alt="image.png" src="https://cdn-images-1.medium.com/proxy/1*tA2LHfv0m-gEbzasyZNMNA.png" /><figcaption>Creating directories for storing saved files (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Now back to the “<strong>saved_files” </strong>directory<strong> </strong>and inside it, create sub-directories for storing your <strong>images</strong> and <strong>plots.</strong></p><p>Also, create another directory named <strong>“logfiles”</strong> for storing your log files. Note, the files inside of the “<strong>logfiles”</strong> directory are just example files. Create your own custom log files according to your requirements.</p><p>After you’ve done it, it should now look like this.</p><figure><img alt="image-2.png" src="https://cdn-images-1.medium.com/proxy/1*NOryldjLH2MOeRAILBy6iQ.png" /><figcaption>Creating logfiles sub-directory (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Also, don’t forget to add your “<strong>requirements.txt”</strong> file as well. But this can be placed along with your main app.py file.</p><figure><img alt="image-4.png" src="https://cdn-images-1.medium.com/proxy/1*g3nwAZysg_BOtvaJuX2PmA.png" /><figcaption>Creating requirements.txt (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Now that we have finished organizing our project directory, it should look something like this though naming the directories and files can be done as per your preference.</p><figure><img alt="image-5.png" src="https://cdn-images-1.medium.com/proxy/1*IM6zgvaj48pyymPOkgUvtA.png" /><figcaption>Final completed structure (image by <a href="https://medium.com/@retinpkumar">author</a>)</figcaption></figure><p>Congrats, if you’ve made it till the end...</p><p>Now that you know how to structure your projects in an ordered way, you can now impress your peers with your unique project ideas in a beautifully structured manner.</p><p>Hope you liked the post. <a href="https://medium.com/@retinpkumar">Follow me</a> for more actionable content and feel free to comment with your valuable feedback and suggestions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa4d7c9f6156" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-organize-a-data-science-project-directory-aa4d7c9f6156">How to Organize a Data Science Project Directory</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a simple Web scraper using Beautifulsoup]]></title>
            <link>https://medium.com/geekculture/how-to-build-a-simple-web-scraper-using-beautifulsoup-2f2f0f97dc47?source=rss-e06b7a5f1c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f2f0f97dc47</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Retin P Kumar]]></dc:creator>
            <pubDate>Tue, 16 Nov 2021 03:07:39 GMT</pubDate>
            <atom:updated>2022-02-10T05:13:56.070Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>A step-by-step guide to building a web scraper using Beautifulsoup with Python.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LmI_3tyeHycq0Q97" /><figcaption>Photo by <a href="https://unsplash.com/@ilyapavlov?utm_source=medium&amp;utm_medium=referral">Ilya Pavlov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Data collection is one of the major steps that come after defining and analyzing a problem in a data science project. Though there are multiple ways to collect data, in this article we will focus on web scraping techniques to collect data for our data science projects.</p><p>Building a universal web scraper is neither feasible nor desirable. Every web scraper should be made for a particular situation and should be modified according to the changes in the page to be scraped.</p><p>In this article, we will focus on creating a simple web scraper class using Beautifulsoup and Requests library that can scrape — by default, links, and images — any static webpage. And of course, we will try to scrape a static webpage as well.</p><h3>Defining the steps</h3><p>We will make use of OOP with python to create this scraper from scratch.</p><p>Let’s list out the functions we require to build this scraper</p><p><strong>1. A method for parsing the URL</strong></p><p>This method should create a custom object that could be used to access any web element on the given page.</p><p><strong>2. A method for fetching all links from the webpage</strong></p><p><strong>3. A method for fetching all image links from the webpage</strong></p><p><strong>4. A method for downloading images from the image links to your current directory.</strong></p><p>Now that we have listed the core functionalities of this scraper, let’s build the scraper step by step.</p><h3>Building the scraper</h3><p>Let’s create the scraper class and define our methods. We will call this scraper by the name “StaticSiteScraper”.</p><pre><strong>class</strong> <strong>StaticSiteScraper</strong>:<br>    <strong>def</strong> <strong>__init__</strong>(self):<br>        <strong>pass</strong><br><br>    <em># A method for parsing the url</em><br>    <strong>def</strong> <strong>url_parse</strong>(self):<br>        <strong>pass</strong><br><br>    <em># A method for fetching all links from the webpage</em><br>    <strong>def</strong> <strong>get_all_links</strong>(self):<br>        <strong>pass</strong><br><br>    <em># A method for fetching all image links from the webpage</em><br>    <strong>def</strong> <strong>get_all_image_links</strong>(self):<br>        <strong>pass</strong><br><br>    <em># A method for downloading images from the image links</em><br>    <strong>def</strong> <strong>download_images</strong>(self):<br>        <strong>pass</strong></pre><h3>1. Fetching the HTML page content</h3><p>Let’s define the function for fetching the HTML page.</p><p>We will use the requests library to fetch the HTML content from the URL and store it in a variable called data.</p><pre>data <strong>=</strong> requests.get(url).text</pre><p>Then, we will create a soup object that will parse this HTML content into readable text format.</p><pre>soup <strong>=</strong> bs(data, parser)</pre><p>And finally our function will return this soup object.</p><p>Putting it all together, we have:</p><pre><strong>def</strong> <strong>url_parse</strong>():<br>    <strong>data</strong> <strong>=</strong> requests.<strong>get</strong>(url).text<br>    soup <strong>=</strong> bs(<strong>data</strong>, parser)<br>    <strong>return</strong> soup</pre><p>The soup object created can be used to access any element within the parsed HTML content.</p><p>For example, we can access the title text of the webpage using the following:</p><pre><strong>soup</strong>.title.text</pre><p>Similarly, you can use the soup object to access any element you need.</p><p>Now that we have our soup object, we can use this object to create methods for fetching all the links and images from the parsed HTML content.</p><h3>2. Fetching all the links</h3><p>Let’s start by defining a list for storing all the links we collect using the method. This list will be the output for our method.</p><p>We know that links are stored in the &lt;a&gt; tag within an HTML page.</p><p>So, to access the links, we need to access the &lt;a&gt; tag first which can be done using:</p><pre><strong>soup</strong>.find_all(&#39;a&#39;)</pre><p>This will fetch us a list of all the &lt;a&gt; tags in the parsed HTML content.</p><p>Now, we will have to iterate through each of the elements within this list and extract the link from the “href” attribute. Then, we will store the collected links in the list that we had defined earlier.</p><pre><strong>for</strong> tag <strong>in</strong> a_tags:<br>    items <strong>=</strong> str(tag).<strong>split</strong>(&quot; &quot;)<br><br>    <strong>for</strong> item <strong>in</strong> items:<br>        <strong>if</strong> &#39;href&#39; <strong>in</strong> item:<br>            link_item <strong>=</strong> item.<strong>split</strong>(&quot;=&quot;)[1].<strong>split</strong>(&quot;\&quot;&quot;)[1]<br>            <strong>if</strong> &#39;http&#39; <strong>in</strong> link_item <strong>or</strong> &#39;www&#39; <strong>in</strong> link_item:<br>                links.append(link_item)</pre><p>Putting it all together, we have:</p><pre><strong>def</strong> <strong>get_all_links</strong>():<br>    links<strong>=</strong>[]<br><br>    a_tags <strong>=</strong> soup.find_all(&#39;a&#39;)<br><br>    <strong>for</strong> tag <strong>in</strong> a_tags:<br>        items <strong>=</strong> str(tag).<strong>split</strong>(&quot; &quot;)<br>        <strong>for</strong> item <strong>in</strong> items:<br>            <strong>if</strong> &#39;href&#39; <strong>in</strong> item:<br>                link_item <strong>=</strong> item.<strong>split</strong>(&quot;=&quot;)[1].<strong>split</strong>(&quot;\&quot;&quot;)[1]<br>                <strong>if</strong> &#39;http&#39; <strong>in</strong> link_item <strong>or</strong> &#39;www&#39; <strong>in</strong> link_item:<br>                    links.append(link_item)<br>    <strong>return</strong> links</pre><h3>3. Fetching all image links</h3><p>This step is exactly similar to the step above where we had collected all the links from the parsed content.</p><p>The only difference here is that instead of &lt;a&gt; tag, we use&lt;img&gt; tag here. And also, instead of “href”, we use “src” attribute for accessing the image links.</p><pre><strong>def</strong> <strong>get_all_image_links</strong>():<br>    img_links<strong>=</strong>[]<br><br>    img_tags <strong>=</strong> soup.find_all(&#39;img&#39;)<br>    <strong>for</strong> tag <strong>in</strong> img_tags:<br>        items <strong>=</strong> str(tag).<strong>split</strong>(&quot; &quot;)<br>        <strong>for</strong> item <strong>in</strong> items:<br>            <strong>if</strong> &#39;src&#39; <strong>in</strong> item:<br>                link_item <strong>=</strong> item.<strong>split</strong>(&quot;=&quot;)[1].<strong>split</strong>(&quot;\&quot;&quot;)[1]<br>                <strong>if</strong> &#39;http&#39; <strong>in</strong> link_item <strong>or</strong> &#39;www&#39; <strong>in</strong> link_item:<br>                    img_links.append(link_item)<br>    <strong>return</strong> img_links</pre><h3>4. Downloading the images</h3><p>Now, we have all the image links with us, we will now proceed to download all the images from these links and store them in our local directory. For that, we will pass the image list as a parameter to this method.</p><p>We will start by creating a date_ variable that stores the current day date in “dd-mm-yyyy” format.</p><pre><strong>day</strong> <strong>=</strong> datetime.date.today().<strong>day</strong><br><strong>month</strong> <strong>=</strong> datetime.date.today().<strong>month</strong><br><strong>year</strong> <strong>=</strong> datetime.date.today().<strong>year</strong><br><br>date_ <strong>=</strong> str(<strong>day</strong>)<strong>+</strong>&quot;-&quot;<strong>+</strong>str(<strong>month</strong>)<strong>+</strong>&quot;-&quot;<strong>+</strong>str(<strong>year</strong>)<strong>+</strong>&quot;_&quot;</pre><p>Before creating a new directory to store the images, we will check for the presence of the “images” directory in our current working directory.</p><p>If not found, we will create a new “images” directory. Inside the “images” directory, we will create a new directory with the current day date as its name to store the images.</p><pre><strong>if</strong> &#39;images&#39; <strong>not</strong> <strong>in</strong> os.listdir():<br>        os.mkdir(&quot;images&quot;)<br>        os.chdir(&quot;images&quot;)<br><br>    os.mkdir(date_)<br>    os.chdir(date_)</pre><p>Now, we will initiate an image number variable to name the images. We can access the &lt;alt&gt; tag to fetch the image name from the URL, but sometimes, images won’t have an alt name. So, to keep image names uniform, we will number the images.</p><pre>img_no <strong>=</strong> 1</pre><p>Next, we will iterate through the image links, fetch the images and store them in our local directory.</p><pre><strong>for</strong> <strong>link</strong> <strong>in</strong> image_list:<br>    img_response <strong>=</strong> requests.<strong>get</strong>(<strong>link</strong>)<br><br>    img_format <strong>=</strong> link.<strong>split</strong>(&quot;.&quot;)[<strong>-</strong>1]<br><br>    filename <strong>=</strong> &quot;img&quot; <strong>+</strong> str(img_no) <strong>+</strong> &quot;.&quot; <strong>+</strong> img_format<br><br>    <strong>with</strong> <strong>open</strong>(filename, &quot;wb+&quot;) <strong>as</strong> f:<br>        f.<strong>write</strong>(img_response.content)<br>    img_no <strong>+=</strong> 1</pre><p>Putting it all together, we have:</p><pre><strong>def</strong> <strong>download_images</strong>(image_list):<br>    <strong>day</strong> <strong>=</strong> datetime.date.today().<strong>day</strong><br>    <strong>month</strong> <strong>=</strong> datetime.date.today().<strong>month</strong><br>    <strong>year</strong> <strong>=</strong> datetime.date.today().<strong>year</strong><br><br>    date_ <strong>=</strong> str(<strong>day</strong>)<strong>+</strong>&quot;-&quot;<strong>+</strong>str(<strong>month</strong>)<strong>+</strong>&quot;-&quot;<strong>+</strong>str(<strong>year</strong>)<strong>+</strong>&quot;_&quot;<br><br>    <strong>if</strong> &#39;images&#39; <strong>not</strong> <strong>in</strong> os.listdir():<br>        os.mkdir(&quot;images&quot;)<br>        os.chdir(&quot;images&quot;)<br><br>    os.mkdir(date_)<br>    os.chdir(date_)<br><br>    img_no <strong>=</strong> 1<br><br>    <strong>for</strong> link <strong>in</strong> image_list:<br>        img_response <strong>=</strong> requests.get(link)<br><br>        img_format <strong>=</strong> link.split(&quot;.&quot;)[<strong>-</strong>1]<br><br>        filename <strong>=</strong> &quot;img&quot; <strong>+</strong> str(img_no) <strong>+</strong> &quot;.&quot; <strong>+</strong> img_format<br><br>        <strong>with</strong> <strong>open</strong>(filename, &quot;wb+&quot;) <strong>as</strong> f:<br>            f.write(img_response.content)<br>        img_no <strong>+=</strong> 1</pre><p>Now that we have defined all our methods, we will put it all together to create our StaticSiteScraper class. I’m not going to go deep here as this is self-explanatory.</p><h3>Implementing our static scraper</h3><p>Finally, we have made our scraper class.</p><p>It’s time that we now test our StaticSiteScraper. For that, I’m now moving on to craigslist.org’s New York real estate page.</p><p>Getting the URL from the page</p><p>url = “https://newyork.craigslist.org/d/real-estate/search/rea&quot;</p><p>Creating a <strong>craig</strong> object using our StaticSiteScraper</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*PzO3rZHNH3MsQ0Vw3l5iOA.png" /></figure><p>Creating a <strong>page object</strong> that can be used for accessing any other HTML elements within this page if required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*fENJ2V-LnzPnZoqHzgtjHA.png" /></figure><p>Now, we will access all the links within this webpage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*5TuuGhJni7QLZEsChncQgA.png" /></figure><p>Similarly, we will now fetch the links of all the images on the webpage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/478/1*--tIBFGVF058EpQg_nG15w.png" /></figure><p>And we have our output as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*p5XdIeEoXVS5326hMc0U3Q.png" /></figure><p>Before proceeding to download all the images, make sure to check the following:</p><p><strong>1. If we were able to fetch image links</strong></p><p>Why I’m saying this is because almost all of the websites post their images within a &lt;div&gt; tag and maybe within more tags inside the &lt;div&gt; tag. So, it will be very hard to fetch all the images in one go.</p><p>Here, you will have to make use of the <strong>page object</strong> we had created before to dig deeper into the tags where you can find the image link. From there, you have to fetch each image and store it into a list.</p><p><strong>2. If image links are in a proper format</strong></p><p>Most of the time, image links might not be in a proper format and we might need to rearrange them in proper format before passing them to the function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*ZuPyoYapl3gMMvA6zpGVLQ.png" /></figure><p>Now our output looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*kGC1ec7rvV3BOUZvb4zWhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*VAzFySgKls1Z56awq9Revw.png" /></figure><p>With that, we have now created a web scraper that can fetch all the static content of a given URL.</p><p>Finally, I hope you liked it. Follow me for more actionable content and feel free to post your valuable feedback and suggestions.</p><p><a href="https://medium.com/@retinpkumar">Retin P Kumar - Medium</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f2f0f97dc47" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/how-to-build-a-simple-web-scraper-using-beautifulsoup-2f2f0f97dc47">How to build a simple Web scraper using Beautifulsoup</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>