<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Atif Noorul Hasan on Medium]]></title>
        <description><![CDATA[Stories by Atif Noorul Hasan on Medium]]></description>
        <link>https://medium.com/@atifnhasan2?source=rss-3cc5aa980e2c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4ipWPH-W5h6Pwjn97QkISg.jpeg</url>
            <title>Stories by Atif Noorul Hasan on Medium</title>
            <link>https://medium.com/@atifnhasan2?source=rss-3cc5aa980e2c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:30:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@atifnhasan2/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pharmaceutical Sales Data Analysis, Visualization, and Profit Prediction]]></title>
            <link>https://medium.com/@atifnhasan2/pharmaceutical-sales-data-analysis-visualization-and-profit-prediction-44a347905e91?source=rss-3cc5aa980e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/44a347905e91</guid>
            <category><![CDATA[visualization-power-bi]]></category>
            <category><![CDATA[data-analysis-project]]></category>
            <category><![CDATA[data-analysis-with-sql]]></category>
            <category><![CDATA[data-analysis-with-python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Atif Noorul Hasan]]></dc:creator>
            <pubDate>Thu, 07 Jul 2022 07:29:32 GMT</pubDate>
            <atom:updated>2025-03-11T08:10:10.410Z</atom:updated>
            <content:encoded><![CDATA[<p>Sinofi Pharmaceutical Private Limited is a pharmaceutical marketing startup operational in different Indian states. This project is based on its sales data. As a data analyst, the job is to carry out detailed sales insight analysis, predict future sales &amp; net profit, and suggest suggestions to help grow the company.</p><h4>The Data</h4><p>The data is in the form of CSV files and it has four-year (2017, 2018, 2019, 2020) of sales data in the form of tables, taken from the company archives on specific terms and conditions for maintaining privacy.</p><p>The first table is ‘<strong>sinofi_raw.csv</strong>’ and it has millions of rows and the following columns: product_code, product_type, customer_code, market_code, market_name, order_date, sales_qty, upc (Unit Product Cost), sales_amount, and zone.</p><p>Later this raw table would be fragmentized (<strong>Data Fragmentation</strong>) into different specific tables.</p><p>The second table is ‘<strong>date.csv</strong>’ and it has the following columns: date, cy_date, year, month, date_yy_mmm.</p><p><em>Note: The data has product code and customer code but not the product&#39;s name (or composition), nor does it have the customer name. This is due to the company privacy agreement.</em></p><h4><strong>Aim &amp; Objectives</strong></h4><p>The job of a data analyst is to analyze the available data and bring out useful insights that otherwise are not visible. The project would focus on the complete statistical analysis including:</p><ol><li><strong>Descriptive Analysis</strong> — What has happened?</li><li><strong>Diagnostic Analysis</strong> — Why did it happen?</li><li><strong>Predictive Analysis</strong> — What could happen?</li><li><strong>Prescriptive Analysis</strong> — What should happen?</li></ol><p>This project would try to answer the following business questions:</p><ol><li>What is the company’s yearly revenue?</li><li>What is the company’s yearly net profit?</li><li>Is the company making a profit or is running on a loss?</li><li>Which product and product type are making the most of the revenue?</li><li>Which product and product type are making most of the net profit?</li><li>Which employee, zonal manager, and the zone are not performing well?</li><li>Under the current sales pattern what would be the revenue and net profit in the next year?</li><li>What improvements need to be done?</li><li>After improvement what could be the revenue and net profit in the next year?</li><li>Make a revenue and net profit predictive model based on the previous year&#39;s data?</li><li>Make an interactive dashboard for sales insights.</li></ol><h4>Data Analysis Pipeline</h4><h4><strong>1. Load the data:</strong></h4><p>upload the CSV files to any Python IDE of your choice. In my case, it was <strong>Jupiter Notebook</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f663a915b42a2cde0b9bc0b2c496d4bc/href">https://medium.com/media/f663a915b42a2cde0b9bc0b2c496d4bc/href</a></iframe><h4><strong>2. Data processing/Cleaning</strong></h4><p>Before the data can be analyzed, it needs to be analyzable and ready, which means it needs to be cleaned, normalized, engineered, etc in all possible ways to avoid ill analysis and/or prediction. This process could be carried out on SQL or Python but I find Pyhton easy for such tasks.</p><p><strong>Data Cleaning</strong></p><p>Checking, null values, negative values, and duplicate rows in the data and treating them with appropriate data actions such as row deletion, filling blanks, etc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7fc0c5489414e29b7fc7faa29a4514a/href">https://medium.com/media/e7fc0c5489414e29b7fc7faa29a4514a/href</a></iframe><p><strong>Feature Engineering</strong></p><p>All the features (column labels) are provided but sometimes those features are not enough for analysis, prediction, or visualization and therefore other features need to be added. But we can not add features on our own rather we derive features from the existing or a combination of existing features.</p><p>In this case, we have sales quantity (sales_qty), unit product cost (upc), and sales amount(sales_amount) but we don’t have a net profit column.</p><p><strong>Net profit = sales_amount — (sales_qty * UPC)</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97d216d0787c2f41898b180aed27ac96/href">https://medium.com/media/97d216d0787c2f41898b180aed27ac96/href</a></iframe><p>The new feature ‘net_profit’ has been added to the ‘raw’ dataframe and the ‘raw’ dataframe is downloaded as a CSV file (raw.csv)</p><p>Now, this raw.csv would be used for data fragmentation.</p><p><strong>Data Fragmentation</strong></p><p>For a better downstream data analysis and Power BI visualization, the single table can be fragmentized into a few more tables such as a product table, customer table, market table, etc. Data fragmentation is done in python but could be done in SQL as well.</p><p><strong>Data fragmentation with Python</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef2501b94a72121ce469ef884d858520/href">https://medium.com/media/ef2501b94a72121ce469ef884d858520/href</a></iframe><p><strong>Data fragmentation with SQL</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/26eb057c18a82bd1122f8e0f40c86c61/href">https://medium.com/media/26eb057c18a82bd1122f8e0f40c86c61/href</a></iframe><p>After fragmentation, we have the following table created and downloaded in CVS format —</p><ol><li>customers.csv</li><li>markets.csv</li><li>products.csv</li><li>upc.csv</li><li>transactions.csv</li></ol><p>The ‘tansactions.csv’ is the main table which has all the features and information. It is just a copy of ‘raw.csv’.</p><p>At this stage, data processing is not finished. The next step would be data <strong>normalization</strong>, but the normalized data is usually required for machine learning and prediction. We will normalize our data when we would carry predictions with machine learning. For analysis purposes our data is ready.</p><h4>3. Data Analysis with SQL queries</h4><p>All the CSV tables are loaded to the PostgreSQL server. Below are some queries and the result of those queries.</p><ol><li><strong>/* Calculate net profit */</strong><br>Select sum(sales_amount)as Tot_Sales_Amount,<br>round(sum(upc)::numeric, 2) as Tot_Unit_Cost, <br>round(sum(sales_amount)-sum(sales_qty*upc)::numeric, 2) as Net_Profit <br>from transactions;</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*l1rZYKrhNgNSHkjaGrniNw.png" /><figcaption><strong>Overall Net Profit</strong></figcaption></figure><p><strong>2. /* Select top product w.r.t. sales amount */</strong><br>Select product_code, <br>sum(sales_amount)as tot_sales_amount<br>from transactions <br>group by product_code <br>order by sum(sales_amount) desc;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*23nNloR0dbr9kfpRydsFZQ.png" /><figcaption><strong>Top 3 product w.r.t. total sales amount</strong></figcaption></figure><p><strong>3. /* Which product type has the highest sales */</strong><br>select product_type,<br>sum(sales_amount)as tot_sales_amount <br>from transactions <br>group by product_type <br>order by sum(sales_amount) desc;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*CaGd_FJSwK4nPD6NBChNeg.png" /><figcaption><strong>Top 3 product types w.r.t. total sales amount</strong></figcaption></figure><p>Injectable seems to be the product type with the highest total sales amount, but is it the product type with the highest net profit?</p><p>4. <strong>/* Which product type has the highest net profit */<br></strong>select product_type,<br>round(sum(net_profit)::numeric,2)as tot_net_profit <br>from transactions <br>group by product_type <br>order by sum(net_profit) desc;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*X2u9JZFtQRbL3xl0XyUsqw.png" /><figcaption><strong>Top 3 product types w.r.t. total net sales</strong></figcaption></figure><h4>Indeed, the Injectable product type has the highest net profit with the highest total sales.</h4><h4>4 . ETL (Extract, transform, load)</h4><p><strong>Extract: </strong>The raw data was provided by the company in the form of CSV files.</p><p><strong>Transform:</strong> The data was transformed in Python but it could be transformed in SQL or even on Power BI. For convenience I used Python. The detailed data transformation could be found in <strong>section 2</strong> (Data processing/Cleaning)</p><p><strong>Load:</strong> The tables were loaded to the PostgreSQL server and some crucial insights were extracted with the help of SQL queries. Finally, those tables were loaded on to Power BI dashboard by connecting the PostgreSQL server with Power BI.</p><h4>5. Data Visualization With Power BI</h4><p>Data visualization helps you find KPIs and brings out hidden insights which otherwise could not be seen just by looking at the data.</p><p>For visualization, we will use Power BI. All the table transactions, customers, markets, products, upc, and date are exported to the Power BI dashboard. Our file could directly be loaded to the Power BI dashboard but I would connect the PostgreSQL server with Power BI and load data(CSV files) from there.</p><p><strong>Data Modeling in Power BI</strong></p><p>Power BI provides an option to model or transform your data. The photo below is our data model. It shows the relationship between different tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*9H8umb8J0D-QG2k-AFC9vQ.jpeg" /><figcaption><strong>Data Model in Power BI</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fD5Ch-viSK-I-L8afXzE2g.png" /><figcaption><strong>Dashboard 1</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eidJI_ifCNYyRmR8tbHYdg.png" /><figcaption><strong>Dashboard 2</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uz_AYz0M_vlPJ_QC_zOjpQ.png" /><figcaption><strong>Dashboard 3</strong></figcaption></figure><p><strong>Insights From Power BI Visualization</strong></p><ol><li>The company is making a decent profit but the sales amount trend shows that the sales are decreasing which may lead to loss eventually.</li><li>Injectable product types are the ones contributing to most of the sales amount and net profit and ointments and power product types are contributing to the loss.</li><li>It is not necessary that the customer that purchases more provides more net profit.</li></ol><p>4. Patna has the highest sales amount but it provides less profit margin percentage whereas Jamshedpur has a low sales amount but still provides the highest profit margin percentage. The company should consider increasing the market in Jamshedpur as it provides a high-profit margin percentage.</p><p>Visualization tools like Power BI can make many interactive dashboards, can answer any questions, and can provide endless insights. It is modified as per management or the client&#39;s demand and queries.</p><h4><strong>6. Predicting Net Profit With Machine Learning Model</strong></h4><p>The available data could be used to train a machine learning model to predict net sales amount or net profit. Below is the complete approach to the process of selecting the best ML-Algorithm for the data and training that ML model to predict with the least errors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fcebc31c4c53345c1e07348a4f592b8/href">https://medium.com/media/8fcebc31c4c53345c1e07348a4f592b8/href</a></iframe><p>If provided with a separate test set the above model could predict net sales with a significantly low error which you could see in the comparison table of actual test net profit and predicted net profit.</p><h4>7. Inferences</h4><ol><li>SQL, Python, and any data visualization tool such as Power BI when combinedly used can bring out hidden insights from the data.</li><li>In sales, it is not necessary that a feature corresponding to the highest revenue would correspond to the highest net profit.</li><li>If a product&#39;s sales quantity is very low and net profit is also very low or negative, then the management should consider discontinuation of that product. Even though the management wants to continue that product then it should conder making new strategies that would increase the product&#39;s sales and would result in increased net profit.</li><li>If a customer or a market is having a low sales quantity and low revenue collection but it has a high net profit percentage, then it’s time to expand that market and supply more products to that customer and market.</li></ol><h4>8. Appeal</h4><p>This is not all! and never it will be. I request the readers to register their reviews and suggestions that what else could be done to improve any section or this project as a whole.</p><p><strong>Looking forward to your suggestions and reviews.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44a347905e91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essential Concepts of Machine Learning With Linear Regression in Python]]></title>
            <link>https://medium.com/@atifnhasan2/essential-concepts-of-machine-learning-with-linear-regression-in-python-273443d0fc3e?source=rss-3cc5aa980e2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/273443d0fc3e</guid>
            <category><![CDATA[different-regressions]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression-python]]></category>
            <category><![CDATA[machine-learning-python]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Atif Noorul Hasan]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 13:21:54 GMT</pubDate>
            <atom:updated>2022-02-14T13:29:44.161Z</atom:updated>
            <content:encoded><![CDATA[<h4>Machine Learning With Linear Regression: Essential Concepts, EDA, Data Manipulation, One-Hot-Encoding, Feature Scaling &amp; Regression Comparative Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yU9jAgyFQpYny7OPTj7yFA.jpeg" /></figure><blockquote>Data Science is not a destination, it’s a journey and across this journey you will encounter and you will have to keep learning many things. While learning among all the different things, I consider MACHINE LEARNING (ML) to be one of the most important Data Science skills.</blockquote><blockquote><strong><em>Prerequisite:</em></strong><em> Knowledge of Python, Data frames, Python libraries like, pandas, matplotlib, seaborn, etc.</em></blockquote><p>In this article I have used the most basic ML algorithm, i.e. Linear Regression. Implementing any ML algorithm is a minute work, but just copy pasting codes won’t make you a <strong>smart learner</strong>. Take an extra leap and go through this article. ML codes you can find every where but the explanation and behind the scene matters you will find here. What makes this article different from other is:</p><ol><li>It will inspire to dive deep and not just copy pasting codes.</li><li>It starts from basics and ends with advance.</li><li>It has a correct approach towards ML.</li><li>It deals with real life like problem and problem solving.</li><li>It will inspire you to think out of the box.</li></ol><p><strong>Machine Learning: </strong>You get a new dog at your home. Now you want to play with him. You threw a boll away from you and you were expecting that the dog would run and grab the ball and bring it back to you, but it didn’t happened.</p><p>You realized, you need to train your dog for this task. After few months of throwing ball and getting it back the dog learnt that whenever my master throws a ball, I have to get it back. This learning is achieved by a living organism and that living organism went under training.</p><p>like wise, if we train a computer program (Machines) with data which has inputs and outputs, it will learn how a particular output comes and then if we provide any similar random input it will provide us with output based on previous learnings. <strong>This is the core process of any machine learning.</strong></p><blockquote><strong>Example :</strong> Have you noticed the pattern of YouTube recommending the types of video to your home feed. Most of the recommended videos are the kind of videos which you like to watch. With your searching habits and videos you have watched in previous times, the algorithm of YouTube has learnt that, “What kind of videos do you like”. The YouTube goal is simple, feed the viewer with the content they like as much as possible so that they spend more and more time on YouTube.</blockquote><h4><strong>Types of Machine Learning</strong></h4><p>Bases on the data which it gets, machine learning could be divided into two broad categories. <strong>Supervised learning</strong> (Data is labeled) and <strong>Un-Supervised Learning</strong> (Data is not labeled).</p><blockquote><strong>But on the first place what is a data and when it becomes an information</strong>?</blockquote><p><strong>Data</strong> is a collection of facts which is unorganized. In the figure below (Fig-1) the right group of photos shows some animals, this is data. There is no clear information about what animals are they. But the left group of animals have names. We can draw an <strong>information</strong> from that group of animals that, this group of animal consists of two dogs, one black one yellow and two cats, one yellow and one black and the yellow cat is in sitting position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*zyS7c8DaazuWyCpCCbJKvQ.jpeg" /><figcaption>Fig-1: Labeled &amp; Unlabeled Data</figcaption></figure><p>The above diagram(Fig-1) shows how a clear difference between labeled and unlabeled data.</p><p>Typically you would encounter data in tabular form with <strong>Rows</strong> and <strong>Columns</strong>. Rows being the data points and the columns are the <strong>attributes/instances/features </strong>which has some values in the row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*6y0ffcpyY3-rl6IqZBkTHg.png" /><figcaption>Fig-2: A Typical labeled data</figcaption></figure><p>Now it’s clear that what is information and different types of data, we should now get back to supervised and unsupervised learning.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxtOg44r6dsE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxtOg44r6dsE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxtOg44r6dsE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d3caa5846f00bc46a4fe12c7b53de27/href">https://medium.com/media/0d3caa5846f00bc46a4fe12c7b53de27/href</a></iframe><p>It is advisable to watch this video to get a brief insights on machine learning and supervised and unsupervised learning.</p><p>One of the most common regression algorithm is <strong>Linear regression.</strong> Linear regression is of two types, <strong>Simple Linear Regression</strong> and <strong>Multiple Linear Regression</strong>.</p><p>Have a look on the first few rows of the data on which we are going to work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*GY_mFr-uN8QpGHh-rqdqUg.jpeg" /><figcaption>Fig-3: The Data we are going to work on</figcaption></figure><p>Before knowing about types of linear regression, first we need to learn what is <strong>dependent variables </strong>and <strong>non dependent variables</strong>. The table above (Fig-3) attributes, age, sex, bmi, children, smoker, region and charges. Suppose we want to know the charges for a person when only his age is provided. In this case the value which we want to predict, i.e. ‘charges’ <strong>is dependent variable, </strong>whereas ‘age’ <strong>is independent variable</strong>. We want to estimate the charges when particular age is provided. In this situation we are hoping that charges will change with age.</p><p>So, when linear regression is carried out with one dependent variable and one independent variable, it is called <strong>simple linear regression</strong>.</p><p>In another situation, we want to estimate ‘charges’ (dependent variable) when ‘age’, ‘sex’, ‘bmi’, etc (independent variables) are given. Here we are telling the algorithm that a person’s charges would depend on many factors.</p><p>So, when a linear regression is carried out with one dependent variable and more than one independent variables, it is called <strong>multiple linear regression</strong>.</p><p><strong>So, after a brief introduction, let’s begin the Machine Learning.</strong></p><h3>Predicting Insurance Premiums</h3><h4><strong>Problem Statement:</strong></h4><p><strong>QUESTION</strong>: ACME Insurance Inc. offers affordable health insurance to thousands of customer all over the United States. As the lead data scientist at ACME, <strong>you’re tasked with creating an automated system to estimate the annual medical expenditure for new customers</strong>, using information such as their age, sex, BMI, children, smoking habits and region of residence.</p><p>Estimates from your system will be used to determine the annual insurance premium (amount paid every month) offered to the customer. Due to regulatory requirements, you must be able to explain why your system outputs a certain prediction.</p><p>You’re given a <a href="https://raw.githubusercontent.com/Atif2227/Machine-Learning-/main/Linear%20Regression%20Simple%20%26%20Multiple/medical.csv"><strong>CSV file</strong></a> containing verified historical data, consisting of the aforementioned information and the actual medical charges incurred by over 1300 customers.</p><p><strong>Dataset source:</strong> <a href="https://github.com/Atif2227/Machine-Learning-/blob/main/Linear%20Regression%20Simple%20%26%20Multiple/medical-charges.csv">https://github.com/Atif2227/Machine-Learning-/blob/main/Linear%20Regression%20Simple%20%26%20Multiple/medical-charges.csv</a></p><p><strong>Exercise: </strong>Before proceeding further, take a moment to think about how you can approach this problem. List five or more ideas that comes to your mind below:</p><p>The annual medical expenditure would determine the annual premium for a customer. Means if the annual medical expense is more then the company would charge more annual premium in order to make a profit.</p><ol><li>Do some Basic EDA/Visualization to look for some relation and dependencies among the attributes.</li><li>Find the strongest co-relation between attribute/s and charges.</li><li>The attribute/s with strongest co-relation could be used for training, testing and validation.</li><li>Further the model based on above inference could be used for prediction.</li></ol><h4><strong>Importing important libraries:</strong></h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D27&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D27&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/90d1ac1808e456746fb97e41281a6c89/href">https://medium.com/media/90d1ac1808e456746fb97e41281a6c89/href</a></iframe><h4>Downloading The Data</h4><p>To begin, let’s download the data using the urlretrieve function from urllib.request.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F7%26cellId%3D29&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F7%26cellId%3D29&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F5aace73e575a4590a2b9f26941fc027c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/11c1c145449b719a62c6dbbd529cbe76/href">https://medium.com/media/11c1c145449b719a62c6dbbd529cbe76/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D30&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D30&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/93ee32e26fecbabeed00d152b0867883/href">https://medium.com/media/93ee32e26fecbabeed00d152b0867883/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D31&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D31&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/42002065e4d1f6e687c550da7e7c6ee6/href">https://medium.com/media/42002065e4d1f6e687c550da7e7c6ee6/href</a></iframe><p>We can now create a Pandas dataframe using the downloaded file, to view and analyze the data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D33&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D33&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8264b2684a0a86646b888441892777b/href">https://medium.com/media/a8264b2684a0a86646b888441892777b/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D34&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D34&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7991037f5b93742322d277346eaf2cc7/href">https://medium.com/media/7991037f5b93742322d277346eaf2cc7/href</a></iframe><p>The dataset contains 1338 rows and 7 columns. Each row of the dataset contains information about one customer. Our objective is to find a way to estimate the value in the “charges” column using the values in the other columns. If we can do so for the historical data, then we should able to estimate charges for new customers too, simply by asking for information like their age, sex, BMI, no. of children, smoking habits and region. Let’s check the data type for each column.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D36&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D36&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e90b0bdcdcfe7ec7faeb1a8592ff1c9/href">https://medium.com/media/8e90b0bdcdcfe7ec7faeb1a8592ff1c9/href</a></iframe><p>Looks like “age”, “children”, “bmi” (<a href="https://en.wikipedia.org/wiki/Body_mass_index"><strong>body mass index</strong></a>) and “charges” are numbers, whereas “sex”, “smoker” and “region” are strings (possibly categories). None of the columns contain any missing values, which saves us a fair bit of work!</p><p>Here are some statistics for the numerical columns:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D38&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D38&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/16fb07b9840f31e75fee9111ae95d7c5/href">https://medium.com/media/16fb07b9840f31e75fee9111ae95d7c5/href</a></iframe><p>The ranges of values in the numerical columns seems reasonable too (no negative ages!), so we may not have to do much data cleaning or correction. The “charges” column seems to be significantly skewed however, as the median (50 percentile) is much lower than the maximum value.</p><p><strong>EXERCISE</strong>: What other inferences can you draw by looking at the table above? Add your inferences below:</p><ol><li>The columns like ‘age’ and ‘bmi’ seems to be distributed evenly, as their min, 25%, 50% 75% and max values have similar range.</li><li>Where as the column children and charges are highly skewed.</li></ol><h4>Exploratory Analysis and Visualization</h4><p>Let’s explore the data by visualizing the distribution of values in some columns of the dataset, and the relationships between “charges” and other columns.</p><p>We’ll use libraries Matplotlib, Seaborn and Plotly for visualization.</p><p>The following settings will improve the default style and font sizes for our charts.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D42&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D42&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7eb002985fb9e7bbe601097f245a3ce1/href">https://medium.com/media/7eb002985fb9e7bbe601097f245a3ce1/href</a></iframe><h4><strong>Let’s see how Age, Sex, BMI, Children, Smoker, Region and Charges are distributed.</strong></h4><p><strong>Age: </strong>Age is a numeric column. The minimum age in the dataset is 18 and the maximum age is 64. Thus, we can visualize the distribution of age using a histogram with 47 bins (one for each year) and a box plot. We’ll use <strong>plotly</strong> to make the chart interactive, but you can create similar charts using Seaborn.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D45&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D45&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1422492acf3175fe2498f07c3ccdecf/href">https://medium.com/media/b1422492acf3175fe2498f07c3ccdecf/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D46&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D46&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd58085b01af8b606f852e5f4f1b09fe/href">https://medium.com/media/bd58085b01af8b606f852e5f4f1b09fe/href</a></iframe><p>The distribution of ages in the dataset is almost uniform, with 20–30 customers at every age, except for the ages 18 and 19, which seem to have over twice as many customers as other ages. The uniform distribution might arise from the fact that there isn’t a big variation in the <a href="https://www.statista.com/statistics/241488/population-of-the-us-by-sex-and-age/"><strong>number of people of any given age</strong></a><strong> </strong>(between 18 &amp; 64) in the USA.</p><p><strong>EXERCISE:</strong> Can you explain why there are over twice as many customers with ages 18 and 19, compared to other ages?</p><p><strong>Answer:</strong> At 18 an individual turns eligible for having insurance and there is an strong probability that the insurance company might be providing lower insurance premium for this age group.</p><p><strong>Sex:</strong> Let’s look at the distribution of sex of customers, using a histogram and box plot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D49&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D49&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/47d31eef3cfff04922c2aaf79dba3e54/href">https://medium.com/media/47d31eef3cfff04922c2aaf79dba3e54/href</a></iframe><p>Looks like the population is equally distributed between male and female but smoker males are comparatively more than smoker females</p><p><strong>Body Mass Index:</strong> Let’s look at the distribution of BMI (Body Mass Index) of customers, using a histogram and box plot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D52&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D52&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/29749ebb0d96a932df0255042cf5661a/href">https://medium.com/media/29749ebb0d96a932df0255042cf5661a/href</a></iframe><p>The measurements of body mass index seem to form a <a href="https://en.wikipedia.org/wiki/Normal_distribution"><strong>Gaussian distribution</strong></a> centered around the value 30, with a few outliers towards the right. Here’s how BMI values can be interpreted <a href="https://study.com/academy/lesson/what-is-bmi-definition-formula-calculation.html"><strong>(source)</strong></a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/0*dAqG3tiAzOsDOejt.jpg" /></figure><p><strong>EXERCISE:</strong> Can you explain why the distribution of ages forms a uniform distribution while the distribution of BMIs forms a gaussian distribution?</p><p><strong>Answer:</strong> As per studies, most of the people in the population are above “under weight” to “slightly over weight” and thus for this age group the BMI lies between 24 to 38. Whereas there are few people as compared to the previously mention weight category who are either under weight or over weight. Thus population with lower and higher BMI is comparatively less.<br>Let’s plot a line diagram between ‘age’ and ‘bmi’.</p><p><strong>Children:</strong> Let’s look at the distribution of children of customers, using a histogram and box plot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D55&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D55&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b4b53105902331753de7c0bb767944f/href">https://medium.com/media/7b4b53105902331753de7c0bb767944f/href</a></iframe><p>Most of the population has no children and there are very few people having 5 children. It would be interesting to see if children has any relation with charges or not.</p><p><strong>Smoker:</strong> Look at the distribution of smoker and non-smokers within the population.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D58&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D58&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2666b546ac2586c4edb5a30cc90fdee0/href">https://medium.com/media/2666b546ac2586c4edb5a30cc90fdee0/href</a></iframe><p>Vast majority of population is non-smoker which is good.</p><p><strong>Region:</strong> Let’s visualize the distribution of population in different regions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D61&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D61&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8bd670baf96ac531c4ad30ccfe1db16/href">https://medium.com/media/d8bd670baf96ac531c4ad30ccfe1db16/href</a></iframe><p>Population among different region is almost equally distributed except slightly more for southeast, they may have more children than other region.</p><p><strong>Charges:</strong> Let’s visualize the distribution of “charges” i.e. the annual medical charges for customers. This is the column we’re trying to predict. Let’s also use the categorical column “smoker” to distinguish the charges for smokers and non-smokers.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D64&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D64&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0680e87f6eee7e6a0c0b9f37306bbe4f/href">https://medium.com/media/0680e87f6eee7e6a0c0b9f37306bbe4f/href</a></iframe><p>We can make the following observations from the above graph:</p><ol><li>For most customers, the annual medical charges are under $10,000. Only a small fraction of customer have higher medical expenses, possibly due to accidents, major illnesses and genetic diseases. The distribution follows a <strong>“power law”</strong>.</li><li>There is a significant difference in medical expenses between smokers and non-smokers. While the median for non-smokers is $7300, the median for smokers is close to $35,000.</li></ol><p>So it’s very much evident that less people smoke as compared to non-smokers and smoking increases charges too.</p><p><strong>Having looked at individual columns, we can now visualize the distribution/relationship between “charges” (the value we wish to predict) and other columns.</strong></p><p><strong>Age and Charges:</strong> Let’s visualize the relationship between “age” and “charges” using a scatter plot. Each point in the scatter plot represents one customer. We’ll also use values in the “smoker” column to color the points.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D68&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D68&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2F8c997aea7c0d40abaebcb5e459869813%2Fpreview%2Fb1d75446ccfd4265b00286937c3a0cac&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/42c7a9e15ab68039629437d5c6c87d03/href">https://medium.com/media/42c7a9e15ab68039629437d5c6c87d03/href</a></iframe><p>We can make the following observations from the above chart:</p><ol><li>The general trend seems to be that medical charges increase with age, as we might expect. However, there is significant variation at every age, and it’s clear that age alone cannot be used to accurately determine medical charges.</li><li>We can see three “clusters” of points, each of which seems to form a line with an increasing slope:</li></ol><p>The first and the largest cluster consists primary of presumably “healthy non-smokers” who have relatively low medical charges compared to others</p><p>The second cluster contains a mix of smokers and non-smokers. It’s possible that these are actually two distinct but overlapping clusters: “non-smokers with medical issues” and “smokers without major medical issues”.</p><p>The final cluster consists exclusively of smokers, presumably smokers with major medical issues that are possibly related to or worsened by smoking.</p><p><strong>EXERCISE:</strong> What other inferences can you draw from the above chart?</p><p>So, with the above scatter plot we have 3 clusters. The 1st cluster shows, all the population is non smoker and charges increases with age. The 2nd cluster shows that there is a mix population of smoker and non smoker. But the charges remain higher that the non smoker cluster and even for this cluster the charges increases with age. The 3rd cluster show that the population is smoker and hence they have highest charges and even in this cluster charges increases with age.</p><p>So it can be <strong>concluded that</strong> people who smoke have higher annual charges and the annual charges increase with age irrespective of the smoking habit. There are some cases where the population is non smoker but they still have high charge and this could be because of various external features such as, any accident or any other body ailment.</p><p><strong>Sex and Charges:</strong> Let’s visualize the distribution of the “smoker” column (containing values “yes” and “no”) using a histogram.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D71&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D71&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/183e3be47a3440301d778c58a2aac1a3/href">https://medium.com/media/183e3be47a3440301d778c58a2aac1a3/href</a></iframe><p>It appears that median charges for smoker females are slightly more than smoker males. Even the outliers for smoker females seems to be more above than smoker males. The reason behind this could be that smoker females might get more sick than smoker males.</p><p>However non-smoker male median charge is far more that non-smoker females. This might be because of some major body ailment, accident or other external factor which is more obvious for males.</p><p>Overall, it seems that sex could not be a good measure to decide for medical charges.</p><p><strong>BMI and Charges:</strong> Let’s visualize the relationship between BMI (body mass index) and charges using another scatter plot. Once again, we’ll use the values from the “smoker” column to color the points.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D74&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D74&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bb0b44b5a815251a4c1a78b5347aa1b/href">https://medium.com/media/3bb0b44b5a815251a4c1a78b5347aa1b/href</a></iframe><p>It appears that for non-smokers, an increase in BMI doesn’t seem to be related to an increase in medical charges. However, medical charges seem to be significantly higher for smokers with a BMI greater than 30.</p><p><strong>What other insights can you gather from the above graph?</strong></p><p><strong>Answer:</strong> With the above scatter plot it is clear that there is no direct linear relation between BMI and charges. For non-smokers the BMI increases but the charges remains around 0 t0 20000 (With some outliers/exception: May be due to some other medical condition). For smokers-BMI, the charges is usually more than non-smokers-BMI. But when BMI Exceeds 30 there is steep increase in charges for smokers-BMI.</p><p><strong>Children and charges</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D77&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D77&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/84fd20a03ae9fa0ff5df1e44d11ead44/href">https://medium.com/media/84fd20a03ae9fa0ff5df1e44d11ead44/href</a></iframe><ol><li>The minimum charges increases with increasing children.</li><li>There is no clear relation between children and charges.</li></ol><p><strong>Smoker and Charges</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D80&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D80&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ef78a1c76e5f2d133a034bad7ca7561/href">https://medium.com/media/0ef78a1c76e5f2d133a034bad7ca7561/href</a></iframe><p><strong>Region and charges</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D83&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D83&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/af2f99584e5d65e7de1acd79ab6933a7/href">https://medium.com/media/af2f99584e5d65e7de1acd79ab6933a7/href</a></iframe><p>There is no trend or evidence which could say that population belonging to a particular region may have increased or decreased annual charges. No clear relation between region and charges found.</p><h4>Correlation</h4><p>As you can tell from the analysis, the values in some columns are more closely related to the values in “charges” compared to other columns. E.g. “age” and “charges” seem to grow together, whereas “bmi” and “charges” don’t. Moreover smokers has more charges than non-smokers.</p><p>This relationship is often expressed numerically using a measure called the <em>correlation coefficient</em>, which can be computed using the .corr method of a Pandas series.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D86&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D86&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/524b16fd8907c49fc5687e7257ba2dca/href">https://medium.com/media/524b16fd8907c49fc5687e7257ba2dca/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D87&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D87&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b984d47451813030dc0e094d37f9a543/href">https://medium.com/media/b984d47451813030dc0e094d37f9a543/href</a></iframe><p>To compute the correlation for categorical columns, they must first be converted into numeric columns.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D89&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D89&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/85d9827c98075eb5e3eac9a7b7698bc1/href">https://medium.com/media/85d9827c98075eb5e3eac9a7b7698bc1/href</a></iframe><p>smoker has a better correlation than bmi and age.</p><p>Here’s how correlation coefficients can be interpreted (<a href="https://statisticsbyjim.com/basics/correlations"><strong>source</strong></a>):</p><ul><li><strong>Strength</strong>: The greater the absolute value of the correlation coefficient, the stronger the relationship. The extreme values of -1 and 1 indicate a perfectly linear relationship where a change in one variable is accompanied by a perfectly consistent change in the other. For these relationships, all of the data points fall on a line. In practice, you won’t see either type of perfect relationship.</li><li>A coefficient of zero represents no linear relationship. As one variable increases, there is no tendency in the other variable to either increase or decrease.</li><li>When the value is in-between 0 and +1/-1, there is a relationship, but the points don’t all fall on a line. As r approaches -1 or 1, the strength of the relationship increases and the data points tend to fall closer to a line.</li><li><strong>Direction</strong>: The sign of the correlation coefficient represents the direction of the relationship.</li><li>Positive coefficients indicate that when the value of one variable increases, the value of the other variable also tends to increase. Positive relationships produce an upward slope on a scatterplot.</li><li>Negative coefficients represent cases when the value of one variable increases, the value of the other variable tends to decrease. Negative relationships produce a downward slope.</li></ul><p>Here’s the same relationship expressed visually (<a href="https://www.cuemath.com/data/how-to-calculate-correlation-coefficient/"><strong>source</strong></a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*faMil3xj-YnUL3kk.png" /></figure><p>The correlation coefficient has the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*YeFUKypXcMO-irH8.png" /></figure><p>You can learn more about the mathematical definition and geometric interpretation of correlation here (<a href="https://www.youtube.com/watch?v=xZ_z8KWkhXE"><strong>Video</strong></a>)</p><p>Pandas dataframes also provide a .corr method to compute the correlation coefficients between all pairs of numeric columns.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D92&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D92&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9a9922e61c892c65fb457b563b00a4a/href">https://medium.com/media/d9a9922e61c892c65fb457b563b00a4a/href</a></iframe><p>The result of .corr is called a correlation matrix and is often visualized using a heatmap.</p><p>Let’s arrange this table in ascending order by charges column in order to get the top attributes with highest correlation with charges.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D95&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D95&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/77b01c43abb5edacd24be02a30279bf6/href">https://medium.com/media/77b01c43abb5edacd24be02a30279bf6/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D96&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F1%26cellId%3D96&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fcd726a3d7ba54c0bbb7bc94faefc4bb4&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfa52f095fa83ae739e193b780942b44/href">https://medium.com/media/dfa52f095fa83ae739e193b780942b44/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D97&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D97&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2689a2eaa37b8a3091a03b31092eec7e/href">https://medium.com/media/2689a2eaa37b8a3091a03b31092eec7e/href</a></iframe><p>It is very much evident with the help of correlation table and correlation heat map that ‘age’ and ‘bmi’ attributes have the highest correlation with charges. Thus we can use these two as our independent variables for predicting the annual medical charges.</p><p><strong>Correlation vs causation fallacy:</strong> Note that a high correlation cannot be used to interpret a cause-effect relationship between features. Two features <strong>X</strong> and <strong>Y</strong> can be correlated if <strong>X</strong> causes <strong>Y</strong> or if <strong>Y</strong> causes <strong>X</strong>, or if both are caused independently by some other factor <strong>Z</strong>, and the correlation will no longer hold true if one of the cause-effect relationships is broken. It’s also possible that <strong>X</strong> are <strong>Y</strong> simply appear to be correlated because the sample is too small.</p><h4><strong>Linear Regression using a Single Feature</strong></h4><p>We now know that the “age” and “bmi” columns have the strongest correlation with “charges”. Let’s try to find a way of estimating the value of “charges” using the value of “bmi” first, then we can use “age” later.</p><p>We will do the basics of linear regression by own with codes, in order to understand what all procedure goes behind the scene.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D102&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D102&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dfd492500a8fa64fbccd39c7e2c998d/href">https://medium.com/media/2dfd492500a8fa64fbccd39c7e2c998d/href</a></iframe><p>Apart from a few exceptions, the points seem to form a line. We’ll try and “fit” a line using this points, and use the line to predict charges for a given age. A line on the X&amp;Y coordinates has the following formula:</p><p><em>y=wx+by=wx+b</em></p><p>The line is characterized two numbers: ww (called “slope”) and bb (called “intercept”).</p><h4>Model</h4><p>In the above case, the x axis shows “age” and the y axis shows “charges”. Thus, we’re assume the following relationship between the two:</p><p><em>charges=w×age+b</em></p><p>We’ll try determine <em>w</em> and <em>b</em> for the line that best fits the data.</p><ul><li>This technique is called <em>linear regression</em>, and we call the above equation a <em>linear regression model</em>, because it models the relationship between “age” and “charges” as a straight line.</li><li>The numbers <em>w</em> and <em>b</em> are called the <em>parameters</em> or <em>weights</em> of the model.</li><li>The values in the “age” column of the dataset are called the <em>inputs</em> to the model and the values in the charges column are called “targets”.</li></ul><p>Let define a helper function estimate_charges, to compute <em>charges </em>, given <em>bmi</em>, <em>w</em> and <em>b</em>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D104&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D104&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7b6c863650a07a40714cf2d62f2243c/href">https://medium.com/media/f7b6c863650a07a40714cf2d62f2243c/href</a></iframe><p>The estimate_charges function is our very first <em>model</em>.</p><p>Let’s <em>guess</em> the values for <em>w</em> and <em>b</em> and use them to estimate the value for charges.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D106&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D106&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ca3fe7114b38b04f40e3e043810861c/href">https://medium.com/media/1ca3fe7114b38b04f40e3e043810861c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D107&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D107&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9aedc73818f46d6aa28a90df56f1b02/href">https://medium.com/media/b9aedc73818f46d6aa28a90df56f1b02/href</a></iframe><p>1600 could be one of the value of charges for given bmi as 30. When we se the graph between bmi and charges, we see that for bmi 30 we have many varying charges.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D109&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D109&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4c0429460b9a6f28159eaa02729c045/href">https://medium.com/media/f4c0429460b9a6f28159eaa02729c045/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D110&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D110&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/43ed9defe7646cd2a62ddaa69fee39e5/href">https://medium.com/media/43ed9defe7646cd2a62ddaa69fee39e5/href</a></iframe><p>Thus, our linear model is predicting charges way off than actual, we need to fix this.</p><p>We can plot the estimated charges using a line graph.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D113&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F2%26cellId%3D113&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Ff23cfbeeb727442b8fe3e40a12e4a18c&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/99b71df23cb511d213f669936c629a1e/href">https://medium.com/media/99b71df23cb511d213f669936c629a1e/href</a></iframe><p>As expected, the points lie on a straight line. We can overlay this line on the actual data, so see how well our <em>model</em> fits the data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D115&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D115&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c29af5b85473e5c306abd8aaa04b67e9/href">https://medium.com/media/c29af5b85473e5c306abd8aaa04b67e9/href</a></iframe><p>Clearly, the our estimates are quite poor and the line does not “fit” the data. However, we can try different values of <em>w</em> and <em>b</em> to move the line around. Let’s define a helper function try_parameters which takes w and b as inputs and creates the above plot.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D117&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D117&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/41577f5cb16634a07f6b4dbfc1a99835/href">https://medium.com/media/41577f5cb16634a07f6b4dbfc1a99835/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D118&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D118&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d0067fc266d64f674b05c9fc81da52f/href">https://medium.com/media/2d0067fc266d64f674b05c9fc81da52f/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D119&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D119&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0a3715974a3f72cac99e5ae0167203a/href">https://medium.com/media/b0a3715974a3f72cac99e5ae0167203a/href</a></iframe><p><strong>EXERCISE</strong>: Try various values of <em>w</em> and <em>b</em> to find a line that best fits the data. What is the effect of changing the value of <em>w</em>? What is the effect of changing <em>b</em>?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D121&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D121&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1b2025a1bc3df6ac45e5ede9f916efd/href">https://medium.com/media/f1b2025a1bc3df6ac45e5ede9f916efd/href</a></iframe><p>As we change the values, of <em>w</em> and <em>b</em> manually, trying to move the line visually closer to the points, we are <em>learning</em> the approximate relationship between “age” and “charges”.</p><p>Wouldn’t it be nice if a computer could try several different values of w and b and <em>learn</em> the relationship between &quot;age&quot; and &quot;charges&quot;? To do this, we need to solve a couple of problems:</p><ol><li>We need a way to measure numerically how well the line fits the points.</li><li>Once the “measure of fit” has been computed, we need a way to modify w and b to improve the the fit.</li></ol><p>If we can solve the above problems, it should be possible for a computer to determine w and b for the best fit line, starting from a random guess.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D123&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D123&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0abce96c0b8a545ab0f33ee3f1ac329d/href">https://medium.com/media/0abce96c0b8a545ab0f33ee3f1ac329d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D124&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D124&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/df3b0609a9309c88e3aaee9cac9d0bb9/href">https://medium.com/media/df3b0609a9309c88e3aaee9cac9d0bb9/href</a></iframe><p>We can compare our model’s predictions with the actual targets using the following method:</p><ul><li>Calculate the difference between the targets and predictions (the differenced is called the “residual”)</li><li>Square all elements of the difference matrix to remove negative values.</li><li>Calculate the average of the elements in the resulting matrix.</li><li>Take the square root of the result</li></ul><p>The result is a single number, known as the <strong>root mean squared error</strong> (RMSE). The above description can be stated mathematically as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*MpAf4gwE19lkrJP6.png" /></figure><p>Geometrically, the residuals can be visualized as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*GxfhNuKJTZR8gP9k.png" /></figure><p>Let’s define a function to compute the RMSE.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D126&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D126&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/f133386085209b319ce5bd33fc3e4517/href">https://medium.com/media/f133386085209b319ce5bd33fc3e4517/href</a></iframe><p>Let’s compute the RMSE for our model with a sample set of weights.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D128&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D128&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6859f0b5b6757333d54b3284375cdcf/href">https://medium.com/media/d6859f0b5b6757333d54b3284375cdcf/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D129&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F3%26cellId%3D129&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2F4f9a298344a646f5a597ebbc839f0986&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e54d486ce1494b2a31dcc2932469866/href">https://medium.com/media/8e54d486ce1494b2a31dcc2932469866/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D130&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D130&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5b3b14eecb69b67468150077d591aeb/href">https://medium.com/media/d5b3b14eecb69b67468150077d591aeb/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D131&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D131&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ae6a5c754af705dba82b2865464f55a/href">https://medium.com/media/9ae6a5c754af705dba82b2865464f55a/href</a></iframe><p>Here’s how we can interpret the above number: <em>On average, each element in the prediction differs from the actual target by $16751</em>.</p><p>The result is called the <em>loss</em> because it indicates how bad the model is at predicting the target variables. It represents information loss in the model: the lower the loss, the better the model.</p><p>Let’s modify the try_parameters functions to also display the loss.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D133&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D133&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/283139f8f67f2406a3414bc574ecdc37/href">https://medium.com/media/283139f8f67f2406a3414bc574ecdc37/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D134&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D134&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a767c860f9f2c3f4635fbb3ba4ddf86/href">https://medium.com/media/0a767c860f9f2c3f4635fbb3ba4ddf86/href</a></iframe><p><strong>EXERCISE</strong>: Try different values of <em>w</em> and <em>b</em> to minimize the RMSE loss. What’s the lowest value of loss you are able to achieve? Can you come with a general strategy for finding better values of <em>w</em> and <em>b</em> by trial and error?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D136&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D136&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/97440c43ed0d94e1746fe94fcf91459d/href">https://medium.com/media/97440c43ed0d94e1746fe94fcf91459d/href</a></iframe><p>Now, this time our weight(<em>w</em>) and bias (<em>b</em>) seems to be little better than previous ones as the overall loss is less than previous loss.</p><h4>Optimizer</h4><p>Next, we need a strategy to modify weights w and b to reduce the loss and improve the &quot;fit&quot; of the line to the data.</p><ul><li><strong>Ordinary Least Squares</strong>: <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DszXbuO3bVRk">https://www.youtube.com/watch?v=szXbuO3bVRk</a> (better for smaller datasets)</li><li><strong>Stochastic gradient descent</strong>: <a href="https://jovian.ai/outlink?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsDv4f4s2SB8">https://www.youtube.com/watch?v=sDv4f4s2SB8</a> (better for larger datasets)</li></ul><p>Both of these have the same objective: to minimize the loss, however, while ordinary least squares directly computes the best values for w and b using matrix operations, while gradient descent uses a iterative approach, starting with a random values of w and b and slowly improving them using derivatives.</p><p>Here’s a visualization of how <strong>gradient descent</strong> works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*iXnZBo1fj2g1oIDq.gif" /></figure><p>Doesn’t it look similar to our own strategy of gradually moving the line closer to the points? What we we have done in previous few steps is Stochastic gradient descent.</p><h4>Linear Regression (Simple with one variable) using Scikit-learn</h4><p>In practice, you’ll never need to implement either of the above methods yourself. You can use a library like scikit-learn to do this for you.</p><p>Let’s use the LinearRegression class from scikit-learn to find the best fit line for &quot;bmi&quot; vs. &quot;charges&quot; using the ordinary least squares optimization technique.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D141&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D141&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a63debc8f84258f2b42e10cb87d20d0/href">https://medium.com/media/1a63debc8f84258f2b42e10cb87d20d0/href</a></iframe><p>First, we create a new model object.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D143&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D143&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/35981821ca034f48ed4c19d1f8499c9e/href">https://medium.com/media/35981821ca034f48ed4c19d1f8499c9e/href</a></iframe><p>Next, we can use the `fit` method of the model to find the best fit line for the inputs and targets.</p><p>Not that the input `X` must be a 2-d array, so we’ll need to pass a dataframe, instead of a single column.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D147&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D147&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/23feae4ec691af0c6d0f32ef6b4de20a/href">https://medium.com/media/23feae4ec691af0c6d0f32ef6b4de20a/href</a></iframe><p>Let’s fit the model to the data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D149&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D149&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/71b960ee2d0bc6d4d829953167f083b4/href">https://medium.com/media/71b960ee2d0bc6d4d829953167f083b4/href</a></iframe><p>We can now make predictions using the model. Let’s try predicting the charges for the ages 23, 37 and 61.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D151&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D151&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f45679912ad51d2488d81d56f874e7b/href">https://medium.com/media/9f45679912ad51d2488d81d56f874e7b/href</a></iframe><p>Do these values seem reasonable? Compare them with the scatter plot above. Let’s compute the predictions for the entire set of inputs.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D153&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D153&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/732581a5c8d4cd0551a813a43d03a652/href">https://medium.com/media/732581a5c8d4cd0551a813a43d03a652/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D154&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D154&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/53683bb835534861ea8944b4f17748cd/href">https://medium.com/media/53683bb835534861ea8944b4f17748cd/href</a></iframe><p>Let’s compute the RMSE loss to evaluate the model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D156&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D156&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d57cb765562a639d7cd2e6528a3616f2/href">https://medium.com/media/d57cb765562a639d7cd2e6528a3616f2/href</a></iframe><p>Seems like our prediction is off by $11864 on average, which explanes the fact that there are several outliers.</p><p>The parameters of the model are stored in the `coef_` and `intercept_` properties.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D159&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D159&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/db8e1d8612643b045817a131afd7ff3c/href">https://medium.com/media/db8e1d8612643b045817a131afd7ff3c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D160&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D160&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f858a5c5fae90f6c13679afbd361433/href">https://medium.com/media/6f858a5c5fae90f6c13679afbd361433/href</a></iframe><p>Are these parameters close to your best guesses? Let’s visualize the line created by the above parameters.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D162&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D162&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/769fc67210f8ca9e2a273573dedfa044/href">https://medium.com/media/769fc67210f8ca9e2a273573dedfa044/href</a></iframe><p>Indeed the line is quite close to the points. It is slightly above the cluster of points, because it’s also trying to account for the outliers.</p><p><strong>EXERCISE</strong>: Repeat the steps is this section to train a linear regression model to estimate medical charges with age. Visualize the targets and predictions, and compute the loss.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D165&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D165&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a0101c6405617de774ecc2c58c56a95/href">https://medium.com/media/4a0101c6405617de774ecc2c58c56a95/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D166&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D166&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a29f5d19ecc47094e9a61a4e4e639937/href">https://medium.com/media/a29f5d19ecc47094e9a61a4e4e639937/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D167&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D167&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f30e7db582f7be09829e57e34c787ba/href">https://medium.com/media/3f30e7db582f7be09829e57e34c787ba/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D168&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D168&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8a6d5e1d0b717a3b5c76965fc381b84/href">https://medium.com/media/e8a6d5e1d0b717a3b5c76965fc381b84/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D169&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D169&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b441199a27cf0ebd57ed4a5d6fd96ea/href">https://medium.com/media/6b441199a27cf0ebd57ed4a5d6fd96ea/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D170&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D170&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/13c7d27f9e0e8db714c726124f495c7f/href">https://medium.com/media/13c7d27f9e0e8db714c726124f495c7f/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D171&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D171&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b067d008342548c1ba5f448e1f7437fd/href">https://medium.com/media/b067d008342548c1ba5f448e1f7437fd/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D172&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D172&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bf148aee249a60b584b35ecd6c724e7/href">https://medium.com/media/0bf148aee249a60b584b35ecd6c724e7/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D173&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D173&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/16711ec1852e5b33e1a3f89f2d366f01/href">https://medium.com/media/16711ec1852e5b33e1a3f89f2d366f01/href</a></iframe><p>Thus we can see that rmse loss with ‘age’ is slightly less than with BMI.</p><h4>Machine Learning</h4><p>Congratulations, you’ve just trained your first <em>machine learning model!</em> Machine learning is simply the process of computing the best parameters to model the relationship between some feature and targets.</p><p>Every machine learning problem has three components:</p><ol><li><strong>Model</strong></li><li><strong>Cost Function</strong></li><li><strong>Optimizer</strong></li></ol><p>We’ll look at several examples of each of the above in future tutorials. Here’s how the relationship between these three components can be visualized:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*J0QfnQGCy61tOX8L.png" /></figure><p>As we’ve seen above, it takes just a few lines of code to train a machine learning model using scikit-learn.</p><h3>Linear Regression using Multiple Features</h3><p>So far, we’ve used on the “bmi” and “age” feature separately to estimate “charges”. Now we should take “age”, “bmi”, “children” and “smoker” attributes all together as independent variables and then try to estimate the charges.</p><p><em>charges=w1×age+w2×bmi+w3×children+w4×smoker+b</em></p><h4><strong>Binary Category</strong></h4><p>The “smoker” category has just two values “yes” and “no”. Let’s create a new column “smoker_code” containing 0 for “no” and 1 for “yes”.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D179&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D179&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/25ac38dbe11e9ba130153073a9ed2d76/href">https://medium.com/media/25ac38dbe11e9ba130153073a9ed2d76/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D180&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D180&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d85a8ee21053b67b5f8ffc067c32ac3e/href">https://medium.com/media/d85a8ee21053b67b5f8ffc067c32ac3e/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D181&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D181&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b3b8953bd3409a96cedb59adc326700/href">https://medium.com/media/7b3b8953bd3409a96cedb59adc326700/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D182&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D182&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/394138ab75b50c1fc4f0d42ff88a82d8/href">https://medium.com/media/394138ab75b50c1fc4f0d42ff88a82d8/href</a></iframe><p>We can now use the smoker_code column for linear regression.</p><p><em>charges=w1×age+w2×bmi+w3×children+w4×smoker+b</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D184&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D184&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc833f32bb18585c023fc7ef8f3cedb2/href">https://medium.com/media/bc833f32bb18585c023fc7ef8f3cedb2/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D185&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D185&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/123517e36b684b179218700cd5b15c89/href">https://medium.com/media/123517e36b684b179218700cd5b15c89/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D186&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D186&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c248da936251c9a178da38648de083b/href">https://medium.com/media/9c248da936251c9a178da38648de083b/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D187&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D187&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c20b1622a029a2de253aa1d3a82b2577/href">https://medium.com/media/c20b1622a029a2de253aa1d3a82b2577/href</a></iframe><p>The loss reduces from 11355 to 6056, almost by 50%! This is an important lesson: never ignore categorical data.</p><p>Let’s try adding the “sex” column as well.</p><p><em>charges=w1×age+w2×bmi+w3×children+w4×smoker+w5×sex+b</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D189&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D189&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/394178e391bb762cd9487a025b0f126a/href">https://medium.com/media/394178e391bb762cd9487a025b0f126a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D190&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D190&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c4e90d999df21fba2f0e9f85af4fd6e2/href">https://medium.com/media/c4e90d999df21fba2f0e9f85af4fd6e2/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D191&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D191&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c9d9f239f4ee16febbb7f941a1cb4cc/href">https://medium.com/media/2c9d9f239f4ee16febbb7f941a1cb4cc/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D192&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D192&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3828db857ea3a4556f72f63f10e19aaa/href">https://medium.com/media/3828db857ea3a4556f72f63f10e19aaa/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttp%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D193&amp;dntp=1&amp;display_name=Jovian&amp;url=http%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D193&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c985826bd40a5cab271e3d855c79532a/href">https://medium.com/media/c985826bd40a5cab271e3d855c79532a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D194&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D194&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9a3f44914f8351d7522458c378728b4/href">https://medium.com/media/a9a3f44914f8351d7522458c378728b4/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D195&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D195&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/99697ce4ef63ac3e1120bd7fb54e81e9/href">https://medium.com/media/99697ce4ef63ac3e1120bd7fb54e81e9/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D196&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D196&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f36aeb079f2db6e26be34990c432d48/href">https://medium.com/media/7f36aeb079f2db6e26be34990c432d48/href</a></iframe><p>As you might expect, this does have a significant impact on the loss.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D198&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F4%26cellId%3D198&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fac826b2218ff4b69b86b1dedb2038c7d&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e8060a2d6b3cde620335ea554660d4c/href">https://medium.com/media/0e8060a2d6b3cde620335ea554660d4c/href</a></iframe><h3><strong><em>EXTRAS</em> — <em>Let’s learn some more useful concepts.</em></strong></h3><h4>What is One-hot Encoding</h4><p>The “region” column contains 4 values, so we’ll need to use hot encoding and create a new column for each region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RfP-wQxSCpc8c-vR.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D201&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D201&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9dc83e686876b942d8dad8ee221502d/href">https://medium.com/media/c9dc83e686876b942d8dad8ee221502d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D202&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D202&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/27d1f788aea31f14c40f36832e9f5ab4/href">https://medium.com/media/27d1f788aea31f14c40f36832e9f5ab4/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D203&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D203&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/916b356ceda3913bf4e66517cd96feb9/href">https://medium.com/media/916b356ceda3913bf4e66517cd96feb9/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D204&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D204&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b654732f68fc7e00e2ae56be85ca976a/href">https://medium.com/media/b654732f68fc7e00e2ae56be85ca976a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D205&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D205&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/921a5db8bfd97b7f49783c76e59f10ad/href">https://medium.com/media/921a5db8bfd97b7f49783c76e59f10ad/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D206&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D206&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/d61252b7815038590b5ca761ec9c2fa5/href">https://medium.com/media/d61252b7815038590b5ca761ec9c2fa5/href</a></iframe><p>You could see that each region has a separate column.</p><p>Let’s include the region columns into our linear regression model.</p><p><em>charges=w1×age+ w2×bmi+ w3×children+ w4×smoker+ w5×sex+ w6×region + b</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D209&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D209&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3651c02baa87ce5029b2e514718da41/href">https://medium.com/media/e3651c02baa87ce5029b2e514718da41/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D210&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D210&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3b3d2aa0b723349446ef0a4bffa9c4a/href">https://medium.com/media/b3b3d2aa0b723349446ef0a4bffa9c4a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D211&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D211&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d5219d131e63a51c380bdb558ebb33c/href">https://medium.com/media/9d5219d131e63a51c380bdb558ebb33c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D212&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D212&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3362bbcbd532467725eea8dad869e65a/href">https://medium.com/media/3362bbcbd532467725eea8dad869e65a/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D213&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D213&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a0262755f1fab14d7c7d001793807b0/href">https://medium.com/media/2a0262755f1fab14d7c7d001793807b0/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D214&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D214&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/767f45ba5c1c93547ed9c20c56d23970/href">https://medium.com/media/767f45ba5c1c93547ed9c20c56d23970/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D215&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D215&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e20572f58b96674007546ceae6c2967/href">https://medium.com/media/7e20572f58b96674007546ceae6c2967/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D216&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D216&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/14afaa49964386334d4410f68cfd1a24/href">https://medium.com/media/14afaa49964386334d4410f68cfd1a24/href</a></iframe><p>Once again, this leads to a fairly small reduction in the loss.</p><h4><strong>Graphical representation of all the RMSE’s</strong></h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D219&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D219&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/cce508f3b999a69b4a997fb4d54a4a83/href">https://medium.com/media/cce508f3b999a69b4a997fb4d54a4a83/href</a></iframe><h3>Model Improvements</h3><p>Let’s discuss and apply some more improvements to our model.</p><h4>Feature Scaling</h4><p>Recall that due to regulatory requirements, we also need to explain the rationale behind the predictions our model.</p><p><em>charges=w1×age+w2×bmi+w3×children+w4×smoker+w5×sex+ w6×region+b</em></p><p>To compare the importance of each feature in the model, our first instinct might be to compare their weights.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D221&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D221&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/c24807548916fadda7a03b6f5531d08e/href">https://medium.com/media/c24807548916fadda7a03b6f5531d08e/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D222&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D222&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c26a075215803bd1019e5e5588de639/href">https://medium.com/media/5c26a075215803bd1019e5e5588de639/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D223&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D223&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e4ef8b025257a2c5b658d002cb3232e/href">https://medium.com/media/7e4ef8b025257a2c5b658d002cb3232e/href</a></iframe><p>While it seems like BMI and the “northeast” have a higher weight than age, keep in mind that the range of values for BMI is limited (15 to 40) and the “northeast” column only takes the values 0 and 1.</p><p>Because different columns have different ranges, we run into two issues:</p><ol><li>We can’t compare the weights of different column to identify which features are important</li><li>A column with a larger range of inputs may disproportionately affect the loss and dominate the optimization process.</li></ol><p>For this reason, it’s common practice to scale (or standardize) the values in numeric column by subtracting the mean and dividing by the standard deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/0*sKCB0PslFNMNGhvz.png" /></figure><p>We can apply scaling using the <strong>StandardScaler</strong> class from scikit-learn.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D225&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D225&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6824401f609c589e085822fff6ce5eea/href">https://medium.com/media/6824401f609c589e085822fff6ce5eea/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D226&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D226&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/216e7c8b4d18c658e187f25a1fec7a0d/href">https://medium.com/media/216e7c8b4d18c658e187f25a1fec7a0d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D227&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D227&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a926741c6e6d3ad754becee0759e9ae/href">https://medium.com/media/6a926741c6e6d3ad754becee0759e9ae/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D228&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D228&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1e9e9779b427ad9016c86dae4084848/href">https://medium.com/media/a1e9e9779b427ad9016c86dae4084848/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D229&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D229&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a54c9a3d7677cfec5349f8d85254f350/href">https://medium.com/media/a54c9a3d7677cfec5349f8d85254f350/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D231&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D231&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bce8fb8068129c61e127e62837c524c/href">https://medium.com/media/5bce8fb8068129c61e127e62837c524c/href</a></iframe><p>These can now be combined with the categorical data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D233&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D233&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e824dcf2b9f4d4d2fea7b5afbd47168/href">https://medium.com/media/1e824dcf2b9f4d4d2fea7b5afbd47168/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D234&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D234&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/190786d4ebac903806e59d790b5d1edf/href">https://medium.com/media/190786d4ebac903806e59d790b5d1edf/href</a></iframe><p>We can now compare the weights in the formula:</p><p><em>charges=w1×age+w2×bmi+w3×children+w4×smoker+w5×sex+ w6×region+b</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D236&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D236&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/a26ec597b85bbaff679e469072b841fc/href">https://medium.com/media/a26ec597b85bbaff679e469072b841fc/href</a></iframe><p>As you can see now, the most important feature are:</p><ol><li>Smoker (smoke_code)</li><li>Age</li><li>BMI</li></ol><h4>Creating a Test Set</h4><p>Models like the one we’ve created in this tutorial are designed to be used in the real world. It’s common practice to set aside a small fraction of the data (e.g. 10%) just for testing and reporting the results of the model.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D239&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D239&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a4d0510ec1a8f2017f181b4833be349/href">https://medium.com/media/4a4d0510ec1a8f2017f181b4833be349/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D240&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D240&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a40fc965dbadd647e0c33c8e3f91a76/href">https://medium.com/media/3a40fc965dbadd647e0c33c8e3f91a76/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D241&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D241&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/5187a8a0f7a7015a0b94c243b3f8a824/href">https://medium.com/media/5187a8a0f7a7015a0b94c243b3f8a824/href</a></iframe><p>Let’s compare this with the training loss.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D243&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F5%26cellId%3D243&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fbcfe986c32d1479c82bd0e31770f0be7&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/59f8e1cc1f773b6c5884615fdad938de/href">https://medium.com/media/59f8e1cc1f773b6c5884615fdad938de/href</a></iframe><h3>How to Approach a Machine Learning Problem</h3><p>Here’s a strategy you can apply to approach any machine learning problem:</p><ol><li>Explore the data and find correlations between inputs and targets</li><li>Pick the right model, loss functions and optimizer for the problem at hand</li><li>Scale numeric variables and one-hot encode categorical data</li><li>Set aside a test set (using a fraction of the training set)</li><li>Train the model</li><li>Make predictions on the test set and compute the loss</li></ol><h3>Comparative analysis of various regressors for predicting medical charges.</h3><h4>So far</h4><p>we have done a good EDA, so in this approach we will only focus on MLs. We have learnt that ‘smoker’, ’age’ and ‘bmi’ have highest correlation with charges.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D248&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D248&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ea723dfabe7195a3ec7528167564f90/href">https://medium.com/media/0ea723dfabe7195a3ec7528167564f90/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D249&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D249&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba48f69d92778e4a1f202ff76d0ee721/href">https://medium.com/media/ba48f69d92778e4a1f202ff76d0ee721/href</a></iframe><h4>Covert object data type into categories.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D251&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D251&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/37e48f14a069990a90c7851047357f1e/href">https://medium.com/media/37e48f14a069990a90c7851047357f1e/href</a></iframe><h4>Make a separate data frame for numeric data types.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D253&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D253&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e74ef5d0439e66c76de4499d81477e23/href">https://medium.com/media/e74ef5d0439e66c76de4499d81477e23/href</a></iframe><p><strong>Now join both the data frames</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D255&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D255&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/17b1a4ada9077faed6c08a32d3c1f955/href">https://medium.com/media/17b1a4ada9077faed6c08a32d3c1f955/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D256&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D256&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ebd88a7db9c8998525dd352b2a5c2fd/href">https://medium.com/media/3ebd88a7db9c8998525dd352b2a5c2fd/href</a></iframe><h4>Now convert categories into numerical with Label Encoder.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D258&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D258&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/eabdfcdf2fcdb64c437f0b9dc000fd3b/href">https://medium.com/media/eabdfcdf2fcdb64c437f0b9dc000fd3b/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D259&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D259&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed0d2c3298a7bc490aff8134d91a714c/href">https://medium.com/media/ed0d2c3298a7bc490aff8134d91a714c/href</a></iframe><h4><strong><em>Have you noticed any problem? Do you know, ‘4’ is assigned to which region. For that you will have to look up in df table.</em></strong></h4><h4>Let’s see correlations among different attributes with a heat map.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D262&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D262&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/688243effb8f49dbc4866dd80c63bcc7/href">https://medium.com/media/688243effb8f49dbc4866dd80c63bcc7/href</a></iframe><h4>Multiple Linear Regression</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D264&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D264&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb39e283b3c0a88566240c9bc889281b/href">https://medium.com/media/bb39e283b3c0a88566240c9bc889281b/href</a></iframe><h4>Ridge Regression</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D266&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D266&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/b119479860b9d0355786cdb1063a49c5/href">https://medium.com/media/b119479860b9d0355786cdb1063a49c5/href</a></iframe><h4>Lasso Regression</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D268&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D268&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1ec840aeeb955706c38a5dd65ec6e3a/href">https://medium.com/media/e1ec840aeeb955706c38a5dd65ec6e3a/href</a></iframe><h4><strong>Polynomial Regressor</strong></h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ml%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D270&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ml%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D270&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a4210d95b1995ac702e5d63d93260fd/href">https://medium.com/media/0a4210d95b1995ac702e5d63d93260fd/href</a></iframe><p><strong><em>Nice, we have got the best score so far.</em></strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D272&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D272&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/aae69431fa38710708d50b3eb0ebec6d/href">https://medium.com/media/aae69431fa38710708d50b3eb0ebec6d/href</a></iframe><p><strong><em>Root Mean Squared Error for Polynomial Regression is also lowest among all the algorithms.</em></strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fjovian.ai%2Fembed%3Furl%3Dhttps%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D274&amp;dntp=1&amp;display_name=Jovian&amp;url=https%3A%2F%2Fjovian.ai%2Fatifnhasan%2Fmachine-learning-with-linear-regression-essential-concepts-eda-data-manipulation-one-hot-e%2Fv%2F6%26cellId%3D274&amp;image=https%3A%2F%2Fjovian.ai%2Fapi%2Fgist%2Fac3e148ffdd24f0c8baeffd2978a6334%2Fpreview%2Fc7fa41537a734ed99cca99c4183eb287&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=jovian" width="800" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/355c07f4efea0b0b5a441c745a840b65/href">https://medium.com/media/355c07f4efea0b0b5a441c745a840b65/href</a></iframe><h3>Conclusions</h3><ol><li>For this prediction ‘smoker’, ‘age’, ‘bmi’ are the highest importance attributes.</li><li>Polynomial regression turns out to be the best regression in this case.</li></ol><h3>Revision Questions</h3><ol><li>Why do we have to perform EDA before fitting a model to the data?</li><li>What is a parameter?</li><li>What is correlation?</li><li>What does negative correlation mean?</li><li>How can you find correlation between variables in Python?</li><li>What is causation? Explain difference between correlation and causation with an example.</li><li>Define Linear Regression.</li><li>What is univariate linear regression?</li><li>What is multivariate linear regression?</li><li>What are weights and bias?</li><li>What are inputs and targets?</li><li>What is loss/cost function?</li><li>What is residual?</li><li>What is RMSE value? When and why do we use it?</li><li>What is an Optimizer? What are different types of optimizers? Explain each with an example.</li><li>What library is available in Python to perform Linear Regression?</li><li>What is sklearn.linear_model ?</li><li>What does model.fit() do? What arguments must be given?</li><li>What does model.predict() do? What arguments must be given?</li><li>How do we calculate RMSE values?</li><li>What is model.coef_?</li><li>What is model.intercept_?</li><li>What is SGDRegressor? How is it different from Linear Regression?</li><li>Define Machine Learning. What are the main components in Machine Learning?</li><li>How does loss value help in determining whether the model is good or not?</li><li>What are continuous and categorical variables?</li><li>How do we handle categorical variables in Machine Learning? What are the common techniques?</li><li>What is feature scaling? How does it help in Machine Learning?</li><li>How do we perform scaling in Python?</li><li>What is sklearn.preprocessing?</li><li>What is a Test set?</li><li>How do we split data for model fitting (training and testing) in Python?</li><li>How do you approach a Machine Learning problem?</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=273443d0fc3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>