<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marcos Augusto on Medium]]></title>
        <description><![CDATA[Stories by Marcos Augusto on Medium]]></description>
        <link>https://medium.com/@marcos_augusto_business?source=rss-1fb23d027e4c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*f33Y2z2dRYAqnNO9T20akw.jpeg</url>
            <title>Stories by Marcos Augusto on Medium</title>
            <link>https://medium.com/@marcos_augusto_business?source=rss-1fb23d027e4c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 22:32:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@marcos_augusto_business/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Regularization Techniques for Regression: Part 1]]></title>
            <link>https://medium.com/@marcos_augusto_business/regularization-techniques-for-regression-part-1-f583c0929c92?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f583c0929c92</guid>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ridge-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lasso-regression]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Mon, 03 Oct 2022 15:38:44 GMT</pubDate>
            <atom:updated>2023-06-12T17:51:14.046Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn what is Ridge and Lasso and why they are such powerful tools</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IKmkFDE4OGQQqIZdmKOPPw.png" /></figure><p>Regularization consists of adjusting a regression model by adding a procedure that restricts or regularizes the coefficients estimates through a penalty that “shrinks” those estimates to the origin (to zero).</p><p>Although it is not immediately apparent why regularizing improves the model adjustment, generally, this group of “shrinking ”techniques helps reduce the variance (which can lead to increasing the predictive capability of the model) and allows for variable selection.</p><p>Specifically for linear models (including, <a href="https://en.wikipedia.org/wiki/Generalized_linear_model">Generalized Linear Models</a>), there is another advantage: those techniques can remediate <a href="https://en.wikipedia.org/wiki/Multicollinearity">multicollinearity</a>, which usually produces inconsistent estimates for the coefficients.</p><p>The most used Regularization Techniques are Ridge and Lasso, and the only difference between them is in the format of the penalty applied to the coefficients.</p><p>Next, the techniques will be presented according to the procedure of finding the beta_0, beta_1, …, beta_p that minimizes the residual sum of squares (RSS), although other functions could also be used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*Z3zV4Jl4QI1CrR2-s-jGHg.png" /></figure><p>Taking alpha as zero, we have the Ridge, for alpha as 1, we have the Lasso.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5N4lqUGMpXK1dDxO1v_vDw.png" /></figure><p><strong>Ridge Regression</strong></p><p>The Ridge Regression shrinks the coefficients through a penalty which is the format of the sum of the squares of the coefficients.</p><p>The lambda is called a tuning parameter that is determined separately. Note that when lambda is zero, the penalty has no effect and the Ridge Regression will produce the same coefficients estimates of the <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">Ordinary Least Squares</a> method.</p><p>When lambda goes to infinity, the impact of the penalty increases and the coefficient estimates will approach zero (not exactly zero).</p><p>For each lambda, a distinct set of coefficients is produced. Selecting a good value of lambda is critical and the method to select it will be explained later.</p><p>In the Ridge Regression, the coefficient estimates can be modified substantially when a predictor is multiplied by a constant. Therefore, it is recommended to standardize the predictors before.</p><p>It is worth mentioning that for linear models (including, Generalized Linear Models), the penalty in the estimation of the coefficients helps the problems that come with multicollinearity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*AoQq0uOUyOnQC-pnYDlRWg.png" /></figure><p>When there is multicollinearity, the estimation can produce inconsistent coefficient estimates (minor changes in the predictors used for model training can lead to substantial changes in the estimates). That phenomenon can be observed in the <strong>left picture</strong> (only two predictors are considered for explanation purposes): the surface has a ridge/crest that is very unstable for finding the coefficients that minimize the function presented earlier.</p><p>But the penalty added creates a well-defined depression (<strong>right picture</strong>) on the surface that stabilizes the estimation procedure, returning more consistent coefficient estimates.</p><p><strong>Lasso Regression</strong></p><p>The Lasso Regression is very similar to the Ridge Regression, but the difference is in the format of the penalty which is the sum of the absolute values and not the square.</p><p>The Lasso Regression also shrinks the coefficient estimates and has the same tuning parameter.</p><p>Furthermore, all the information presented previously for the Ridge Regression is also valid for the Lasso:</p><ul><li>Lamba equals to zero is equivalent to no penalty applied and lambda increasing to infinity return null estimates;</li><li>It helps to solve multicollinearity and reduces variance;</li><li>It is recommended to standardize the predictors.</li></ul><p>The Ridge Regression does not return exactly zero coefficient estimates (can return close to zero), but the Lasso Regression does. Therefore, this regularization technique can be used as a method for predictor selection (many may call feature selection) and for importance ranking them (predictors with estimates equals to zero will be discarded).</p><p><strong>Tuning Parameter Selection</strong></p><p>As I said before, choosing a good value of lambda is a critical decision. So how can we make that decision? We use cross-validation (adicionar link) using the following steps:</p><ul><li>[1] Define a grid of possible values for the tuning parameter;</li><li>[2] Train the model and compute an adequate error measure for each possible value of the grid. For example, MSE (Mean Square Error) or MAE (Mean Absolute Error) for regression or accuracy for classification;</li><li>[3] Choose the tuning parameter as the value that produced the smallest error;</li><li>[4] Train the final model with all the observations and the chosen tuning parameter.</li></ul><p>In Part 2, I will apply those regularization techniques with some code examples.</p><h4><strong>Before you go</strong></h4><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong>Linkedin:</strong> <a href="https://www.linkedin.com/in/marcos-augusto-47o47/">https://www.linkedin.com/in/marcos-augusto-47o47/</a><br><strong>Github:</strong> <a href="https://github.com/MarcosAugusto47">https://github.com/MarcosAugusto47</a></blockquote><h4><strong>Resources</strong></h4><ol><li>Hastie, T., Tibshirani, R., &amp; Friedman, J. H. (2009). <em>The elements of statistical learning: data mining, inference, and prediction.</em> 2nd ed. New York, Springer.</li><li>Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani. (2013). An introduction to statistical learning : with applications in R. New York :Springer.</li></ol><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f583c0929c92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logistic Regression for Diabetes Data]]></title>
            <link>https://medium.com/@marcos_augusto_business/logistic-regression-for-diabetes-data-b2fb5ed1894d?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2fb5ed1894d</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Sat, 06 Aug 2022 04:33:23 GMT</pubDate>
            <atom:updated>2023-06-12T17:50:46.232Z</atom:updated>
            <content:encoded><![CDATA[<h4>If you have a binary outcome, you should not use a Linear Regression!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*db8vLlntNN8v_MEf02xbzw.png" /></figure><p>It is very common to inadvertently try to use Linear Regression for a problem, but the data scientist should carefully understand the nature of what is being studied. A Linear Regression is only suited for a target variable that is numeric and continuous.</p><p>Therefore, for a binary outcome, another approach should be applied. In this article, I will show how to train a simple Logistic Regression (which is a <a href="https://en.wikipedia.org/wiki/Generalized_linear_model">Generalized Linear Model</a>) to predict whether a woman has diabetes or not, given a set of covariables. We will use R and import the <a href="https://www.tidyverse.org/">tidyverse</a> (for some data manipulation utilities) and <a href="https://broom.tidymodels.org/">broom</a>.</p><p><strong>About the data</strong></p><p>The dataset was obtained on <a href="https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database">Kaggle</a> and contains information about patients (only females) at least 21 years old of Pima Indian heritage.</p><p>First, let’s read the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3afc69cfeabb6b8d21a582632490432c/href">https://medium.com/media/3afc69cfeabb6b8d21a582632490432c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*jzuvbO4X2VdwJLNbgB1lig.png" /></figure><p>The Outcome variable identifies if a woman has diabetes (equal to 1) or not (equal to 0). The other variables represent some overall body measures.</p><p><strong>Imputation for zero values</strong></p><p>After checking that there are no null values, we find that there are some zero values for the columns (Glucose, BloodPressure, SkinThickness, Insulin and BMI). That does not make any sense (a human can’t have blood pressure measured as zero!)</p><p>How to handle those weird zero values? There are a bunch of techniques to impute data. We will just replace it by the theseparatedmean of the column separated by the Outcome.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3d9c04f5e37126a636ceb8449f4fb6ea/href">https://medium.com/media/3d9c04f5e37126a636ceb8449f4fb6ea/href</a></iframe><p><strong>Training the model</strong></p><p>The Logistic Regression is a statistical model with (of course) statistical properties, so splitting the data in training and test datasets is not super required, but in a machine learning scenario we will split it. Like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6642a553f35348f1664c2212129196e9/href">https://medium.com/media/6642a553f35348f1664c2212129196e9/href</a></iframe><p>Now we have a training dataset that contains 70% of the original dataframe and a testing dataset with 30%.</p><p>Now, we can train our model. We will use all the covariables and add the square effects for Insulin and Age (this combination was selected because resulted in interesting results later on). To train the model, we write the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/666c3fbb4ac44335812ec0a260156637/href">https://medium.com/media/666c3fbb4ac44335812ec0a260156637/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*Msi8t59stdnHcOx4S4Y8YA.png" /></figure><p>Unlike Linear Regression, the coefficients have an exponential and multiplicative impact on the target variable, holding everything else constant (we will not enter in the details of the interpretation of the coefficients in this article). We can summarise the coefficients impacts as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*b-eQ5lSX7Chy7GytiMciPw.png" /></figure><p>Therefore the DiabetesPedigreeFunction, Age, Pregnancies and Glucose are the most impactful covariables to increase the probability of a woman having diabetes.</p><p><strong>Evaluating the model performance</strong></p><p>Now, let’s use our testing dataset to check the accuracy of the trained model. The accuracy is the proportion of the correct predicted classes, considering all the sample data. It is important to say that accuracy is not the only measure to evaluate the model performance. Other measures are more interesting here, like Recall.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49029f3430f75bdd594a0c76fa354adc/href">https://medium.com/media/49029f3430f75bdd594a0c76fa354adc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/1*rHmPuc-4opvfKliN_6paQg.png" /></figure><p>The accuracy of the model is, approximately, is 82% which is quite high. The Recall is, approximately, is 68% indicating that the trained model is moderately sensible to detect diabetes, given that the sample in fact has diabetes.</p><p><strong>Conclusion</strong></p><p>In this article, a simple Logistic Regression model was trained to predict whether a woman has diabetes or not.</p><p>The imputation method was just by using a mean, but other more complex techniques could be used, such as <a href="https://cran.r-project.org/web/packages/miceRanger/vignettes/miceAlgorithm.html">MICE (Multiple Imputation by Chained Equations).</a></p><p>To evaluate the model performance we looked at the accuracy and the recall metrics.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2fb5ed1894d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Use ‘break’, ‘pass’, and ‘continue’ Statements in Python]]></title>
            <link>https://python.plainenglish.io/break-pass-continue-statements-in-python-58874bc4a265?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/58874bc4a265</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Wed, 04 May 2022 22:10:15 GMT</pubDate>
            <atom:updated>2023-06-12T17:50:12.444Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn how to use those statements and write better code.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rTXBzDn1epJbovq8BQbUaw.png" /></figure><p>Applying loops on Python is required a ton of times, but did you know you can have more control over how the iteration is executed along the code body or that you can exit completely from the loop? Well, that’s when <strong>break</strong>, <strong>pass</strong> and <strong>continue </strong>statements come into place.</p><p>After I completed my first Python programming course, I did not even know that the statements <strong>break</strong>, <strong>pass</strong> and <strong>continue </strong>existed, so I had to research and study a bit more to master those concepts. So my purpose, in this post, is to share the knowledge.</p><p>I think that an effective way to teach something is by showing a very illustrative example. So let’s begin!</p><p><strong>break</strong></p><p>The break statement is very simple. If you add a break statement inside a loop and if your condition is True to trigger the break statement, the loop will be completely exited. Here is the flowchart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*wr3buR5Cz0m1MdXqVzD6YA.png" /></figure><p>Let’s see an illustrative python example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*cLBqAFpyHvkzy_oOYxLZuQ.png" /></figure><p>We are starting a loop that iterates from 1 to 9 and prints the current iteration and if the iteration is the fifth the break statement is executed. As you can see the loop stopped and exited when<em> i</em> reached 5!</p><p><strong>continue</strong></p><p>After the continue statement is triggered the remaining body of the loop will be ignored and the next iteration will promptly start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*pryEYewQd3zo_v883p4VZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*WNkUrv-TOaMf3DqMBDrxuw.png" /></figure><p>You can see that the print command on the bottom was completely ignored after <em>i</em> reached 5.</p><p><strong>pass</strong></p><p>The pass function is the statement that simply does nothing! After the pass statement is triggered, the remaining body of the loop will be executed and the next iteration will start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*ZySu6qrcujbBx35x33vsLg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*XtMNs7dqYk2qaUgQoe8k9g.png" /></figure><p>If the pass statement does nothing why should you use it? Well, to be completely honest, I have never used the pass statement but you can use it in very specific situations to handle errors or exceptions. Let’s imagine that we want to pass to run further statements after a zero division error is raised. Like this one below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*Lgfpqp8WPxu3x4FtyJa3fg.png" /></figure><p>If the pass statement had not been used, it would raise a <strong>ZeroDivisionError </strong>and the loop would be exited right there. The pass statement literally passes seamlessly to the remaining body of the function and the next iterations.</p><p><strong>Conclusion</strong></p><p>We learned that <strong>break</strong>, <strong>pass,</strong> and <strong>continue </strong>are powerful statements that can significantly change the behavior of a loop. If you are interested, check the official documentation here: <a href="https://docs.python.org/3/tutorial/controlflow.html">More Control Flow Tools</a>.</p><p>If you want to know my opinion about whether Python or R is the best programming language, you can check <a href="https://medium.com/@marcosbsb2014/python-or-r-who-wins-84be0400b23">Python or R? Who wins?</a>.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Join our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>community Discord</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58874bc4a265" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/break-pass-continue-statements-in-python-58874bc4a265">How to Use ‘break’, ‘pass’, and ‘continue’ Statements in Python</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a Data Lake? Same thing as Data Warehouse?]]></title>
            <link>https://medium.com/@marcos_augusto_business/what-is-a-data-lake-same-thing-as-data-warehouse-c87e0ef74aed?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c87e0ef74aed</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 23:50:34 GMT</pubDate>
            <atom:updated>2023-06-12T17:51:51.198Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn what is a Data Lake, why it is so important nowadays and what are the differences to a Data Warehouse</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xx7pLfl8MPK3-2Yv0DNDHA.png" /></figure><p>A data lake can be defined as a centralized repository that enables you to safely store all your structured and unstructured data <strong>without scaling restrictions</strong>. The main advantage of a data lake is that allows you to store data without having to think about its structure. The data can be viewed as organic information that flows naturally (that explains the “lake” in the name).</p><p>Efficient data lakes should allow you to store any kind of raw data, such as images, data texts, video, binary files and much more.</p><p>Data lakes attempts to support multiple functionalities of a data project, like data ingestion, data processing, feeding data to machine learning models and visualizations on dashboards, as opposed to a simple database.</p><p>The main purposes of creating a data lake are:</p><ul><li><strong>Increase operational efficiency;</strong></li><li><strong>Easily make data available to other departmental silos;</strong></li><li><strong>Lower transactional costs (pay only for what you really use);</strong></li><li><strong>Offload capacity from databases and data warehouses.</strong></li></ul><p>I would like to expand a bit more on the second topic (Easily make data available to other departmental silos): let’s think about a streaming service (like Netflix, Disney+, Amazon Prime Video) with operations all around the world. With the information about payments collected, business users might want to use that data to create financial reports to present to investors. While another department, maybe an analytics department might be interested to use that same data to later train machine learning models. Those are different tasks, but they are viable in a data lake implementation because you can “easily make data available to other departmental silos”.</p><p>A well-created data lake should have at least four important components:</p><ul><li><strong>Ingest and store: </strong>the data lake should be equipped with appropriate data ingestion and storing tools;</li><li><strong>Catalog and search: </strong>the data lake should be decently cataloged to enable other users to quickly search certain data. If the data is not organized and indexed, your data lake will become a data swamp, like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06AqQO7_FcPWYMpi2nCqxw.png" /></figure><ul><li><strong>Process and serve: </strong>the data lake should be cost-efficient. In other words, only paying for what you use;</li><li><strong>Protect and secure: t</strong>he data lake should be flexible to easily manage permission, control accesses and define good data governance standards;</li></ul><p>Now let’s define what is Data Warehouse: is also a centralized data repository, but demands the definition of very strict and rigorous table structures, before data ingestion. Before ingesting with data, the columns and data types need to be set (data schema).</p><p>Some differences between data lakes and databases/data warehouses can be summarised as follows:</p><ul><li>Data lakes are flexible to handle unstructured data and setting data schemas for the tables before the data ingestion is not required;</li><li>Databases only handle structured data and require the definition of data schema. The data ingestion should be preformatted to fit a certain table;</li><li>Databases predominantly require SQL syntax to be used for querying. This is clearly a limitation;</li><li>Databases are less flexible to scale without data processing because they normally have storage and processing data mechanisms tied together. When those processes are decoupled on a data lake, you can effortlessly use different data processing and visualization components that look at the same data.</li></ul><h4>Conclusion</h4><p>In a nutshell, the main purpose of a data lake is the capability to ingest and store unstructured and structured data that can generate answers for the present or the future and equipped with flexible functionalities.</p><p>This article was heavily inspired by the following Coursera course: <a href="https://www.coursera.org/learn/introduction-to-designing-data-lakes-in-aws">Introduction to Designing Data Lakes on AWS</a>.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c87e0ef74aed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Vs. R: Which is the Better Programming Language?]]></title>
            <link>https://python.plainenglish.io/python-or-r-who-wins-84be0400b23?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/84be0400b23</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Thu, 21 Apr 2022 15:53:43 GMT</pubDate>
            <atom:updated>2023-06-12T17:52:55.807Z</atom:updated>
            <content:encoded><![CDATA[<h4>Find out which is the better programming language.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vYoa0Si8AMdfi3uPr0jJSg.png" /></figure><p>Python is nowadays one of the most popular programming languages around the world. It is used for data engineering, data science, web applications, and much more. R is also famous but not that much, in my opinion, and it is very well known among mathematicians and statisticians, to do statistical/machine learning modeling and data manipulation.</p><p>Which programming language is the better? Well to answer that we will look at three questions:</p><ul><li>Which is the best one for Data Engineering?</li><li>Which is the best one for Data Science?</li><li>Which one is more integrated with other technologies/tools?</li></ul><p>Let’s dive deeply into those topics now:</p><h4>Which is the best one for Data Engineering?</h4><p>Python can be used for data engineering tasks such as extraction, transformation, and load processes. Also, you can easily connect to databases to extract some data or feed the tables.</p><p>R also can be used for data engineering tasks, but R really shines for data manipulation and statistical/machine learning modeling!</p><h4>Which is the best one for Data Science?</h4><p>Python has the popular <a href="https://scikit-learn.org/stable/">scikit-learn</a> library for machine learning and R has already built-in functions to train a bunch of models, specially GLMs (Generalized Linear Models). Is worth mentioning the <a href="https://topepo.github.io/caret/">caret</a> R package that contains a set of functions that attempt to streamline the process of creating predictive models.</p><p>For EDA (Exploratory Data Analysis), I think R is the way to go. The library <a href="https://dplyr.tidyverse.org/">dplyr</a> from the <a href="https://www.tidyverse.org/">tidyverse</a> is the best tool that I have ever used to manipulate data. Even better than pandas, in my opinion. If you combine dplyr with the ggplot2 (data visualization package), you can do all kinds of cool charts to give meaning to your data.</p><h4>Integration with other technologies/tools?</h4><p>Nowadays, we have a bunch of technologies and tools to use on data projects. Cloud computing services, like AWS, Microsoft Azure, and Google Cloud are present on most projects. I have more experience with AWS and I can say that if you want to run serverless Python code, you can easily use the AWS Lamba service, which does not support R currently.</p><p>To connect to databases, both languages have libraries that are very simple to use.</p><p>Python can be used to create web applications and develop APIs. Therefore, Python is more versatile in my opinion.</p><h4>Conclusion</h4><p>It is common for both Python and R to be required for job positions related to data science. For specifically data engineering positions, I would say that Python is preferable.</p><p>In my personal experience, you can smoothly transition from one to the other but I think that it is great to have some experience in both languages.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Join our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>community Discord</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84be0400b23" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/python-or-r-who-wins-84be0400b23">Python Vs. R: Which is the Better Programming Language?</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Key concepts of a Data Pipeline]]></title>
            <link>https://medium.com/@marcos_augusto_business/key-concepts-of-a-data-pipeline-9c4fa26e1bee?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c4fa26e1bee</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Sat, 09 Apr 2022 21:23:05 GMT</pubDate>
            <atom:updated>2023-06-12T17:53:29.419Z</atom:updated>
            <content:encoded><![CDATA[<h4>If you work with data, you should understand the data pipeline process!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*M-wIWWz5rPyxifO3NwXaRw.png" /><figcaption><a href="https://tigosoftware.com/what-data-pipeline">https://tigosoftware.com/what-data-pipeline</a></figcaption></figure><p>Every data project that gathers multiple sources of data needs a well-defined data pipeline. But what is a data pipeline? A data pipeline simply is the path or series of steps that the information goes from the origin to the end. A good data pipeline will bring data organization and more knowledge to the project.</p><p>We can establish three data layers of a data pipeline:</p><ul><li><strong>Bronze layer:</strong> gathers the raw/unprocessed data tables that were directly ingested;</li><li><strong>Silver layer:</strong> gathers refined tables that were cleaned and manipulated somehow from the bronze layer;</li><li><strong>Gold layer:</strong> gathers the purest and most valuable tables that are ready to be used by machine learning models or to just feed a dashboard.</li></ul><p>The data pipeline can be designed with the following figure, which was originally from the Databricks blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0-N022SwJApWI-T-.png" /><figcaption><a href="https://databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html">https://databricks.com/blog/2019/08/14/productionizing-machine-learning-with-delta-lake.html</a></figcaption></figure><p>Now, we will dive more deeply into those concepts:</p><h4>Bronze layer</h4><p>This layer gathers the original source data that was directly ingested. No manipulation or cleaning was applied. For example, you could export .csv, .xlsx, .json files right here! It can be called the raw data.</p><h4>Silver layer</h4><p>The silver layers gathers the data after some operations of data cleaning or data treatment that makes sense to the project goal. Is is commonly called the refined data. You could have very messy data on the bronze layer with a lot of null or wrong values. For example, let’s say we have the following data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*eju5qGtm950jrUt4x6IZyw.png" /></figure><p>As you can see there are two different strings (“egg” and “eggs”) referring to the same thing! We have financial values with different formats and cells with data! How to deal with all of this? We should clean and do some data treatment to save a more organized and better formatted data and finally store it on the silver layer. Like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*Oglf68uYbWiq-QX4la5_zA.png" /></figure><h4>Gold layer</h4><p>The gold layer gathers the most curated and valuable data, that is completely ready to be used on dashboards or to feed machine learning models. Let’s assume that wehave some files on the silver layer that consists of purchases by customers on a certain time period. If you want to know about the total quantity purchases or the sum of the money spent by the customer, you could aggregate the data and save that on the gold layer. Like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*WyHh1l2uEhav8mN1dMw53Q.png" /></figure><p>It is common to apply some dimensional model technique to create the gold tables (dimension and fact tables), each one with a clear purpose and schema.</p><h4>Conclusion</h4><p>A data pipeline is a fundamental process of every data project. There are multiple variations, but the key concepts will almost always be present.</p><p>Finally, you should definitely check <a href="https://hevodata.com/">Hevo</a>. Hevo is an end-to-end data pipeline platform (Fully Automated No Code ETL) that enables you to easily pull data from all your sources to the warehouse, run transformations for analytics, and deliver operational intelligence to business tools. Hevo helps you transfer data from 150+ sources to BigQuery and my favorite features are: (1) Completely Automated; (2) Provides preload transformations with Python; (3) <a href="https://hevodata.com/integrations/pipeline/">Supports 150+ Integrations</a>; (4) Real-time Data Transfer and (5) Schema Management. Learn more about it at <a href="https://hevodata.com/">https://hevodata.com/</a>.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c4fa26e1bee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 functions from dplyr that you must know, with StarWars data!]]></title>
            <link>https://medium.com/@marcos_augusto_business/7-functions-from-dplyr-that-you-must-know-with-starwars-data-5b7424d883f1?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b7424d883f1</guid>
            <category><![CDATA[dplyr]]></category>
            <category><![CDATA[tidyverse]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[star-wars]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Sat, 02 Apr 2022 20:45:08 GMT</pubDate>
            <atom:updated>2023-06-12T17:54:16.514Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn some simple functions to start right away your data exploration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S9MRaQtnlgLMuHcH2RAZcQ.png" /></figure><p>The incredible data manipulation package <a href="https://dplyr.tidyverse.org/">dplyr</a> gives the possibility to clean/treat your data in a very organized and clear manner. This package is from the <a href="https://www.tidyverse.org/">tidyverse</a>, that contains other really interesting data science libraries.</p><p>This article is going to present 7 functions that you must know, so then you can already start doing your own analyses and data treament for your projects.</p><p>For illustration, we are going to use the built-in Star Wars dataframe, from dplyr.</p><p>First, we import the dplyr library and then we print the first rows of the dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c8584b821b8a3b154ddb4ad106d6ab6/href">https://medium.com/media/0c8584b821b8a3b154ddb4ad106d6ab6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HyzvYGgajMGo8mq-nWTWeA.png" /></figure><p>As you can see there a lot of cool information about the Starwars characters. Let’s see some important functions to explore the data.</p><h3>select</h3><p>The first function is the <strong>select </strong>function that is capable of selecting columns, by passing the column names. Very simple! We will save the dataframe on a object called df.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/647eb22c1cd861ba813079f2a9a4fae8/href">https://medium.com/media/647eb22c1cd861ba813079f2a9a4fae8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*EjbP1QOt0pQqUPhnWyCwfg.png" /></figure><h3>rename</h3><p>Maybe we do not like the column names and we want to change it. For example, we can rename the column homeworld to origin.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4114d29ab82096bc3a42358aada29a84/href">https://medium.com/media/4114d29ab82096bc3a42358aada29a84/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AaJpV4uV6o2Kdqb8cZT2Rg.png" /></figure><h3>arrange</h3><p>The <strong>arrange </strong>function orders your data by a certain column. You can also pass multiple columns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08c01752c5e9713c2f87a4623587f360/href">https://medium.com/media/08c01752c5e9713c2f87a4623587f360/href</a></iframe><p>Here, we can order the data from the lowest height to the highest height. But we can also order it on a descending way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*jkli7AuJUer7Mz68_epnKg.png" /></figure><h3>filter</h3><p>This functions is used to filter rows for categorical or numerical data. Let’s filter only the rows for the Human species.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d62316802799649084618e86fd0f2cc2/href">https://medium.com/media/d62316802799649084618e86fd0f2cc2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUXbcDEGeG50hDVNKGuFKA.png" /></figure><h3>mutate</h3><p>The <strong>mutate </strong>function is very useful function, because it is capable of creating new columns (some may call variables or features). We have the height and the mass of the characters. Let’s try to calculate the Body Mass Index (BMI).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e545642466a189945f5e653702b197ad/href">https://medium.com/media/e545642466a189945f5e653702b197ad/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*HNPWBZk7e478smAEw8OG-w.png" /></figure><p>There you go! We have the BMI calculated, for each character. As you can see we had to provide the columns height and mass.</p><h3>groupby + summarise</h3><p>These functions are very powerful when combined, we can group the data by a category and summarise/aggregate by other colum to make comparative analyses. Let’s study the mean and median values of mass, grouping the data by the species.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/759ac167391ba487c9e655a226d1843d/href">https://medium.com/media/759ac167391ba487c9e655a226d1843d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*K9n1inb_jFkORGNc8PO9AA.png" /></figure><p>Now, lets create a single pipeline using an extense manipulation. Try to read it line by line and identify what it is going on!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea05febc512c88dad0ef70a8c544169d/href">https://medium.com/media/ea05febc512c88dad0ef70a8c544169d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*97Aty6xMOVu61W0za240Cw.png" /></figure><h3>Conclusion</h3><p>With only 7 functions, now you can start doing your own data exploration! I also suggest you to download the dplyr cheatsheet on <a href="https://www.rstudio.com/resources/cheatsheets/">RStudio</a>.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b7424d883f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aprenda as métricas de classificação para modelos de machine learning]]></title>
            <link>https://medium.com/@marcos_augusto_business/entenda-a-matriz-de-confus%C3%A3o-df1ead239cc4?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/df1ead239cc4</guid>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[binary-classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Sat, 12 Mar 2022 15:22:00 GMT</pubDate>
            <atom:updated>2023-06-12T17:55:00.793Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*aWGZdPAMWJe8Nkol.jpg" /></figure><p>Enteder as métricas da matriz de confusão costuma gerar bastante confusão, pois possui diversas métricas associadas e cada pessoa gosta de nomear elas de um modo distinto.</p><p>Com o objetivo de entender com clareza esse conceito, iremos tomar, como exemplo, uma matriz de confusão de dimensão (2 x 2), em um contexto de detecção de vírus (ou uma pessoa tem o vírus ou a pessoa não tem o vírus).</p><p>É importante sempre definir, com clareza, qual é o evento de interesse. O nosso evento de interesse é a pessoa estar contaminada pelo vírus. Normalmente, o evento de interesse é o evento mais raro de ocorrência.</p><p><strong>A matriz em si</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*fXo__uB1UPQOy6K1mTyR_g.png" /></figure><p>A matriz apresenta dados de contagem de ocorrência. Nas linhas, as classes verdadeira são dispostas. (Negativo se a pessoa não tem o vírus e Positivo, caso contrário). Nas colunas, há as predições das classes (Negativo se o modelo prediz que a pessoa não tem o vírus e Positivo, caso contrário).</p><h4>As métricas</h4><p>Nesta parte, recomendo fortemente que você tenha disponível a matriz acima em uma outra aba/tela do computador, para ir acompanhando. Irá ajudar bastante a entender os conceitos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*mtQes9CvJxzNUAyxYsbpKw.png" /></figure><p>A acurácia é medida que corresponde a proporção de acerto do modelo considerando toda a amostra dos dados. Deve-se ter cuidado ao analisá-la, pois é possível obter uma acurácia bastante alta, mas que é inútil para o propósito do modelo de predição.</p><p>Por exemplo, para uma amostra de 100 pessoas com 95 pessoas <strong>não contaminadas </strong>e 5 pessoas <strong>contaminadas</strong>, é possível encontrar os seguintes resultados:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*vl8wEfk_808o-zw9LvZVdQ.png" /></figure><p>Neste caso, a acurácia é de 92%. A primeira vista, parece que o modelo de predição está bom demais! Cuidado! Vamos analisar com calma: das 5 pessoas que tinham, de fato, o vírus, o modelo classificou apenas umas delas como tendo o vírus. O modelo é inútil, não gera valor algum…</p><p>A acurácia é recomendada em cenários com dados mais balanceados, ou seja, em que as frequências das classes são próximas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*Y8TtcJsJFF458JNjXlSB2Q.png" /></figure><p>A precisão é a medida que corresponde a proporção de pessoas que, de fato, estão contaminadas, dado que modelo prediz presença de vírus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*HLmfpAZ3sMr94JhZIgdqkA.png" /></figure><p>A sensibilidade é a medida que corresponde a proporção de acerto quando o modelo prediz que há presença de vírus, dado que a amostra é apenas de clientes que, de fato, estão contaminados pelo vírus. Essa métrica é de fundamental importância, para o nosso contexto, pois é de interesse que o modelo seja sensível a detecção do vírus.</p><p>A sensibilidade também é conhecida como Recall ou Taxa de Verdadeiros Positivos.</p><p>Voltemos a matriz com resultados de exemplo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*vl8wEfk_808o-zw9LvZVdQ.png" /></figure><p>A sensibilidade é de 20% (1/(4+1)), ou seja, o modelo ajustado não consegue detectar bem a presença do vírus.</p><p><strong>Conclusão</strong></p><p>Nota-se, portanto, que a matriz de confusão é bastante informativa para analisar os resultados de uma modelagem.</p><p>As métricas apresentadas já demonstram os conceitos básicos e análises comuns, para os modelos de classificação, em geral. Obviamente, há outras métricas que podem ser derivadas da matriz de confusão, que focam em outros aspectos dos resultados. Não acho relevante expandir tanto aqui.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df1ead239cc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uma análise de dados imobiliários super simples com dplyr]]></title>
            <link>https://medium.com/@marcos_augusto_business/uma-an%C3%A1lise-super-simples-com-dplyr-396b1cef047?source=rss-1fb23d027e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/396b1cef047</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[imoveis]]></category>
            <category><![CDATA[dplyr]]></category>
            <category><![CDATA[r-programming]]></category>
            <category><![CDATA[tidyverse]]></category>
            <dc:creator><![CDATA[Marcos Augusto]]></dc:creator>
            <pubDate>Wed, 02 Mar 2022 16:22:22 GMT</pubDate>
            <atom:updated>2023-06-12T17:55:28.138Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/240/0*3qwHy2-Z1E57fY28.png" /></figure><p>Este artigo irá trazer uma análise de dados super simples sobre dados de imóveis, no Brasil, com a utilização do pacote <a href="https://dplyr.tidyverse.org/">dplyr</a>.</p><h4>Importação dos dados</h4><p>Os dados foram coletados por <a href="https://www.linkedin.com/in/rubenssjr/">Rubens Junior</a> e disponibilizados em <a href="https://www.kaggle.com/rubenssjr/brasilian-houses-to-rent">https://www.kaggle.com/rubenssjr/brasilian-houses-to-rent</a>. Primeiro, importamos a biblioteca dplyr e, após baixar os dados na minha máquina local, podemos fazer a leitura para o ambiente R.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35776efb28b68723de970121bd51864d/href">https://medium.com/media/35776efb28b68723de970121bd51864d/href</a></iframe><h4>Tratamento dos dados</h4><p>Agora, vamos brevemente olhar a estrutura dos dados e verificar presença de valores nulos no dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc857861b07f80ef63b888b0c880c7a9/href">https://medium.com/media/dc857861b07f80ef63b888b0c880c7a9/href</a></iframe><pre>## Rows: 10,692<br>## Columns: 13<br>## $ city                &lt;chr&gt; &quot;SÃ£o Paulo&quot;, &quot;SÃ£o Paulo&quot;, &quot;Porto...<br>## $ area                &lt;int&gt; 70, 320, 80, 51, 25, 376, 72, 213,...<br>## $ rooms               &lt;int&gt; 2, 4, 1, 2, 1, 3, 2, 4, 2, 1, 1, 1...<br>## $ bathroom            &lt;int&gt; 1, 4, 1, 1, 1, 3, 1, 4, 2, 1, 1, 1...<br>## $ parking.spaces      &lt;int&gt; 1, 0, 1, 0, 0, 7, 0, 4, 1, 0, 0, 1...<br>## $ floor               &lt;chr&gt; &quot;7&quot;, &quot;20&quot;, &quot;6&quot;, &quot;2&quot;, &quot;1&quot;, &quot;-&quot;, &quot;7&quot;...<br>## $ animal              &lt;chr&gt; &quot;acept&quot;, &quot;acept&quot;, &quot;acept&quot;, &quot;acept&quot;...<br>## $ furniture           &lt;chr&gt; &quot;furnished&quot;, &quot;not furnished&quot;, &quot;not...<br>## $ hoa..R..            &lt;int&gt; 2065, 1200, 1000, 270, 0, 0, 740,...<br>## $ rent.amount..R..    &lt;int&gt; 3300, 4960, 2800, 1112, 800, 8000,...<br>## $ property.tax..R..   &lt;int&gt; 211, 1750, 0, 22, 25, 834, 85,...<br>## $ fire.insurance..R.. &lt;int&gt; 42, 63, 41, 17, 11, 121, 25, 41,...<br>## $ total..R..          &lt;int&gt; 5618, 7973, 3841, 1421, 836, 8955,...</pre><p>É importante notar que variável <strong>floor, </strong>que representa o andar do imóvel está como string, o que não faz sentido. Então, é feita a transformação para um valor numérico inteiro. Além disso, as variáveis <strong>city, animal </strong>e <strong>furniture </strong>serão transformadas para categorias, por meio da função <strong>as.factor().</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11e47db65d922773603651fb9a99aa72/href">https://medium.com/media/11e47db65d922773603651fb9a99aa72/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91aa5c86b58fa1709b031523aa551019/href">https://medium.com/media/91aa5c86b58fa1709b031523aa551019/href</a></iframe><pre>## city                area               rooms            bathroom <br>##   0                   0                   0                   0 <br>## parking.spaces      floor              animal           furniture <br>##    0                2461                   0                   0 <br>## hoa..R..  rent.amount..R.. property.tax..R.. fire.insurance..R.. <br>## 0                   0                   0                   0 <br>## total..R.. <br>##   0</pre><p>Há um número considerável de valores nulos (n = 2461), para a variável <strong>floor. </strong>Não há muito explicação na documentação desse dataset. Acredito que seja razoável assumir que são imóveis no andar zero (provavelmente, casas). Vamos deixar nulo mesmo.</p><p>É fácil perceber que os nomes das variáveis estão mal definidos. Portanto, fazemos a renomeação:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b537f720cb39a52bb73d47aade54fca/href">https://medium.com/media/0b537f720cb39a52bb73d47aade54fca/href</a></iframe><p>Agora, é interessante usar a função <strong>summary(), </strong>pois ela irá nos trazer diversas medidas que nos ajudarão a entender melhor a distribuição de cada uma das variáveis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1cc1bbc55f88f62d6c208ac03afcd8e0/href">https://medium.com/media/1cc1bbc55f88f62d6c208ac03afcd8e0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*j2ZUbjsfDa1WH5YgUxQaKA.png" /></figure><p>Algumas considerações são interessantes:</p><ul><li>A variável <strong>cidade</strong> possui o valor “SÃ£o Paulo” que está escrito incorretamente;</li><li>A variável <strong>aceita_animal</strong> possui valores em inglês;</li><li>A variável <strong>mobilia</strong> possui valores em inglês;</li><li>Valores extremamente altos (por exemplo, um apartamento no 301° andar !) que não fazem sentido algum, para as variáveis <strong>andar</strong>, <strong>valor_condominio</strong>, <strong>valor_aluguel</strong>, <strong>taxa_propriedade</strong>.</li></ul><p>Todos os problemas podem ser resolvidos com os seguintes comandos:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7b2e8eadbbcfefeda7932af3bd7b70a/href">https://medium.com/media/b7b2e8eadbbcfefeda7932af3bd7b70a/href</a></iframe><h4>Análise dos dados</h4><p>O trabalho do cientista de dados consiste em extrair insights dos dados. Sendo assim, essas são algumas perguntas interessantes para investigarmos:</p><ul><li>Quais cidades possuem os maiores valores de aluguéis e de condomínio?</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/395c7c5c7932744861f980daf2b6bbf6/href">https://medium.com/media/395c7c5c7932744861f980daf2b6bbf6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubGGSrk666TPQhyrOsobLQ.png" /></figure><p>Analisando com média e mediana, São Paulo é a cidade com os maiores valores. Vale ressaltar que há apenas 5 cidades contempladas nesse conjunto de dados.</p><ul><li>Será que imóveis mobiliados têm aluguéis mais altos?</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/451d192fe34af457219452e7df9b3b0c/href">https://medium.com/media/451d192fe34af457219452e7df9b3b0c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*RjVajKRGQgXtP36499HF1g.png" /></figure><p>Sim, imóveis imobiliados possuem aluguéis mais altos. Isso já era esperado.</p><ul><li>Será que o andar influencia no valor do aluguel?</li></ul><p>Para responder essa pergunta, nós iremos criar uma nova variável (<strong>andar_categoria</strong>) que irá assumir uma da seguintes categorias: baixo, médio, alto ou muito alto, que dependem do intervalo no qual o número do andar está inserido.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d2292657fa48d53342271cbb714acc1/href">https://medium.com/media/0d2292657fa48d53342271cbb714acc1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*-U01TNOor2q4_1s9MDB5iw.png" /></figure><p>Nota-se que imóveis em andares muito altos possuem aluguéis mais caros.</p><p><strong>Conclusão</strong></p><p>Nota-se, portanto, que a utilização da biblioteca dplyr, na linguagem R, é extremamente conveniente para tarefas de exploração e de análise de dados.</p><h3>Before you go</h3><p>Follow me on <a href="https://twitter.com/marcoz_auguzto">Twitter,</a> where I regularly tweet about Data Science and Machine Learning.</p><p>I will leave some other info here!</p><blockquote><strong><em>Linkedin:</em></strong><em> </em><a href="https://www.linkedin.com/in/marcos-augusto-47o47/"><em>https://www.linkedin.com/in/marcos-augusto-47o47/</em></a><em><br></em><strong><em>Github:</em></strong><em> </em><a href="https://github.com/MarcosAugusto47"><em>https://github.com/MarcosAugusto47</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=396b1cef047" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>